Imaging system, control method and device thereof, computing equipment and storage medium

By combining non-Bayer arrays and Bayer array imaging devices, color enhancement technology is used to solve the image brightness and color problems in low-illumination environments, high-quality imaging is achieved, and the reliability and cost control of the imaging device are improved.

CN118972705APending Publication Date: 2024-11-15HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310544924.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The prior art is difficult to obtain bright brightness and bright color images in low illumination environments, and the reliability and cost control of the imaging device are difficult.

Method used

Using an imaging device with a non-Bayer array and a Bayer array, a color information of the second image is used to compensate the first image by a color enhancement device to achieve higher quality low-illumination imaging.

Benefits of technology

Obtaining higher brightness and brighter color images in low-illumination environments avoids the color distortion problem caused by infrared fill light, and improves the reliability and cost-effectiveness of the imaging device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972705A_ABST
    Figure CN118972705A_ABST
Patent Text Reader

Abstract

The invention provides an imaging system, a control method of the imaging system, a control device of the imaging system, computing equipment and a computer readable storage medium. The imaging system includes: a first imaging device having a non-Bayer array for taking a first image for an imaging target; a second imaging device having a Bayer array for taking a second image for the imaging target; and the color enhancement device is used for enhancing the first image by using the color information of the second image so as to form a final image of the imaging target. According to the method and the device, the brightness information can be improved through the high sensitivity of the non-Bayer array, meanwhile, richer color information is obtained through the color sensitivity of the Bayer array, the color information and the brightness information are combined to obtain a bright image with bright colors, and higher-quality low-illumination full-color imaging is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of imaging technology, and in particular to an imaging system, a control method for an imaging system, a control device for an imaging system, a computing device, and a computer-readable storage medium. Background Art

[0002] There are technologies for obtaining color images under low ambient illumination in the prior art. However, the low-illumination scene images generated by these technologies either have color cast problems, poor reliability problems, or limited scope of use problems. How to obtain brighter brightness and more vivid colors for images in low-illumination environments, with higher color reproduction, high reliability of imaging devices and controllable costs has always been a problem to be solved in the field. Summary of the invention

[0003] To this end, the present application is dedicated to providing an imaging system, an imaging system control method, an imaging system control device, a computing device and a computer-readable storage medium, which can better achieve full-color imaging under low illumination.

[0004] In one aspect, the present application provides an imaging system, comprising: a first imaging device with a non-Bayer array, used to capture a first image of an imaging target; a second imaging device with a Bayer array, used to capture a second image of the imaging target; and a color enhancement device, used to enhance the first image using color information of the second image to form a final image of the imaging target.

[0005] According to this aspect, an image with higher brightness is obtained by an imaging device with a non-Bayer array, and an image with more vivid colors is obtained by an imaging device with a Bayer array. The former image is enhanced by the latter image to obtain an image with both brightness and color. This can accurately restore the color information of the imaging target under low illumination and avoid the problem of color distortion caused by infrared or other band fill light.

[0006] According to a particular embodiment of the present application, the first imaging device has a first focal length, and the second imaging device has a second focal length shorter than the first focal length.

[0007] According to this embodiment, the focal length of the first imaging device is longer than that of the second imaging device, so that the second imaging device can have a larger aperture, thereby obtaining better color information, which is conducive to compensating the color of the first image through the second image.

[0008] According to a particular embodiment of the present application, the first imaging device has a telephoto lens, and the second imaging device has a wide-angle lens.

[0009] According to this embodiment, the telephoto lens and the wide-angle lens are mature lenses in the art and have their own imaging advantages. Using these two lenses to make the first and second imaging devices can achieve better coordination and synergy effects, and facilitate providing better brightness information and color information.

[0010] According to a particular embodiment of the present application, the imaging system also includes: a first exposure control device, used to obtain the sensitivity of the first imaging device; an exposure interaction control device, used to issue an exposure strategy adjustment notification based on the sensitivity; and a second exposure control device, used to adjust the exposure strategy of the second imaging device based on the exposure strategy adjustment notification.

[0011] According to this embodiment, by interactively controlling the exposure parameters of the first and second imaging devices, the first and second imaging devices can adjust the aperture and exposure time according to objective conditions, thereby obtaining a higher quality image and facilitating subsequent color enhancement.

[0012] According to a particular embodiment of the present application, the second exposure control device is further configured to: adjust the aperture and exposure time of the second imaging device according to the exposure strategy adjustment notification.

[0013] According to this embodiment, in a low-light environment, by adjusting the aperture and exposure time of the second imaging device, it is helpful to obtain a second image with richer color information, thereby better realizing color compensation for the first image.

[0014] According to a particular embodiment of the present application, the imaging system also includes: a calibration device for calibrating the images captured by the first and second imaging devices, so as to obtain mapping transformation parameters between the first and second imaging devices; a parallax correction device for correcting the parallax of the first and second imaging devices according to the mapping transformation parameters, so as to align the first and second images.

[0015] According to this embodiment, by calibrating the first and second imaging devices, the affine transformation relationship between the first and second imaging devices can be obtained, and the second image taken by the second imaging device can be matched to the first image, thereby facilitating color enhancement of the first image by the second image.

[0016] According to a particular embodiment of the present application, the color information includes color temperature, hue, saturation and / or color style.

[0017] According to this embodiment, it is proven that these specific color information can well restore the color of the object itself, and has a great influence on the color quality of the final image.

[0018] According to a particular embodiment of the present application, the non-Bayer array includes an RGBW array, an RGBIR array, a CMYW array or a RYYB array.

[0019] According to the present embodiment, these non-Bayer arrays are proven to be able to obtain brightness information well while avoiding excessive interference or loss of color information.

[0020] According to a particular embodiment of the present application, the color enhancement device is arranged in a server, and the server communicates with the first and second imaging devices.

[0021] According to this embodiment, by setting a color enhancement device that consumes more computing power in a server (such as a cloud server or an edge server) independent of the first and second imaging devices, the computing power requirements of the front-end device (such as a camera) can be reduced, so that the technical solution of the present application can have a wider applicability.

[0022] On the other hand, the present application provides a control method for an imaging system, comprising: acquiring a first image of an imaging target through a first imaging device having a non-Bayer array; acquiring a second image of the imaging target through a second imaging device having a Bayer array; and enhancing the first image using color information of the second image to form a final image of the imaging target.

[0023] According to a particular embodiment of the present application, before acquiring a second image of the imaging target through a second imaging device having a Bayer array, the method also includes: acquiring the sensitivity of the first imaging device; issuing an exposure strategy adjustment notification based on the sensitivity; and adjusting the exposure strategy of the second imaging device based on the exposure strategy adjustment notification.

[0024] According to a particular embodiment of the present application, adjusting the exposure strategy of the second imaging device according to the exposure strategy adjustment notification includes: adjusting the aperture and exposure time of the second imaging device according to the exposure strategy adjustment notification.

[0025] According to a particular embodiment of the present application, before acquiring a first image of an imaging target by a first imaging device having a non-Bayer array, the method further includes: calibrating images captured by the first and second imaging devices, thereby acquiring mapping transformation parameters between the first and second imaging devices. Wherein, before enhancing the first image using color information of the second image to form a final image of the imaging target, the method further includes: correcting the parallax of the first and second imaging devices according to the mapping transformation parameters, thereby aligning the first and second images.

[0026] On the other hand, the present application provides a control device for an imaging system, comprising: a first acquisition module, used to acquire a first image of an imaging target through a first imaging device with a non-Bayer array; a second acquisition module, used to acquire a second image of the imaging target through a second imaging device with a Bayer array; and an enhancement module, used to enhance the first image using color information of the second image to form a final image of the imaging target.

[0027] According to a special embodiment of the present application, the device also includes: a third acquisition module, used to obtain the sensitivity of the first imaging device; a notification module, used to issue an exposure strategy adjustment notification based on the sensitivity; and an adjustment module, used to adjust the exposure strategy of the second imaging device based on the exposure strategy adjustment notification.

[0028] According to a particular embodiment of the present application, the adjustment module is further configured to: adjust the aperture and exposure time of the second imaging device according to the exposure strategy adjustment notification.

[0029] According to a special embodiment of the present application, the device also includes: a calibration module, used to calibrate the images captured by the first and second imaging devices, so as to obtain the mapping transformation parameters between the first and second imaging devices; a correction module, used to correct the parallax of the first and second imaging devices according to the mapping transformation parameters, so as to align the first and second images.

[0030] On the other hand, the present application provides a computing device, which includes a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the above-mentioned imaging system control method.

[0031] In another aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the above-mentioned control method of the imaging system.

[0032] On the other hand, the present application provides a computer program product, including program code, which enables a computer to implement the above-mentioned imaging system control method when the computer program product is executed by the computer.

[0033] Any of the above-mentioned methods, devices, computing equipment, computer-readable storage media or computer program products are used to control the imaging system provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding solutions in the corresponding imaging system provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The specific implementation of the present application is described in detail below with reference to the accompanying drawings, wherein:

[0035] Figure 1 A schematic structural diagram of an imaging system according to an embodiment of the present application is shown;

[0036] Figure 2 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown;

[0037] Figure 3 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown;

[0038] Figure 4 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown;

[0039] Figure 5 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown;

[0040] Figure 6 A schematic diagram showing a flow chart of a control method of an imaging system according to an embodiment of the present application;

[0041] Figure 7 Show according to Figure 6 An example diagram of a color enhancement process in a control method of an embodiment;

[0042] Figure 8 A schematic flow chart showing a method for controlling an imaging system according to another embodiment of the present application;

[0043] Fig. 9 Show according to Figure 7 An example diagram of a parallax correction process in a control method of an embodiment;

[0044] Fig.10 A schematic flow chart showing a method for controlling an imaging system according to another embodiment of the present application;

[0045] Fig.11 A schematic diagram showing the structure of a control device for an imaging system according to an embodiment of the present application is shown;

[0046] Fig.12 A schematic diagram of the structure of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make those skilled in the art understand the concept and thought of the present application more clearly, the present application is described in detail below in conjunction with specific embodiments. It should be understood that the embodiments provided herein are only a part of all embodiments that the present application may have. After reading the specification of the present application, those skilled in the art have the ability to make improvements, transformations, or replacements to part or all of the following embodiments, and these improvements, transformations, or replacements are also included in the scope of the present application.

[0048] In this document, the terms "one", "an" and other similar words are not intended to indicate that there is only one of the things described, but rather that the relevant description is only for one of the things described, and the things described may have one or more. In this document, the terms "comprises", "includes" and other similar words are intended to indicate logical relationships, and cannot be regarded as indicating spatial structural relationships. For example, "A includes B" is intended to indicate that B logically belongs to A, but does not mean that B is spatially located inside A. In addition, the meanings of the terms "comprises", "includes" and other similar words should be regarded as open, rather than closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include other elements such as C, D, and E.

[0049] In this document, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this document, the terms "embodiment", "present embodiment", "one embodiment" and "an embodiment" do not mean that the relevant description is only applicable to a specific embodiment, but rather that these descriptions may also be applicable to one or more other embodiments. Those skilled in the art should understand that in this document, any description made for a certain embodiment can be replaced, combined, or otherwise combined with the relevant description in one or more other embodiments, and the new embodiment produced by the replacement, combination, or other combination is easily conceivable by those skilled in the art and belongs to the scope of protection of this application.

[0050] In each embodiment of the present application, imaging may refer to the process of defining light within a certain area and evolving it into a visible image. Imaging can be divided into projection imaging, reflection imaging, refraction imaging, etc. according to the application of different optical principles. From the essence of imaging, it is to receive information from multiple information channels (such as visual light, sound waves, AC circuits, heat, etc.), and make it perceived by people through devices such as encoding and decoding, focal length, deflector, synchronizer, etc. The essence of imaging is to gather light into a beam, and the gathering point is called the focus. When the light is focused on a specific area, a beam of light can be formed, and this beam of light can form an image, which is the so-called imaging. In addition, the angle and amplitude of light focusing also have a great influence on imaging. When light is focused in a specific area, different images can be formed.

[0051] Low-light scenes refer to scenes with insufficient visible light, which is a huge challenge for imaging based on visible light energy, and usually leads to a sharp decline in image quality, manifested in noise, inaccurate colors, etc. Security cameras, traffic control cameras, etc. need to work 24 hours a day, 7 days a week, and half of the time they work in relatively low-light scenes. How to improve low-light imaging quality and accurately reflect the color, outline and other information of the target are key performance indicators.

[0052] There are three types of low-light technology: visible light fill-in, near-infrared fill-in, and dual-spectrum fusion technology. Visible light fill-in refers to the use of a light source of sufficient intensity to increase the ambient illumination to an illumination level that allows high-quality imaging. The side effect in this case is usually severe light pollution, and it may also give obvious hints to potential targets. Near-infrared fill-in refers to the use of near-infrared bands that are invisible to the human eye but to which the complementary metal-oxide-semiconductor (CMOS) sensor still responds, which can avoid irritation to the human eye and increase the infrared illumination. The side effect is that near-infrared light (generally refers to electromagnetic waves with wavelengths between 780nm and 1100nm, which are non-visible light) cannot carry color information. Dual-spectrum fusion technology refers to the use of near-infrared fill-in while using a spectroscopic design to allow different spectra to be imaged separately, with near-infrared obtaining brightness images and visible light obtaining color and other information, and full-color images obtained through image fusion technology. The side effect is the color cast caused by near-infrared, and the spectroscopic structure usually has certain difficulties in terms of cost and stability.

[0053] As the security business has more and more demands for both the overall situation and details, binocular cameras with asymmetric focal lengths (cameras with two lenses in appearance) are becoming increasingly common. A wide-angle lens is responsible for the overall situation, and a telephoto lens is responsible for the details. However, due to structural and computing power limitations, deploying technologies such as dual-spectrum fusion on this type of camera to improve low-light details will make the system very complicated. Therefore, two Bayer color filter array (CFA) sensors are generally used for independent imaging. The low-light capability is basically determined by their respective optical parameters (aperture, pixel size). The low-light capability is inversely proportional to the aperture number (F number). The larger the F number, the smaller the amount of light entering and the weaker the low-light capability; the low-light capability is proportional to the pixel size. The larger the pixel size, the higher the sensitivity and the stronger the low-light capability.

[0054] The following is a detailed analysis of four technologies existing in this field.

[0055] The first technology is a method of using dual-spectral fusion to improve low-light imaging. A spectroscopic unit is used to spectrally split the incident light and project it onto different imaging sensors respectively, thereby obtaining a visible light image (color information) and a non-visible light image (brightness). The two images are then fused based on the calibration information to obtain a color image with a high signal-to-noise ratio.

[0056] The second technology is a method of improving low-light imaging by using dual-spectrum fusion in the time domain. This method synchronizes the fill light with the sensor exposure in the time domain, periodically fills in near-infrared light to obtain a brightness image, and then fuses the color image after not filling in infrared light.

[0057] The third technology uses two image sensors with the same target surface but different resolutions to acquire two images respectively. The sensor with lower resolution is used for color imaging, which can obtain a better image signal-to-noise ratio. After the two images are fused, a relatively good low-light imaging effect is obtained.

[0058] The fourth technology uses an optical architecture with asymmetric focal lengths or even non-parallel dual lenses, and generates images at intermediate magnifications through image synthesis technology. It mainly compensates for the focal length discontinuity of the two fixed-focus lenses through image interpolation, alignment and synthesis technology.

[0059] The first to third technologies all use dual-spectrum imaging fusion technology to improve low-light imaging quality, the difference lies in the different implementation methods. Technology 1 is through a beam splitter prism, technology 2 is to image sequentially in the time domain (analogously, there are two sensors in the time domain), and technology 3 is similar to technology 1, except that the sensor resolution can be different. One sensor receives non-visible light and the other sensor receives visible light. These technologies have the following defects:

[0060] 1. There is color cast. Since non-visible light imaging cannot carry color information, it is difficult to accurately restore the target color after merging it with the color image of visible light imaging.

[0061] 2. The optical structure is complex, and the stability and reliability are poor. The optical structure of prism spectrometry is a relatively sophisticated optical design with relatively high cost. It is also sensitive to temperature, vibration, etc., and requires precise factory calibration and correction during operation. Once a deviation occurs, the image will be blurred and the image quality will drop sharply. Time-domain spectrometry requires the filter to work periodically to filter out stray light in the environment when there is no near-infrared fill light. The current mechanical infrared filter switch (infrared-cut, IR-CUT) structure cannot adapt to such high-frequency switching, and the complex dual-bandpass filter design is costly and still cannot completely avoid the impact of stray light on color brought by the non-visible light passband.

[0062] 3. Limited scope of use. The first to third technologies all use pixel-level fusion technology, so they are currently only applicable to scenes where the focal lengths of the two images are exactly the same. For asymmetric focal lengths, more accurate pixel-level registration technology is required, so they are currently not applicable.

[0063] In some embodiments of the present application, in order to solve the problem of full-color imaging under low illumination, a method and device are proposed in combination with the camera form in security scenarios. The goal is to obtain an image with a good signal-to-noise ratio under typical lighting conditions (such as ≥0.5Lux) without the need for near-infrared fill light and complex optical design, and at the same time, the color restoration is good and there is no obvious color cast.

[0064] In some embodiments of the present application, two or more imaging devices are provided based on a binocular or multi-purpose optical architecture, wherein the first imaging device uses an image sensor with a Bayer array (e.g., RGBW, CMYW, etc.) with higher sensitivity. Taking RGBW as an example, the amount of light entering is increased by replacing RGB pixels with full-pass pixels, thereby improving the brightness information of the image, while still obtaining color information. In order to preserve color fidelity, some embodiments of the present application do not use near-infrared fill light technology, and therefore mainly rely on ambient light or relatively weak visible light fill light, with a typical illumination of ≥0.5 Lux. While the non-Bayer array sensor of the first imaging device obtains better brightness information, the color quality is reduced due to the reduction in the spatial sampling rate of the color pixels and the crosstalk between pixels. Since the second imaging device has a short focal length, the aperture can be made larger, and the second imaging device only pays attention to the situation and not the specific target, so the exposure time can be relatively longer to obtain better color information. At this time, the first and second imaging devices work together to compensate the first imaging device using the color information of the second imaging device.

[0065] Figure 1 A schematic structural diagram of an imaging system according to an embodiment of the present application is shown.

[0066] According to this embodiment, the imaging system includes a first imaging device, a second imaging device and a color enhancement device. The first imaging device has a non-Bayer array and is used to capture a first image for an imaging target. The second imaging device has a Bayer array and is used to capture a second image for the imaging target. The color enhancement device is used to enhance the first image using color information of the second image to form a final image of the imaging target.

[0067] In this embodiment, the imaging device may refer to a device for converting an objective object or scene into a visible image. For example, the imaging device may include a lens. For another example, the imaging device may include a lens, a filter component (such as a filter, etc.) and an image sensor (such as a CMOS, etc.). For another example, the imaging device may include a lens, a filter component, an image sensor and other chips or computing units (such as an ISP, etc.). For another example, the imaging device may include a camera or a video camera that is sold separately as a whole, and the camera or the video camera may perform the function of the imaging device alone or may communicate with other devices to jointly perform the function of the imaging device.

[0068] In this embodiment, the imaging target may refer to an object or scene that a lens of an imaging device is directed at or photographed.

[0069] In this embodiment, the first and second imaging devices can each capture the first and second images at the same time, which can ensure the consistency of the display content of the first and second images in time and avoid large errors in subsequent fusion processing. When the exposure time lengths of the first and second imaging devices are inconsistent, the two imaging devices can keep the same starting point of the exposure time, and the exposure end time can be different.

[0070] In this embodiment, the Bayer array may refer to a color filter array (CFA) with a Bayer pattern, and the non-Bayer array may refer to a color filter array without a Bayer pattern. The Bayer array may refer to a mosaic color filter array formed by arranging RGB color filters on a light sensing component grid. The Bayer pattern arrangement has 50% green, 25% red, and the remaining 25% blue, so it is also called RGBG, GRGB, or RGGB array.

[0071] As examples, non-Bayer arrays include a red-green-blue-white (RGBW) array, a red-green-blue-infrared (RGBIR) array, a cyan-magenta-yellow-white (CMYW) array, or a red-yellow-yellow-bule (RYYB) array.

[0072] In this embodiment, the color information may refer to information related to the color of the image or capable of affecting the displayed color of the image.

[0073] As an example, the color information includes color temperature, hue, saturation, and / or color style.

[0074] As an example, the first imaging device has a first focal length, and the second imaging device has a second focal length shorter than the first focal length. For example, the first imaging device has a telephoto lens, and the second imaging device has a wide-angle lens.

[0075] In this embodiment, the telephoto lens may refer to a photographic lens with a longer focal length than a standard lens, also known as a telephoto lens or a telephoto lens, with a focal length ranging from 135-800mm, and some even longer. Telephoto lenses are divided into two categories: ordinary telephoto lenses and super telephoto lenses. The focal length of an ordinary telephoto lens is close to that of a standard lens, while the focal length of a super telephoto lens is much greater than that of a standard lens. Taking a 135 camera as an example, a photographic lens with a focal length ranging from 85mm to 300mm is an ordinary telephoto lens, and a super telephoto lens is above 300mm.

[0076] In this embodiment, the wide-angle lens may refer to a photographic lens with a shorter focal length than a standard lens, a larger viewing angle than a standard lens, a longer focal length than a fisheye lens, and a smaller viewing angle than a fisheye lens. Wide-angle lenses are divided into two types: ordinary wide-angle lenses and ultra-wide-angle lenses. The focal length of an ordinary wide-angle lens is generally 38-24 mm, and the viewing angle is 60-84 degrees; the focal length of an ultra-wide-angle lens is 20-13 mm, and the viewing angle is 94-118 degrees. Since the wide-angle lens has a short focal length and a large viewing angle, it can capture a larger area of ​​scenery within a shorter shooting distance.

[0077] In this embodiment, the first imaging device may be a telephoto lens module. As a hardware unit, the telephoto lens module has a relatively long lens focal length and a small field of view (FOV). It is mainly used to observe detail information and is equipped with an image sensor with a non-Bayer color filter array (referred to as non-Bayer array) to improve sensitivity. Due to the higher photosensitivity, better image brightness benefits will be obtained.

[0078] In this embodiment, the second imaging device may be a wide-angle lens module. As a hardware unit, the wide-angle lens module has a relatively short lens focal length and a large FOV, and is mainly used to observe global information. An image sensor with a conventional Bayer color filter array (Bayer array for short) is generally used. Since the second imaging device has a short lens focal length, the aperture can be relatively large, and the amount of light entering is significantly better than that at the telephoto end.

[0079] In this embodiment, the color enhancement device can use the wide-angle end color image with a high signal-to-noise ratio to enhance the signal of the telephoto end color image, thereby obtaining a higher quality color image. In addition to the color information obtained by itself, the first imaging device can enhance its own color information using the better color signal of the second imaging device.

[0080] Figure 2 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown.

[0081] In this embodiment, the imaging system also includes a first imaging device, a second imaging device and a color enhancement device. Figure 1The difference between the embodiments is that the imaging system according to the present embodiment further includes a first exposure control device, an exposure interaction control device, a second exposure control device, a calibration device and a parallax correction device. The first exposure control device is used to obtain the sensitivity of the first imaging device. The exposure interaction control device is used to issue an exposure strategy adjustment notification according to the sensitivity. The second exposure control device is used to adjust the exposure strategy of the second imaging device according to the exposure strategy adjustment notification, and specifically adjust the aperture and exposure time of the second imaging device according to the exposure strategy adjustment notification. The calibration device is used to calibrate the images captured by the first and second imaging devices, so as to obtain the mapping transformation parameters between the first and second imaging devices. The parallax correction device is used to correct the parallax of the first and second imaging devices according to the mapping transformation parameters, so as to align the first and second images.

[0082] In this embodiment, the first exposure control device is used to control the exposure of the first imaging device. As a software functional unit, the first exposure control device controls the shutter, sensitivity (ISO), and aperture according to the raw (RAW) data output by the sensor to achieve moderate brightness of the image to meet the needs of human eye observation. The ISO information basically reflects the ambient illumination under normal circumstances. The second exposure control device is used to control the exposure of the second imaging device, and its function is the same as that of the first exposure control device.

[0083] In some embodiments, the imaging system may also have a light-sensitive photometric device, which is hardware and uses a light-sensitive device such as a photoresistor to detect changes in ambient illumination and provide illumination information. The light-sensitive photometric device is an optional device, and some imaging devices (such as cameras) have it. For imaging devices that do not have this device, the exposure control device can provide a decision.

[0084] In this embodiment, the main function of the exposure interaction control device is to control the second exposure control device according to the illumination estimation result (information reported by the photosensitive light metering device or the first exposure control device), and adjust the automatic exposure (AE) strategy of the wide-angle lens module of the second imaging device, so that the wide-angle end can obtain a color image with a better signal-to-noise ratio. Associating the two AE channels can give full play to the potential optical advantages of the wide-angle end in low-light scenes and obtain a better color signal.

[0085] In this embodiment, the first and second imaging devices each include their own image signal processing device (ISP). As a software functional unit, the ISP completes the conversion of the original RAW image data output by the imaging device sensor into RGB (red, green, blue) or YUV (brightness-chroma-density) image data that is beneficial to the visual perception of the human eye through a series of image algorithms, including color adjustment, noise suppression, detail enhancement and other functions.

[0086] In this embodiment, the imaging system includes an image fusion device. The image fusion device is used to fuse the brightness image and the color image output by the first imaging device. As a software functional unit, the image fusion device adjusts and processes the independently input brightness image (black and white, without color information) and the color image (with color information) through a certain fusion algorithm, and finally outputs an image that takes both brightness and color into consideration.

[0087] In this embodiment, the imaging system includes a calibration device, which mainly determines the coordinate mapping relationship between the two cameras through an offline or online calibration method to provide input for online parallax correction.

[0088] In this embodiment, the imaging system includes a parallax correction device, which uses the coordinate mapping relationship input by the calibration device and assists with relevant algorithms to achieve basic alignment of the telephoto end image and the wide-angle end image, thereby eliminating image differences caused by structural factors.

[0089] Figure 3 A schematic diagram of the structure of an imaging system according to another embodiment of the present application is shown. This embodiment shows a physical form of software and hardware devices of the imaging system.

[0090] According to this embodiment, the imaging system includes at least two imaging devices (or lens modules of imaging devices), namely a first imaging device and a second imaging device. The wide-angle lens of at least one second imaging device uses a conventional image sensor using a Bayer array, and the telephoto lens of at least one first imaging device can use a sensor of a non-Bayer array such as RGBW / RGBIR / CMYW / RYYB.

[0091] In this embodiment, the sensors of the first and second imaging devices respectively input the first and second images 1 (for example, RAW data) to the ISP of the system on chip (SOC) for processing, and output the corresponding first and second images 2 (for example, YUV data). Considering that the color information in the ISP is basically modulated in the YUV domain, this embodiment is mainly processed in the YUV domain. If the system can be compatible with the exposure statistics, white balance statistics, etc. that may be caused by color adjustment, this embodiment can also be applied to the RAW domain, or different color dimensions can be processed in the RAW domain and the YUV domain respectively, such as color temperature and color noise in RAW processing, and color style, saturation, etc. in the YUV domain, so as to obtain better results.

[0092] A central processing unit (CPU) of an advanced RISC machines (ARM) architecture runs the first and second exposure control devices respectively, and runs the exposure interaction control device to complete the interactive exposure control.

[0093] After generating two channels of YUV data, the SOC or its affiliated heterogeneous computing resources, such as a digital signal processor (DSP) / neural processing unit (NPU) / field programmable gate array (FPGA), completes parallax correction and color enhancement of the first image generated by the telephoto lens of the first imaging device.

[0094] The YUV data generated by the wide-angle lens of the second imaging device is passed through an encoder to output a second image 3 (for example, data in the format of High Efficiency Video Coding (H.265)). The enhanced data generated by the telephoto lens of the first imaging device is passed through an encoder to output a first image 3 (for example, data in the format of H.265).

[0095] It can be understood that the first image 1, the first image 2 and the first image 3 are all different representations of the first image. Similarly, the second image 1, the second image 2 and the second image 3 are also different representations of the second image. Among them, the first image 3 can be directly used as the final image, or it can be used as an image that needs to be further processed before generating the final image.

[0096] According to this embodiment, a main control chip (such as SOC) is used to implement access of two channels of image data of the first and second imaging devices. The overall software and hardware structure is relatively simple, and frame-level synchronization of the two imaging devices can be easily achieved.

[0097] Figure 4 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown.

[0098] Figure 4 Example and Figure 3 The main difference between the embodiments is that the two ISPs belong to two SOCs for processing respectively, and two ARM architecture CPUs cooperate to perform interactive exposure control.

[0099] In this embodiment, the second image 2 (e.g., YUV data) of the second imaging device is sent to the SOC of the first imaging device through a data channel, and the computing resources of the first imaging device perform color enhancement on the local first image 2 (e.g., YUV data), and then encode them into the first and second images 3 (e.g., data in H.265 format) for output.

[0100] It is understandable that, since the two ARM architecture CPUs cooperate to implement interactive exposure control, the two CPUs each run a portion of the interactive exposure control device, that is, the interactive exposure control device 1 and the interactive exposure control device 2 together constitute the interactive exposure control device.

[0101] According to this embodiment, two low-end computing chips or two chips, one low and one high, can be used to implement the technical solution of this embodiment, and the image transmission paths of the first and second imaging devices are basically separated. In some embodiments, two independent cameras or cameras can be used and partially modified to implement the imaging system according to this embodiment.

[0102] Figure 5 A schematic structural diagram of an imaging system according to another embodiment of the present application is shown.

[0103] In this embodiment, the color enhancement device is disposed in a server, and the server communicates with the first and second imaging devices.

[0104] In this embodiment, the server may refer to a computing device independent of the first and second imaging devices, such as a cloud server or an edge server. The server and the imaging device may communicate via a network (such as the Internet) or via other means (such as Bluetooth).

[0105] Specifically, Figure 5 The embodiment uses an end-edge or end-cloud approach to complete a portion of the imaging system's functions. In terms of the overall form, the first and second imaging devices may belong to two camera individuals, and the two imaging devices independently process and send the first and second images 3 (e.g., videos in H.265 format), which are transmitted over the network to the edge device or cloud device.

[0106] In this embodiment, the first and second imaging devices implement the exposure interactive control between the CPUs of the ARM architecture through a network or other connection methods. That is, the exposure interactive control device 1 and the exposure interactive control device 2 are connected through a network or other communication methods to jointly implement the functions of the exposure interactive control device.

[0107] According to this embodiment, the abundant computing power is used on the edge / cloud side to realize the functions of the parallax correction device and the color enhancement device. This embodiment adopts the end-edge collaboration solution, which is less restricted by computing power. The powerful computing power of the cloud is used to achieve better color enhancement effects, and the limitations of the front-end computing power will be further reduced. At this time, the front-end can use two main control chips (such as SOC) or one main control chip.

[0108] Figure 6 A schematic flow chart of a control method for an imaging system according to an embodiment of the present application is shown.

[0109] According to this embodiment, the control method of the imaging system includes steps S610 to S630, and each step is described in detail below.

[0110] S610 : Acquire a first image of an imaging target by using a first imaging device having a non-Bayer array.

[0111] S620: Acquire a second image of the imaging target by using a second imaging device having a Bayer array.

[0112] For S610 and S620, see Figure 1 The description of the features of the first and second imaging devices in the embodiments will not be repeated here.

[0113] S630: Use the color information of the second image to enhance the first image to form a final image of the imaging target.

[0114] In S630, the color information of the first image can be enhanced by the color enhancement device according to the output second image, and finally the first image with better color information can be output (directly as the final image or as the image that needs to be further processed before being used as the final image). According to the actual needs and the computing power constraints, the color information can be provided from the second imaging device to the first imaging device directly or through a deep learning network, or the enhanced processing such as guided noise reduction can be achieved through complex algorithms. For examples of color enhancement, please refer to Figure 7 .

[0115] Figure 8 A schematic flow chart of a method for controlling an imaging system according to another embodiment of the present application is shown.

[0116] According to this embodiment, the control method of the imaging system includes steps S810 to S880, and each step is described in detail below.

[0117] S810: Calibrate images captured by the first and second imaging devices to obtain mapping transformation parameters between the first and second imaging devices.

[0118] In S810, feature matching and calibration can be performed on a first image output by a telephoto lens module of a first imaging device and a second image output by a wide-angle lens module of a second imaging device in an offline or high-illuminance scenario to obtain mapping transformation parameters between the two lenses, which can basically map the second image onto the first image.

[0119] S820: Acquire a first image of an imaging target by using a first imaging device having a non-Bayer array.

[0120] For S820, see Figure 1 The description of the features of the first imaging device in the embodiment will not be repeated here.

[0121] S830: Acquire the photosensitivity of the first imaging device.

[0122] In S830, in a low-light scene, the first exposure control device of the first imaging device can determine the ISO information under the current illumination from the ISP of the first imaging device. The larger the information is, the larger the current image noise is. The first exposure control device sends the ISO information to the AE interactive control module (exposure interactive control device). Since the image of the first imaging device has a more decisive role in the final image, in this embodiment, the ISO information is determined by the first imaging device, so that the judgment of the ambient illumination that affects the final imaging quality is more accurate. In another embodiment of the present application, ISO information can also be obtained from the second imaging device (for example, the second exposure control device). Other methods or technical means for obtaining ISO information or judging the ambient illumination conditions are also conceivable.

[0123] S840: Issue an exposure strategy adjustment notification based on the sensitivity.

[0124] In S840, the exposure interaction control device decides whether to start the color enhancement mechanism based on the ISO information reported by the first exposure control device. If the ISO is lower than the set threshold, it is not started and the process ends; otherwise, the color enhancement is started. The exposure interaction control device sends an exposure strategy adjustment notification to the second exposure control device (for example, increasing the aperture, reducing the gain, and lengthening the exposure time).

[0125] S850: Adjust the exposure strategy of the second imaging device according to the exposure strategy adjustment notification.

[0126] As an example, S850 may be specifically implemented as: adjusting the aperture and exposure time of the second imaging device according to the exposure strategy adjustment notification.

[0127] In S850, after receiving the policy adjustment notification, the second exposure control device makes adjustments according to the exposure parameters currently obtained from the ISP of the second imaging device. When the ambient illumination is low, the second imaging device can increase the aperture and exposure time as much as possible while maintaining the basic brightness to obtain a better signal-to-noise ratio. It should be understood that the aperture and exposure time of the second imaging device can be adjusted in both directions. When the ambient illumination fluctuates or increases slightly, the aperture can be appropriately reduced and the exposure time can be appropriately shortened. Then, the ISP of the second imaging device controls the exposure according to the latest exposure parameters and outputs the updated second image.

[0128] S860: Acquire a second image of the imaging target by using a second imaging device having a Bayer array.

[0129] For S860, please see Figure 1 The description of the features of the second imaging device in the embodiment will not be repeated here.

[0130] S870: Correct the parallax of the first and second imaging devices according to the mapping transformation parameters, thereby aligning the first and second images.

[0131] In this embodiment, alignment may refer to establishing a correspondence between the contents shown in the first image and the second image, finding out which parts of the first image correspond to the displayed contents in the second image and / or finding out which parts of the second image correspond to the displayed contents in the first image. For example, when the first image is a telephoto lens image and the second image is a wide-angle lens image, the shooting range of the second image is larger than the first image, and alignment may refer to finding out which part of the second image is the illustrated content of the first image, that is, circling the picture frame of the first image in the second image.

[0132] In S870, the second image is transformed using the correction transformation data output in S810 to substantially align with the first image. The alignment process is described in detail in Fig. 9 .like Fig. 9 As shown, the purpose of image parallax correction is to make the first image correspond to the virtual frame part in the second image.

[0133] S880: Use the color information of the second image to enhance the first image to form a final image of the imaging target.

[0134] For S880, see Figure 1 Embodiments of the invention are related to the description of the features of the color enhancement device and Figure 6 The description of the features of S630 in the embodiment will not be repeated here.

[0135] Figure 6 and Figure 8The execution subject of the control method of the imaging system shown can be a computing unit or a control unit of the imaging system. Figure 3 , when executing Figure 6 and Figure 8 In each step of the control method of the imaging system shown in the figure, the computing unit may include ARM (for example, a CPU of ARM architecture) and DSP / NPU / FPGA. Figure 4 , when executing Figure 6 and Figure 8 In each step of the control method of the imaging system shown in FIG. 1 , the computing unit may include two ARMs and a DSP / NPU / FPGA respectively disposed on two SOCs. Figure 5 , when executing Figure 6 and Figure 8 In each step of the control method of the imaging system shown, the computing unit may include two ARMs respectively arranged on two SOCs and a GPU / CPU in an edge / cloud server.

[0136] Fig.10 A schematic flow chart of a method for controlling an imaging system according to another embodiment of the present application is shown.

[0137] Fig.10 The embodiment is characterized in that it associates the process of the control method of the imaging device with various components of the imaging device.

[0138] like Fig.10 As shown, the control method of the imaging system includes two parts, namely, a calibration process part in an offline or high-illuminance scene and a color enhancement process part in a low-illuminance scene.

[0139] In the calibration process, the first and second imaging devices respectively shoot a certain object (e.g., a checkerboard calibration plate), and then the images of the two imaging devices are matched and / or calibrated by the calibration device of the imaging system to determine the mapping transformation parameters between the first and second imaging devices. This parameter will be used in the color enhancement process.

[0140] In the color enhancement process, the first and second imaging devices respectively shoot imaging targets (e.g., specific scenes or objects to be monitored). After the data or the first image output by the first imaging device is processed by the first ISP, the sensitivity information is obtained by the first exposure control device, and the first exposure control device feeds back the sensitivity information to the exposure interaction control device. The exposure interaction control device decides whether it is necessary to adjust the exposure strategy of the second imaging device according to the sensitivity. If adjustment is required, the exposure interaction control device sends the exposure strategy adjustment notification to the second exposure control device, and the second exposure control device controls the exposure process of the second imaging device by controlling the second ISP. The second imaging device shoots the second image under the adjusted exposure parameters, and outputs it to the parallax correction device after being processed by the second ISP. The parallax correction corrects the parallax of the second image according to the mapping transformation parameters sent by the calibration device, so that the second image is aligned with the first image, so as to facilitate the imaging system to extract color information useful to the first image from the second image, which is used to enhance the color of the first image through the color enhancement device, and finally forms a final image for output.

[0141] Based on the above Figure 6 The method embodiment described above, the present application embodiment also provides a control device for an imaging system, and its structural schematic diagram is as follows Fig.11 The device is used to perform the above Figure 6 The various steps in .

[0142] According to this embodiment, the control device of the imaging system includes a first acquisition module 1110, a second acquisition module 1120 and an enhancement module 1130. The first acquisition module 1110 is used to acquire a first image of an imaging target through a first imaging device having a non-Bayer array. The second acquisition module 1120 is used to acquire a second image of the imaging target through a second imaging device having a Bayer array. The enhancement module 1130 is used to enhance the first image using color information of the second image to form a final image of the imaging target.

[0143] It should be noted that Fig.11 The control device 1100 of the imaging system provided in the embodiment shown in the figure only uses the division of the above-mentioned functional modules as an example when executing the method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Figure 6 The control method embodiments of the imaging system shown belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0144] Fig.12It is a schematic diagram of the hardware structure of a computing device 1200 provided in an embodiment of the present application.

[0145] See also Fig.12 The computing device 1200 includes a processor 1210, a memory 1220, a communication interface 1230, and a bus 1240, wherein the processor 1210, the memory 1220, and the communication interface 1230 are connected to each other via the bus 1240. The processor 1210, the memory 1220, and the communication interface 1230 may also be connected in other connection modes besides the bus 1240.

[0146] Among them, the memory 1220 can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical storage, hard disk, etc.

[0147] The processor 1210 may be a general-purpose processor, which may be a processor that performs specific steps and / or operations by reading and executing the contents stored in a memory (e.g., the memory 1220). For example, the general-purpose processor may be a central processing unit (CPU). The processor 1210 may include at least one circuit to execute Figure 6 The illustrated embodiment provides all or part of the steps of the method for controlling the imaging system.

[0148] Among them, the communication interface 1230 includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting devices within the computing device 1200, as well as interfaces for interconnecting the computing device 1200 with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, an optical fiber interface, an ATM interface, etc. The communication interface 1230 can be connected to an external input device and an output device. For example, the input device can be a microphone or a microphone array for capturing voice input signals; it can be a communication network connector for receiving collected input signals from the cloud or other devices; it can also include, for example, a keyboard, a mouse, etc. The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0149] The bus 1240 may be any type of communication bus for interconnecting the processor 1210 , the memory 1220 , and the communication interface 1230 , such as a system bus.

[0150] The above devices may be arranged on independent chips, or at least partially or completely on the same chip. Whether to arrange each device independently on different chips or to integrate them on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above devices.

[0151] Fig.12 The computing device 1200 shown is merely exemplary. During implementation, the computing device 1200 may also include other components, which are not listed one by one herein.

[0152] An embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the control method of the imaging system according to various embodiments of the present application described above in this specification.

[0153] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The concepts, principles and ideas of the present application are described in detail above in conjunction with specific implementation methods (including embodiments and examples). Those skilled in the art should understand that the implementation methods of the present application are more than the several forms given above. After reading the application documents, those skilled in the art can make any possible improvements, replacements and equivalent forms to the steps, methods, devices and components in the above-mentioned implementation methods, and these improvements, replacements and equivalent forms should be deemed to fall within the scope of the present application. The scope of protection of the present application shall be subject only to the claims.

Claims

1. An imaging system, characterized in that: The system comprises: A first imaging device having a non-Bayer array, for capturing a first image of an imaging target; A second imaging device having a Bayer array, used for capturing a second image for the imaging target; The color enhancement device is used to enhance the first image by using the color information of the second image to form a final image of the imaging target.

2. The imaging system according to claim 1, characterized in that The first imaging device has a first focal length, and the second imaging device has a second focal length shorter than the first focal length.

3. The imaging system according to claim 2, characterized in that The first imaging device has a telephoto lens, and the second imaging device has a wide-angle lens.

4. The imaging system according to claim 1, characterized in that The system further comprises: A first exposure control device, used to obtain the sensitivity of the first imaging device; An exposure interaction control device, for issuing an exposure strategy adjustment notification according to the sensitivity; The second exposure control device is used to adjust the exposure strategy of the second imaging device according to the exposure strategy adjustment notification.

5. The imaging system according to claim 4, characterized in that: The second exposure control device is further configured to: According to the exposure strategy adjustment notification, the aperture and exposure time of the second imaging device are adjusted.

6. The imaging system according to claim 1, characterized in that: The system further comprises: A calibration device, used for calibrating the images captured by the first and second imaging devices, so as to obtain mapping transformation parameters between the first and second imaging devices; The parallax correction device is used to correct the parallax of the first and second imaging devices according to the mapping transformation parameters, so as to align the first and second images.

7. The imaging system according to any one of claims 1 to 6, characterized in that: The color information includes color temperature, hue, saturation and / or color style.

8. The imaging system according to any one of claims 1 to 6, characterized in that: The non-Bayer array includes an RGBW array, an RGBIR array, a CMYW array, or a RYYB array.

9. The imaging system according to any one of claims 1 to 6, characterized in that: The color enhancement device is disposed in a server, and the server communicates with the first and second imaging devices.

10. A control method for an imaging system, characterized in that: The method comprises: Acquire a first image of an imaging target by a first imaging device having a non-Bayer array; Acquire a second image of the imaging target by a second imaging device having a Bayer array; The first image is enhanced using the color information of the second image to form a final image of the imaging target.

11. The control method according to claim 10, characterized in that: Before acquiring the second image of the imaging target by using a second imaging device having a Bayer array, the method further includes: Acquiring the photosensitivity of the first imaging device; issuing an exposure strategy adjustment notification according to the sensitivity; The exposure strategy of the second imaging device is adjusted according to the exposure strategy adjustment notification.

12. The control method according to claim 11, characterized in that: The step of adjusting the exposure strategy of the second imaging device according to the exposure strategy adjustment notification includes: According to the exposure strategy adjustment notification, the aperture and exposure time of the second imaging device are adjusted.

13. The control method according to claim 10, characterized in that: Before acquiring the first image of the imaging target by using the first imaging device having a non-Bayer array, the method further includes: Calibrate the images captured by the first and second imaging devices to obtain mapping transformation parameters between the first and second imaging devices; Before enhancing the first image by using the color information of the second image to form a final image of the imaging target, the method further includes: Based on the mapping transformation parameters, parallax of the first and second imaging devices is corrected to align the first and second images.

14. A control device for an imaging system, characterized in that: The device comprises: A first acquisition module, configured to acquire a first image of an imaging target by using a first imaging device having a non-Bayer array; A second acquisition module, configured to acquire a second image of the imaging target through a second imaging device having a Bayer array; The enhancement module is used to enhance the first image by using the color information of the second image to form a final image of the imaging target.

15. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the control method of the imaging system according to any one of claims 10 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the control method of the imaging system according to any one of claims 10 to 13.