A shooting method, device and equipment

Abstract

The present invention discloses a shooting method, which is applied to a shooting terminal, wherein the shooting terminal comprises a first camera, a second camera and a third camera; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, the resolution of the second camera is greater than the resolution of the first camera and the third camera, the first camera, the second camera and the third camera are all fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera; the method comprises: obtaining a target zoom ratio; determining a target lens from the first camera, the second camera and the third camera according to the target zoom ratio; acquiring an image containing a target scene using the target lens; and obtaining an output image of the target scene according to the image containing the target scene. The target scene is a scene that a user expects to shoot. Through the present invention, a lossless zoom effect of approximately 5x is achieved.

Description

Technical Field

[0001] The present invention relates to the field of terminal technology, and in particular to a shooting method, device and equipment. Background Art

[0002] Zoom is one of the most commonly used modes for taking photos. Sometimes users need to take close-ups of distant objects, such as statues in the distance, portraits 3 to 5 meters away, etc. Sometimes users want to adjust the composition of the photo by zooming. For example, mobile phone users most often use 2 to 5x zoom when taking photos with their mobile phones.

[0003] There are many ways to achieve zoom, including optical zoom and digital zoom. Although both can help to magnify distant objects when taking telephoto shots, only optical zoom can support adding more pixels after the main image is formed, making the main image not only larger but also relatively clearer. This kind of zoom, which not only magnifies the main area but also ensures the clarity of the image, is called lossless zoom.

[0004] Camera terminals usually use fixed-focus lenses or zoom lenses. The biggest difference between them is whether they can perform optical zoom. Fixed-focus lenses cannot perform optical zoom. If you want to zoom in on the central scene, you can only move closer or rely on image interpolation algorithms for digital zoom. Zoom lenses can perform optical zoom. If you want to zoom in on distant objects, you only need to adjust the corresponding zoom ratio within the range of its optical zoom. This ensures that no details are lost while zooming in. For zoom lenses, by adjusting the focal length of the lens, distant objects can be magnified, allowing users to see the details of distant objects clearly.

[0005] However, zoom lenses are generally large and thick, and are commonly found in digital cameras. However, directly using such zoom lenses in portable terminal devices (such as thin and light mobile phones) goes against the user's pursuit of thin and light portable terminal devices, especially high-magnification (greater than 3x) zoom lenses. Therefore, the common practice is to use digital zoom technology to magnify distant objects. However, this technology has an upper limit on the improvement of imaging resolution and clarity. When the magnification is large, the image clarity is lost.

[0006] Therefore, there is an urgent need for a technical means that can enable the imaging of the terminal device to obtain higher resolution and clarity while ensuring the light and thin characteristics of the terminal device. Summary of the invention

[0007] The embodiments of the present invention provide a shooting method, apparatus and device, which achieve lossless imaging at a high zoom ratio and improve the user's shooting experience.

[0008] The specific technical solutions provided by the embodiments of the present invention are as follows:

[0009] In a first aspect, an embodiment of the present invention provides a shooting method, which is applied to a shooting terminal, wherein the shooting terminal includes a first camera, a second camera, and a third camera; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, and the first camera, the second camera, and the third camera are all fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera; the method specifically includes:

[0010] Get the target zoom ratio;

[0011] Determine a target lens from the first camera, the second camera, and the third camera according to the target zoom ratio;

[0012] Using a target lens to capture an image containing a target scene;

[0013] According to the image containing the target scene, an output image of the target scene is obtained.

[0014] The target scene is the photo-taking area that the user ultimately expects, and can also be understood as a preview image in the terminal at a target zoom ratio; therefore, there is a corresponding relationship between the target zoom ratio and the target scene.

[0015] In a second aspect, an embodiment of the present invention provides a shooting device, which is applied to a shooting terminal, and the shooting terminal includes a first camera, a second camera, and a third camera; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, and the first camera, the second camera, and the third camera are all fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera: the device also includes:

[0016] An acquisition module is used to obtain a target zoom ratio;

[0017] A determination module, used to determine a target lens from the first camera, the second camera, and the third camera according to a target zoom ratio;

[0018] An acquisition module, used for acquiring an image containing a target scene using a target lens;

[0019] The image processing module is used to obtain an output image of the target scene based on the image containing the target scene.

[0020] According to the technical solution of the above method and device provided by the embodiment of the present invention, the shooting terminal adopts a combination of multiple fixed-focus lenses instead of using a bulky zoom device, so the thickness of the terminal will not be significantly increased, thereby achieving a lossless zoom effect of approximately 5x, ensuring the beauty of the terminal, especially smart handheld devices such as mobile phones; at the same time, it meets the user's requirements for small, light and thin terminals and lossless imaging under large zoom, and improves the user experience.

[0021] According to the first aspect or the second aspect, in a possible design: the resolution of the second camera is greater than the resolutions of the first camera and the third camera.

[0022] According to the first aspect or the second aspect, in a possible design: the resolution of the second camera is greater than the resolution of the output images of the first camera and the third camera.

[0023] According to the first aspect or the second aspect, in a possible design: when the target zoom ratio is (1, 3) times zoom, the first camera and the second camera are determined to be the target lenses; and the first camera and the second camera are used to respectively capture images containing the target scene. The method can be executed collaboratively by the acquisition module, the determination module, and the acquisition module.

[0024] Compared with the third camera, the first camera and the second camera are short-focus lenses. When a low target zoom ratio is required, the first camera and the second camera are used to collect color images and black-and-white images respectively, and then the center cropping, multi-frame zoom, black-and-white and color fusion methods are used to achieve clear imaging at a low target zoom ratio; these algorithms can be executed by the image processing module.

[0025] According to the first aspect or the second aspect, in a possible design: when the target zoom ratio is [3, 5] times zoom, it is also necessary to determine whether the illumination of the target scene is lower than a preset threshold; if the illumination of the target scene is lower than the preset threshold, the second camera and the third camera are determined to be the target lens; and the second camera and the third camera are used to collect images containing the target scene respectively. If the illumination of the target scene is not lower than the preset threshold, the third camera is determined to be the target lens; and the third camera is used to collect images containing the target scene. The method can be executed collaboratively by the acquisition module, the determination module, and the acquisition module.

[0026] Compared with the first and second cameras, the third camera is a short-focus lens. Under the medium target zoom ratio requirement, the third camera is used to collect color images; if the target scene has sufficient lighting, that is, a non-dark light environment, lossless zoom can be approximately achieved through center cropping and multi-frame zooming methods; if the target scene has insufficient lighting, that is, a dark light environment, the second camera needs to be enabled to collect black images to supplement the details of the color images collected by the third camera; lossless zoom can be approximately achieved through center cropping, multi-frame zooming, and telephoto black and white color fusion methods; and clear imaging under medium target zoom ratios can be achieved. These algorithms can be executed by the image processing module.

[0027] According to the first aspect or the second aspect, in a possible design: when the target zoom ratio is (5, 10] times zoom, it is also necessary to determine whether the illumination of the target scene is lower than a preset threshold; if the illumination of the target scene is lower than the preset threshold, the second camera and the third camera are determined to be target lenses; and the second camera and the third camera are used to respectively capture images containing the target scene. If the illumination of the target scene is not lower than the preset threshold, the third camera is determined to be the target lens; and the third camera is used to capture images containing the target scene. The method can be collaboratively executed by an acquisition module, a determination module, and a capture module.

[0028] Compared with the first and second cameras, the third camera is a short-focus lens. Under the requirement of high target zoom ratio, the third camera is used to collect color images; if the target scene has sufficient illumination, that is, a non-dark light environment, lossless zoom can be approximately achieved through center cropping, multi-frame zoom, and digital zoom methods; if the target scene has insufficient illumination, that is, a dark light environment, the second camera needs to be enabled to collect black images to supplement the details of the color images collected by the third camera; lossless zoom can be approximately achieved through center cropping, multi-frame zoom, digital zoom, and telephoto black and white color fusion methods; and clear imaging under high target zoom ratio can be achieved. These algorithms can be executed by the image processing module.

[0029] According to the first aspect or the second aspect, in one possible design: the equivalent focal length of the third camera is 3 times the equivalent focal length of the second camera, and the equivalent focal length of the second camera is equal to the equivalent focal length of the first camera.

[0030] According to the first aspect or the second aspect, in one possible design: the equivalent focal length of the first camera is 27mm, the equivalent focal length of the second camera is 27mm, and the equivalent focal length of the third camera is 80mm; that is, the equivalent focal length of the third camera is approximately 3 times the equivalent focal length of the first / second camera.

[0031] According to the first aspect or the second aspect, in a possible design: the resolutions of the first camera, the second camera and the third camera are 10M, 20M and 10M respectively.

[0032] It should be understood that a variety of different terminals can be determined according to different user zoom requirements. These terminals can have lenses with different characteristics, and can also provide lens combinations and image processing algorithms under different zoom conditions. As mentioned above, 3x and 5x are used as the boundaries, which is only one of the implementation methods. More broadly, the target zoom ratio of the present invention can include three ranges of low, medium and high. For the convenience of expression, these three ranges are expressed as (1, a), [a, b], (b, c]. Among them, the first camera and the second camera are short-focus lenses (such as an equivalent focal length of 27mm), which have strong short-focus imaging capabilities. However, as the value of the target zoom ratio increases, the clarity of the output image obtained by processing the images collected by the first camera and the second camera becomes worse. The processing algorithm includes multi-frame zoom and black and white color fusion. Therefore, under the constraint of clarity, a has a maximum value. The specific value is related to the user's requirements for clarity, lens parameters and algorithms, which are not listed and limited here. Usually, the lens parameters of the third camera are related to the maximum value of a (for example, when a is 3, the equivalent focal length of the third lens can be 80mm), that is, under the degree of medium zoom requirements, the first camera and the second camera can no longer meet the imaging quality. At this time, the main task of collecting images is undertaken by the third camera with a telephoto lens. However, as the value of the target zoom ratio increases further, the third camera with a telephoto lens will be more sensitive to the image quality. The clarity of the output image obtained by processing the image collected by the third camera will also deteriorate accordingly, wherein the processing algorithm includes multi-frame zoom and black and white color fusion; therefore, under the constraint of clarity, b also has a maximum value, and b can also be understood as the maximum zoom capability that the terminal itself can achieve with approximate losslessness. The specific value is related to the user's requirements for clarity, lens parameters and algorithms, and is not listed and limited here. As for the range of (b, c], the lens combination method is similar to the image acquisition method [a, b], except that the digital zoom algorithm is added at the back to achieve imaging at a high target zoom ratio, but the imaging quality has been lost. Therefore, under the functional constraints of the terminal system, devices, etc., c has a maximum value, and c can also be understood as the maximum zoom capability that the terminal itself can achieve under low clarity requirements. The specific value is related to the user's requirements for clarity, lens parameters, and algorithms, and is not listed and limited here. Through the present invention, lossless zoom can be achieved within the zoom ratio range of (1, b].

[0033] In addition, if the user allows a certain loss in the clarity of the zoomed image, or the terminal device allows the use of a telephoto lens with a larger focal length (such as a 4x or 5x telephoto lens, i.e., an equivalent focal length of 108mm or 135mm, respectively) due to the advancement of image processing algorithms, then in the above possible designs, the target zoom ratio range, lens parameters, and lens combination method can be adaptively adjusted based on the above theory to obtain an image that meets the user's needs. For example, the equivalent focal length of the third camera can be greater than 80mm. These possible designs should all fall within the scope of protection of the present invention.

[0034] In addition, if the user allows a certain increase in the noise or details of the zoom image under light-sensitive conditions, or if the terminal device allows the use of a telephoto lens with a larger focal length (such as a 4x or 5x telephoto lens, i.e., an equivalent focal length of 108mm or 135mm, respectively) due to the advancement of image processing algorithms, then in the above possible designs, the target zoom ratio range, lens parameters, and lens combination methods can be adaptively adjusted based on the above theory to obtain an image that meets the user's needs. For example, the equivalent focal length of the second camera can be greater than 27mm. These possible designs should all fall within the scope of protection of the present invention.

[0035] In addition, if the user allows a certain degree of loss in the clarity of the zoomed image, or due to the advancement of image processing algorithms, the value of b can be greater than 5, such as 5.5x or 6x.

[0036] More specifically, the above possible technical implementations can be achieved by the processor calling the programs and instructions in the memory to perform corresponding processing, such as enabling the camera, controlling the camera to capture images, performing algorithmic processing on the captured images, generating and storing the final output image, etc.

[0037] In a third aspect, an embodiment of the present invention provides a terminal device, comprising a memory, a processor, a bus, a first camera, a second camera and a third camera; the memory, the first camera, the second camera, the third camera and the processor are connected via a bus; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, and the first camera, the second camera and the third camera are all fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal length of the first camera and the second camera; the camera is used to collect image signals under the control of the processor; the memory is used to store computer programs and instructions; the processor is used to call the computer programs and instructions stored in the memory, so that the terminal device executes any possible design method as described above.

[0038] According to the third aspect, in a possible design, the terminal device also includes an antenna system, and the antenna system, under the control of the processor, sends and receives wireless communication signals to achieve wireless communication with a mobile communication network; the mobile communication network includes one or more of the following: GSM network, CDMA network, 3G network, 4G network, 5G network, FDMA, TDMA, PDC, TACS, AMPS, WCDMA, TDSCDMA, WIFI and LTE network.

[0039] The above method, apparatus and device can be applied to the scene where the terminal uses the camera software provided by the terminal to take pictures; they can also be applied to the scene where the terminal uses the third-party camera software to take pictures.

[0040] Through the present invention, a lossless zoom effect of approximately 5x can be achieved on a smart phone, and even in a dark environment, a good balance can be achieved between resolution and noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the structure of a terminal in an embodiment of the present invention;

[0042] Figure 2 is a flow chart of a photographing method in an embodiment of the present invention;

[0043] Figure 3 A specific camera design method in an embodiment of the present invention;

[0044] Figure 4 This is a shooting method of the first optional situation in the embodiment of the present invention;

[0045] Figure 5 The target scene in the embodiment of the present invention changes from the actual captured image to the output image;

[0046] Figure 6 This is a second optional shooting method in the embodiment of the present invention;

[0047] Figure 7 This is a third optional shooting method in the embodiment of the present invention;

[0048] Figure 8 This is a fourth optional shooting method in the embodiment of the present invention;

[0049] Fig. 9 This is a fifth optional shooting method in the embodiment of the present invention;

[0050] Fig.10 is a schematic structural diagram of a camera device in an embodiment of the present invention;

[0051] Fig.11Schematic diagram of a face mask in an embodiment of the present invention;

[0052] Fig.12 A schematic diagram of a method for training a deep neural network in an embodiment of the present invention;

[0053] Fig.13 Schematic diagram of a method for image enhancement based on a neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In the embodiments of the present invention, the terminal may be a device that provides photography and / or data connectivity to users, a handheld device with wireless connection function, or other processing devices connected to a wireless modem, such as a digital camera, a SLR camera, a mobile phone (or a "cellular" phone), a smart phone, which may be a portable, pocket-sized, handheld, wearable device (such as a smart watch, etc.), a tablet computer, a personal computer (PC), a PDA (Personal Digital Assistant), a POS (Point of Sales), a vehicle-mounted computer, a drone, an aerial camera, etc.

[0056] Figure 1 An optional hardware structure diagram of the terminal 100 is shown.

[0057] refer to Figure 1 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150, an audio circuit 160, a speaker 161, a microphone 162, a processor 170, an external interface 180, a power supply 190 and other components. In an embodiment of the present invention, there are at least three cameras 150.

[0058] The camera 150 is used to capture images or videos, and can be triggered to start by application instructions to realize the photo or video function. The camera may include imaging lenses, filters, image sensors and other components. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to focus the light emitted or reflected by all objects (also referred to as the objects to be photographed) in the photographing angle of view to form an image; the filter is mainly used to filter out the redundant light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 170 for subsequent processing.

[0059] Those skilled in the art will understand that Figure 1 The portable multifunctional device is merely an example and does not limit the portable multifunctional device, and may include more or less components than those shown in the figure, or a combination of certain components, or different components.

[0060] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and other input devices 132. The touch screen 131 can collect user touch operations on or near it (such as operations performed by the user using any suitable object such as fingers, joints, stylus, etc. on or near the touch screen), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute commands sent by the processor 170; the touch signal at least includes touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control button 132 , a switch button 133 , etc.), a trackball, a mouse, a joystick, and the like.

[0061] The display unit 140 may be used to display information input by the user or information provided to the user and various menus of the terminal 100. In the embodiment of the present invention, the display unit is also used to display images acquired by the device using the camera 150, which may include preview images in certain shooting modes, initial images shot, and target images processed by certain algorithms after shooting.

[0062] Furthermore, the touch screen 131 may cover the display panel 141. When the touch screen 131 detects a touch operation on or near it, it is transmitted to the processor 170 to determine the type of the touch event, and then the processor 170 provides a corresponding visual output on the display panel 141 according to the type of the touch event. In this embodiment, the touch screen and the display unit can be integrated into one component to realize the input, output, and display functions of the terminal 100; for ease of description, the embodiment of the present invention uses a touch display screen to represent the functional set of the touch screen and the display unit; in some embodiments, the touch screen and the display unit can also be two independent components.

[0063] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store the association between the joint touch gesture and the application function; the instruction storage area can store software units such as the operating system, application, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software, and data resources including management computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0064] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on independent chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0065] The radio frequency unit 110 can be used for receiving and sending information or receiving and sending signals during a call. In particular, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0066] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the terminal 100. The audio circuit 160 can transmit the electrical signal converted from the received audio data to the speaker 161, which is converted into a sound signal for output; on the other hand, the microphone 162 is used to collect sound signals, and can also convert the collected sound signals into electrical signals, which are received by the audio circuit 160 and converted into audio data, and then the audio data is processed by the output processor 170, and then sent to another terminal through the radio frequency unit 110, or the audio data is output to the memory 120 for further processing. The audio circuit can also include an earphone jack 163, which is used to provide a connection interface between the audio circuit and the earphone.

[0067] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0068] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0069] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here. All the methods described below can be applied to Figure 1 in the terminal shown.

[0070] See also Figure 2 As shown, an embodiment of the present invention provides a shooting method, which is applied to a shooting terminal, the terminal includes a first camera, a second camera and a third camera; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, the resolution of the second camera is greater than the resolution of the first camera and the third camera, the first camera, the second camera and the third camera are all fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera; the method comprises the following steps:

[0071] Step 21: Get the target zoom ratio;

[0072] Step 22: determining a target lens from the first camera, the second camera, and the third camera according to the target zoom ratio;

[0073] Step 23: using the target lens to capture an image containing the target scene;

[0074] Step 24: Obtain an output image of the target scene based on the collected image containing the target scene. The target scene is the scene that the user expects to shoot, wherein the resolution of the output image is smaller than the resolution of the second camera.

[0075] Among them, the above three cameras can be located on the front of the terminal device or on the back of the terminal device. Their specific arrangement can be flexibly determined according to the needs of the designer, and this application does not limit it.

[0076] Among them, the industry is accustomed to converting the imaging angle of different-sized photosensitive elements into the lens focal length corresponding to the same imaging angle on a 135 film camera (the photosensitive surface of a 135 film camera is fixed and unchanged, 35mm film specifications). This converted focal length is the equivalent focal length of the 135 film camera, that is, the equivalent focal length. Because the size of the photosensitive element (CCD or CMOS) of a digital camera varies from camera to camera (such as 1 / 2.5 inches, 1 / 1.8 inches, etc.), the same focal length lens has different imaging angles on digital cameras with different-sized photosensitive elements. But for users, what really matters is the camera's shooting range (angle of view), that is, people are more concerned about the equivalent focal length rather than the actual focal length.

[0077] In the specific implementation process, the equivalent focal length of the third camera is larger than the equivalent focal lengths of the first camera and the second camera. In addition to the combination of (27mm, 27mm, 80mm) in the above example, the equivalent focal lengths of the first camera, the second camera, and the third camera can also select other values ​​between 25mm and 35mm. The equivalent focal length of the third camera can be 2-4 times the equivalent focal length of the first camera or the second camera. As a telephoto lens, when the images obtained by the first camera and the second camera can no longer achieve lossless zoom through the algorithm, it takes on lossless zoom image acquisition under greater zoom requirements. This multiple is determined by the maximum zoom ratio that can be achieved by the parameters of the first camera and the second camera and the algorithm performance under the premise of achieving lossless output images; such as 2.5, 3, 3.5 times, etc.; this is only for example and is not limited.

[0078] In a specific implementation, the aperture of the second camera is larger than that of the first camera and the third camera. For example, the aperture value of the second camera is 1.65, the aperture value of the first camera is 1.8, and the aperture value of the third camera is 2.4; for example, the aperture value of the second camera is 1.55, the aperture value of the first camera is 1.7, and the aperture value of the third camera is 2.2, etc. This is for example only and is not intended to be limiting.

[0079] In the specific implementation process, the resolution of the second camera is higher than that of the first camera and the third camera. In addition to the (20M, 10M, 10M) combination in the above example, the resolutions of the second camera, the first camera, and the third camera can also be (20M, 10M, 8M), (24M, 12M, 12M), (24M, 12M, 10M), etc. This is only an example and is not limited.

[0080] Among them, the color camera can be understood as an RGB sensor that can capture the color of the target scene and take color photos; the black and white camera can be understood as a monochrome sensor that only shoots black and white scenes; because the monochrome sensor can capture richer details from the scene, the black and white camera can capture the details and contours in the target scene.

[0081] It should be understood that for black and white cameras, their imaging principles determine that black and white cameras have higher resolution and detail presentation capabilities than color cameras of the same resolution. Specifically, if the black and white camera and the color camera have the same resolution and the same pixel size (English: pixelsize), the resolution of the image captured by the black and white camera in the diagonal direction is twice that of the image captured by the color camera. Furthermore, if a black and white camera with a higher resolution is used, for example, the ratio of the imaging resolution of the black and white camera to the imaging resolution of the color camera is T, then the output image synthesized by the black and white camera and the color camera is equivalent to increasing the optical zoom capability by T times in the horizontal and vertical directions and 2T times in the diagonal direction on the basis of the original color camera zoom capability. For example, the resolution of the color camera is 12M (3968 2976), the resolution of the black and white camera is 20M (5120 3840), then on the basis of the zoom capability of the above color camera, the optical zoom capability is improved by 5120 / 3968 times. Among them, the resolution of the camera is determined by the lens design manufacturer and is related to the manufacturing process and materials. There are various types of cameras with different resolutions in the prior art, which are only listed in the present invention without any limitation.

[0082] If the black and white camera and the color camera participate in imaging at the same time; the rich color information captured by the color camera is integrated with the clear details captured by the black and white camera, so that higher quality photos can be obtained.

[0083] Specifically, in step 11, obtaining the target zoom ratio refers to obtaining the magnification ratio selected by the user, for example, 1.5x zoom (1.5x), 2x zoom (2x), 3x zoom (3x), etc. The target ratio can be based on a preset field of view as a reference, and the preset field of view can be flexibly selected by the user or designer; for example, 78FOV is used as a reference. The value of the target zoom ratio is recorded as n. For example, if the adjustable precision of the camera's focal length is 0.1, nx can be 1.1x, 1.2x, 1.3x; if the adjustable precision is 0.5, nx can be 1.5x, 2.0x, 2.5x, etc....

[0084] It should be understood that the user can select the zoom factor using the zoom factor button on the camera device; the user can also select the zoom factor by inputting gesture commands on the display screen of the camera device; or the system may determine the zoom factor based on the user's input at a specific location.

[0085] Furthermore, when the user presses the shooting button of the camera device or receives a gesture command input on the screen of the camera device, the shutter is triggered, and the target camera captures an image containing the target scene; specifically, the target camera can capture at least one image within the exposure time; the terminal processes these captured pictures to obtain an output image of the target scene.

[0086] It should be understood that the target scene is the scene image that the user expects to shoot; the preview image when the camera system is adjusted to the target magnification is the user's most intuitive perception of the target scene. The present invention provides fixed-focus lenses, so the fixed-focus lens has a fixed field of view when the shooting distance is fixed, so the image actually captured by the target camera is a larger field of view than the target scene; that is, the image containing the target scene is captured.

[0087] Combination Figure 2 , see Figure 3 , Figure 3 A more specific camera design method is provided for an embodiment of the present invention, and the design includes three fixed-focus cameras. In combination with the different characteristics of the three cameras, at least one lens is selectively enabled to capture images at different zoom ratios, and image processing is performed on the captured images to achieve approximately 5x lossless zoom. Simply put, if the image after the camera takes a picture at a zoom of nx and the image taken without zoom at a distance of 1 / n from the object have similar details and clarity, it is called lossless zoom. Generally, optical zoom is used as a benchmark, and it is considered that optical zoom is lossless, so a zoom effect similar to optical zoom can be called lossless zoom. There are some objective tests that can be used to measure the resolution and clarity of an image, such as the Siemens Star Chart provided by Image Engineering (IE).

[0088] The present invention can be implemented in a mobile portable terminal or a smart camera terminal such as a mobile phone or a tablet computer. The user enters the zoom mode and selects the zoom ratio. The camera system in the terminal determines the camera to be enabled according to the user's zoom ratio and the preset camera combination mode; the enabled camera is used to continuously shoot multiple frames of images (if multiple cameras are enabled, the multiple cameras are shot synchronously); for the multiple frames of images shot, a clear zoom image is obtained using a preset corresponding algorithm.

[0089] by Figure 3 For the purpose of description, the following will use specific examples to illustrate different shooting methods and image processing methods provided by the present invention at different target zoom ratios.

[0090] The specific parameters of the first camera can be:

[0091] Equivalent focal length 27mm, color camera, resolution 10M;

[0092] The specific parameters of the second camera can be:

[0093] Equivalent focal length 27mm, black and white camera, resolution 20M;

[0094] The specific parameters of the third camera can be:

[0095] Equivalent focal length 80mm, color camera, resolution 10M.

[0096] Scenario 1

[0097] See also Figure 4 .

[0098] S101, when the target zoom ratio is within the range of 1x~3x, the terminal enables the first camera and the second camera.

[0099] When the user adjusts the camera parameters, once the user sets the target zoom ratio in the interval (1, 3), the terminal will enable the first camera (main camera) and the second camera (secondary camera). At this time, the preview image will also change accordingly. At this time, the preview image is the target scene image that the user expects to shoot. Since the first camera is the main camera and the second camera is the secondary camera, the preview image is a part of the actual view image of the first camera. The size of this part is determined by the target zoom ratio and the preset aspect ratio of the output image (such as 4:3 or 16:9, etc.). It should be understood that the actual view images of the first camera and the second camera are different from the image content in the actual preview image (the same applies to the third camera below). Among them, the actual view images of the first camera and the second camera may not be seen by the user; the preview image is the user's intuitive perception of the target scene that he or she expects to shoot, and it is also the most intuitive embodiment of the target zoom ratio.

[0100] For the convenience of explanation, the width of the image actually captured by the first camera is recorded as w 0 , high is denoted as h 0 , w 0 h 0 is the resolution of the first camera; the width of the image actually captured by the second camera is w 1 , high is denoted as h 1 , w 1 h 1 is the resolution of the second camera. Since the resolutions of the first camera and the second camera are fixed, w 0 、h 0 、w 1 、h 1 can be considered as a constant.

[0101] In one case, w0 ,h 0 If the aspect ratio matches the preset output image, the width and height of the final output image are also w 0 ,h 0 ; In another case, w 0 ,h 0 If the aspect ratio does not match the preset output image, the width and height of the final output image will be w 0 ' and h 0 ', the camera system needs to capture the actual w 0 h 0 The image is cropped to become w 0 ' h 0 ', and then perform subsequent image processing. It should be understood that, in order to facilitate the algorithm description below, the five examples of Case 1 to Case 5 are described by taking the former case as an example. As for the latter case, those skilled in the art can deduce it through common mathematical knowledge, which will not be described in detail in the present invention.

[0102] S102, when the photo function is triggered, the first camera and the second camera respectively continuously shoot their respective actual scene, and obtain m 0 Frame color image and m 1 frame black and white image; where m 0 and m 1 is a positive integer, m 0 With m 1 can be equal, and the present invention does not limit the size relationship and specific values ​​of the two; one implementation method m 0 With m 1 Can be 4 or 6, etc.; m 0 The frame color image can be continuous or discontinuous in time sequence. 1 The black and white image frames can also be continuous or discontinuous in time sequence.

[0103] In one implementation, m 0 or m 1 It can be 1, but when it is 1, the subsequent multi-frame zoom operation is not involved, that is, after executing the subsequent S103, it directly goes to S105 for black and white color fusion; and m 0 or m 1 Usually it should be greater than 1, and this will involve subsequent multi-frame zoom operations, that is, executing subsequent S103, S104, and S105.

[0104] It should be understood that since the first camera and the second camera are fixed-focus lenses, the image actually captured also includes other content besides the preview image, and is not just the zoomed target scene that the user expects to capture in the preview image.

[0105] S103, m 0 The center area of ​​the frame color image is cropped (also referred to as center cropping). For the image actually captured by the first camera, m 0 Frame 0 h 0 / n 2 Size of color image. 1 The center area of ​​the frame black and white image is cropped, and the image actually captured by the second camera is cut out. 1 Frame 1 h 1 / n 2 Size black and white image.

[0106] Center area cropping can be understood as intercepting the user's desired photo area, that is, ensuring that the center of the input image remains unchanged and intercepting a valid area of ​​a specified size. The intercepted area is determined by the user's specified target zoom ratio and the equivalent focal length of the camera.

[0107] Therefore, from the user's intuitive perception, the target scene can narrowly refer to the preview image at the target zoom ratio, or broadly refer to the area cropped from the central area.

[0108] S104, the above m 0 Frame 0 h 0 / n 2 The color image of size is zoomed multiple frames to obtain the color multi-frame zoom result, that is, 1 frame w 0 h 0 / n 2 Size of the color zoom image; the above m 1 Frame 0 h 0 / n 2 The black-and-white image of size is zoomed multiple frames to obtain the black-and-white multi-frame zoom result, that is, 1 frame w 0 h 0 / n 2 Size black and white zoom image.

[0109] The target scene is cropped from the actual captured image, and a single frame of the target scene image is obtained by zooming multiple frames. The changes in the target scene image area and size can be found in Figure 5 .

[0110] In the specific implementation process, users will inevitably have shaking when shooting with their hands, so multiple frames will inevitably have different image contents, and the clarity of the same object in the target scene will be slightly different in different frames. Therefore, by using the information sampled at each position of multiple frames to complement each other, it can be fused into a frame with higher resolution, higher clarity and less noise.

[0111] Among them, an optional multi-frame zoom algorithm process is as follows:

[0112] 1) Selecting a reference frame. Common methods include: selecting the first frame, selecting a frame taken at an intermediate moment, or selecting the clearest frame. For example, the clearer frame of the first two frames can be selected as the reference frame.

[0113] 2) Align each other frame input image to the reference frame, and perform motion compensation on the aligned image according to the reference frame; then step 3) or step 4) can be performed);

[0114] 3) Interpolate and enlarge the multiple frames of images that have completed motion compensation. You can choose Bicubic, Lanczos and other methods to enlarge the size of the image in the central area from w 0 h 0 / n 2 becomes w 0 h 0 ; At the same time, keep the image content in the central area unchanged; this step is optional;

[0115] 4) Input the multiple frames of images obtained in 2) into a pre-trained convolutional neural network to obtain a zoomed frame of image, and reduce the size of the image in the central area from w 0 h 0 / n 2 becomes w 0 h 0 ; while keeping the image content of the central area unchanged. Or the multiple frames of images obtained in 3) are input into another pre-trained convolutional neural network to obtain a zoomed frame of image with a size of w 0 h 0 .

[0116] In the above process, there are many implementation methods such as motion compensation, interpolation amplification, and convolutional neural network, and there are also many implementation methods for the multi-frame zoom algorithm. The present invention does not limit the algorithm. Those skilled in the art should understand that there are many open source algorithms that can be used to call and implement the above process, so they are not elaborated in detail here.

[0117] S105, performing black-and-white and color fusion on the one frame of color zoomed image and the one frame of black-and-white zoomed image obtained in S104 to obtain one frame of w 0 h 0 The color output image, that is, the output image of the target scene, is also the image that the user can save. Here, the resolution of the output image of the target scene is the same as the resolution of the first camera or the third camera.

[0118] Among them, an optional black and white color fusion algorithm process is as follows:

[0119] Algorithm flow:

[0120] 1) Select the fusion branch based on factors such as scene distance; for example, you can fuse color information based on black and white, or fuse the high frequency of black and white based on color;

[0121] 2) Align the black-and-white and color images using global registration combined with local block matching;

[0122] 3) According to the branch selected in 1), information fusion is performed on the aligned black-and-white image and the color image to obtain a fused result;

[0123] 4) Sharpen the fused result.

[0124] In the above algorithm flow, the processing methods involved can adopt mature algorithms in the prior art, such as fusion, alignment, sharpening, etc., which are not limited or elaborated in the present invention.

[0125] As mentioned earlier, two cameras participate in imaging at the same time. The rich color information captured by the color camera is combined with the clear details captured by the black and white camera to obtain higher quality photos.

[0126] Scenario 2

[0127] See also Figure 6 .

[0128] S201, when the target zoom ratio is in the range of 3x~5x, the system needs to determine whether the target scene belongs to a dark light environment; if it does not belong to a dark light environment, the terminal enables the third camera.

[0129] Among them, the judgment of a dark light environment can be determined by whether the preset conditions are met. If the light condition is less than 100Lux, it is considered to be a dark light environment, and if it is greater than or equal to 100Lux, it is considered not to be a dark light environment; the preset value of illumination is freely determined by the user or the terminal design manufacturer, and is not limited in the present invention. In the specific implementation process, the terminal can make a judgment based on the ISO value during normal exposure. For example, when the ISO is greater than or equal to 400, it is determined to be a dark light environment, and when the ISO is less than 400, it is determined to be a non-dark light environment; the preset value of ISO is freely determined by the user or the terminal design manufacturer, and is not limited in the present invention.

[0130] When the user adjusts the camera parameters, once the target zoom ratio set by the user is in the range of [3, 5], the third camera will be enabled, and the preview image will also change accordingly; at this time, the preview image is a part of the actual view image of the third camera, and the size of this part is determined by the target zoom ratio and the preset aspect ratio of the output image.

[0131] S202, when the photo function is triggered, the third camera continuously shoots the actual scene, and obtains m 2 Frame color image.

[0132] The width of the image captured by the third camera is recorded as w 2 , high is denoted as h 2 , w 2 h 2 is the resolution of the third camera. Since the resolution of the third camera is the same as that of the first camera, w 2 = w 0 ,h 2 =h 0 In addition, m 2 Can also be used with m 0 The remaining steps and drawings are also written as w 0 、h 0 and m 0 To express.

[0133] S203, m 0 The center area of ​​the frame color image is cropped, and for the image actually captured by the third camera, m 0 Frame 0 h 0 / (n / n 0 ) 2 Since the equivalent focal length of the third camera is 80mm, that is, a telephoto lens, at the same shooting distance, the image captured is larger than that of a standard lens, but the field of view becomes smaller; therefore, the central area is determined by the target zoom ratio and the equivalent focal length; here n 0Approximately equal to 3 (80mm / 27mm), which is determined by the equivalent focal length of the lens.

[0134] S204, the above m 0 Frame 0 h 0 / (n / n 0 ) 2 The color image of size is zoomed multiple frames to obtain the color multi-frame zoom result, that is, 1 frame w 0 h 0 The color zoom image of the target scene is the output image of the target scene, which is also the picture that the user can save. Here, the resolution of the output image of the target scene is the same as the resolution of the first camera or the third camera.

[0135] The multi-frame zoom algorithm in S204 may refer to the multi-frame zoom algorithm in S104.

[0136] Scenario 3

[0137] See also Figure 7 .

[0138] S301, when the target zoom ratio is within the range of 3x~5x, the system needs to determine whether the target scene belongs to a dark light environment; if it belongs to a dark light environment, the terminal enables the second camera and the third camera. The determination of the dark light environment can refer to S201.

[0139] When the user adjusts the camera parameters, once the user sets the target zoom ratio in the range of [3, 5], the third camera (main camera) and the second camera (secondary camera) will be enabled, and the preview image will also change accordingly; at this time, the preview image is a part of the actual view image of the third camera, and the size of this part is determined by the target zoom ratio and the preset aspect ratio of the output image.

[0140] S302, when the photo function is triggered, the third camera and the second camera respectively continuously shoot their respective actual scene, and obtain m 2 Frame color image and m 1 Black and white image.

[0141] The width of the image actually captured by the third camera is recorded as w 2 , high is denoted as h 2 , w 2 h 2 is the resolution of the third camera. Since the resolution of the third camera is the same as that of the first camera, w 2 = w 0 ,h 2=h 0 In addition, m 2 Can also be used with m 0 The remaining steps and drawings are also written as w 0 、h 0 and m 0 To express.

[0142] The width of the image actually captured by the second camera is recorded as w 1 , high is denoted as h 1 , w 1 h 1 is the resolution of the second camera.

[0143] It should be understood that since the third camera and the second camera are fixed-focus lenses, the image actually captured also includes other content besides the preview image, and is not just the zoomed target scene that the user expects to capture in the preview image.

[0144] S303, m 0 The center area of ​​the frame color image is cropped, and for the image actually captured by the third camera, m 0 Frame 0 h 0 / (n / n 0 ) 2 Size of color image. 1 The central area of ​​the frame black and white image is cropped, and the image actually captured by the second camera is cut out. 1 Frame 1 h 1 / n 2 Size black and white image. Here n 0 Approximately equal to 3.

[0145] S304, the m obtained in S303 0 Frame 0 h 0 / (n / n 0 ) 2 The color image of size is zoomed multiple frames to obtain the color multi-frame zoom result, that is, 1 frame w 0 h 0 The color zoom image of size. And the m obtained in S303 1 Frame 0 h 0 / n 2 The black-and-white image of size is zoomed multiple frames to obtain the black-and-white multi-frame zoom result, that is, 1 frame w 0 h 0 / n 2 Size black and white zoom image.

[0146] The multi-frame zoom algorithm in S304 may refer to the multi-frame zoom algorithm in S104.

[0147] S305, the 1 frame w obtained in S304 0 h 0 Color zoom image of size and 1 frame w 0 h 0 / n 2 The size of the black and white zoom image is telephoto black and white fused; get a frame size of w 0 h 0 The color output image is the output image of the target scene. Here, the resolution of the output image of the target scene is the same as the resolution of the first camera or the third camera.

[0148] Among them, an optional telephoto black and white fusion algorithm process is as follows:

[0149] 1) Using the color zoom image corresponding to the telephoto lens (also called Tele image) as a reference, align the black and white zoom image to the Tele image; and obtain the motion area mask;

[0150] 2) Input the Tele image, the aligned black-and-white image and the motion region mask into the pre-trained convolutional neural network to obtain the fused result;

[0151] 3) Sharpen the fused result in 2).

[0152] Telephoto black and white fusion takes advantage of the higher resolution and clarity of Tele images, and the lower noise of black and white images, ensuring the image quality of high-magnification zoom scenes in low-light conditions, almost reaching a lossless level.

[0153] In the above algorithm flow, the processing methods involved can adopt mature algorithms in the prior art, such as fusion, alignment, sharpening, etc., which are not limited or elaborated in the present invention.

[0154] Scenario 4

[0155] See also Figure 8 .

[0156] S401, when the target zoom ratio is within the range of 5x~10x, the system needs to determine whether the target scene belongs to a dark light environment; if it does not belong to a dark light environment, the terminal enables the third camera. The determination of the dark light environment can refer to S201.

[0157] When the user adjusts the camera parameters, once the target zoom ratio set by the user is in the range of (5, 10], the third camera will be enabled, and the preview image will also change accordingly; at this time, the preview image is a part of the actual view image of the third camera, and the size of this part is determined by the target zoom ratio and the preset aspect ratio of the output image.

[0158] S402, when the photo function is triggered, the third camera continuously shoots the actual scene, and obtains m 2 Frame color image.

[0159] The width of the image captured by the third camera is recorded as w 2 , high is denoted as h 2 , w 2 h 2 is the resolution of the third camera. Since the resolution of the third camera is the same as that of the first camera, w 2 = w 0 ,h 2 =h 0 In addition, m 2 Can also be used with m 0 The remaining steps and drawings are also written as w 0 、h 0 and m 0 To express.

[0160] S403, m 0 The center area of ​​the frame color image is cropped, and for the image actually captured by the third camera, m 0 Frame 0 h 0 / (n / n 0 ) 2 Since the equivalent focal length of the third camera is 80mm, that is, a telephoto lens, at the same shooting distance, the image captured is larger than that of a standard lens; therefore, the central area is determined by the target zoom ratio and the equivalent focal length; here n 0 Approximately equal to 3 (80mm / 27mm).

[0161] S404, the above m 0 Frame 0 h 0 / (n / n 0 )2 The color image of size is zoomed multiple frames to obtain the color multi-frame zoom result, that is, 1 frame w 0 h 0 / (n / n 1 ) 2 The size of the color zoom image. Here n 1 Refers to the lossless zoom capability of the terminal camera system, that is, the maximum zoom ratio under lossless conditions; for example, 5x in this example, n 1 It is determined by the parameter performance of the entire camera system in the terminal and can be regarded as a constant.

[0162] The multi-frame zoom algorithm in S404 may refer to the multi-frame zoom algorithm in S104.

[0163] S405, the 1 frame w obtained in S404 0 h 0 / (n / n 1 ) 2 The color zoom image of size is digitally zoomed to obtain a frame w 0 h 0 The color zoom image of the target scene is the output image of the target scene. Here, the resolution of the output image of the target scene is the same as the resolution of the first camera or the third camera.

[0164] There are many methods of digital zoom, such as interpolation magnification, and commonly used methods include: bilinear, bicubic, Lanczos, etc. Digital zoom can only enlarge the image resolution to the target size, but cannot guarantee the image clarity and resolution, so compared with lossless zoom, it is considered to be a zoom with certain losses. However, it also expresses the camera's certain imaging capabilities.

[0165] Scenario 5

[0166] See also Fig. 9 .

[0167] S501, when the target zoom ratio is within the range of 5x~10x, the system needs to determine whether the target scene belongs to a dark light environment; if it belongs to a dark light environment, the terminal enables the second camera and the third camera. The determination of the dark light environment can refer to S201.

[0168] When the user adjusts the camera parameters, once the user sets the target zoom ratio in the range of (5, 10], the third camera (main camera) and the second camera (secondary camera) will be enabled, and the preview image will also change accordingly; at this time, the preview image is a part of the actual view image of the third camera, and the size of this part is determined by the target zoom ratio and the preset aspect ratio of the output image.

[0169] S502, when the photo function is triggered, the third camera and the second camera respectively continuously shoot their respective actual scene, and obtain m 2 Frame color image and m 1 Black and white image.

[0170] The width of the image actually captured by the third camera is recorded as w 2 , high is denoted as h 2 , w 2 h 2 is the resolution of the third camera. Since the resolution of the third camera is the same as that of the first camera, w 2 = w 0 ,h 2 =h 0 In addition, m 2 Can also be used with m 0 The remaining steps and figures are also written as w 0 、h 0 and m 0 To express.

[0171] The width of the image actually captured by the second camera is recorded as w 1 , high is denoted as h 1 , w 1 h 1 is the resolution of the second camera.

[0172] It should be understood that since the third camera and the second camera are fixed-focus lenses, the image actually captured also includes other content besides the preview image, and is not just the zoomed target scene that the user expects to capture in the preview image.

[0173] S503, m 0 The center area of ​​the frame color image is cropped, and for the image actually captured by the third camera, m 0 Frame 0 h 0 / (n / n 0 ) 2 Size of color image; m 1The center area of ​​the frame black and white image is cropped, and the image actually captured by the second camera is cut out. 1 Frame 1 h 1 / n 2 Size black and white image. Here n 0 Approximately equal to 3.

[0174] S504, the m obtained in S503 is 0 Frame 0 h 0 / (n / n 0 ) 2 The color image of size is zoomed multiple frames to obtain the color multi-frame zoom result, that is, 1 frame w 0 h 0 / (n / n 1 ) 2 The color zoom image of size. And the m obtained in S503 1 Frame 0 h 0 / n 2 The black-and-white image of size is zoomed multiple frames to obtain the black-and-white multi-frame zoom result, that is, 1 frame w 0 h 0 / (n / n 1 ) 2 Size black and white zoom image.

[0175] The multi-frame zoom algorithm in S504 may refer to the multi-frame zoom algorithm in S104.

[0176] S505, the 1 frame w obtained in S504 is 0 h 0 / (n / n 1 ) 2 Color zoom image of size and 1 frame w 0 h 0 / (n / n 1 ) 2 The size of the black and white zoom image is telephoto black and white fused; get a frame size of w 0 h 0 / (n / n 1 ) 2 The size of the color zoom image. Here n 1 Refers to the lossless zoom capability of the terminal camera system, that is, the maximum zoom ratio under lossless conditions; for example, 5x, n1 Determined by the performance parameters of the entire camera system in the terminal; can be regarded as a constant.

[0177] S506, the 1 frame w obtained in S505 0 h 0 / (n / n 1 ) 2 The color zoom image of size is digitally zoomed to obtain a frame w 0 h 0 The color zoom image of the target scene is the output image of the target scene. Here, the resolution of the output image of the target scene is the same as the resolution of the first camera or the third camera.

[0178] The digital zoom algorithm is an existing mature technology, and can be referred to in S405.

[0179] It should be understood that the above five situations are only some optional real-time methods in the present invention; and due to the differences in camera parameter design, algorithm implementation, user settings, terminal operating system, and terminal environment, the specific parameters mentioned above may have certain errors; in addition, the expression of some parameters is different due to different reference standards. The settings of specific parameters cannot be listed one by one by exhaustive enumeration. Those skilled in the art should understand that the present invention aims to obtain pictures with different lens combinations according to different zoom requirements of users, obtain the final picture according to the corresponding algorithm, and achieve the image quality in the entire large zoom range from 1 to 5x. Still remain lossless. If the maximum target magnification of the lossless condition is appropriately adjusted, those skilled in the art can use the lens combination method according to the embodiment of the present invention, adaptively change the lens parameters, or use different types of algorithms to achieve approximate lossless zoom. If the user allows a certain loss in the clarity of the zoomed image, or the terminal device allows the use of a larger telephoto lens, the zoom range and lens combination in the above embodiment can be adaptively adjusted based on the above theory, so as to obtain an image that meets the user's needs, and these deformed technical solutions should all fall within the scope of protection of the present invention.

[0180] It should also be understood that during use, users often use different zoom ranges in a short period of time in the process of focusing due to their actual needs; these changes in zoom ranges will directly cause changes in camera enablement. Taking the above five situations as an example, the specific enablement status of each camera can be referred to Figure 3The states of the three cameras under different zoom ranges in the image. For example, initially the target zoom ratio is 1.5, at which time the first camera and the second camera are turned on and the third camera is turned off. When the target zoom ratio is adjusted from 1.5 to 3.5, and the target scene is in dark light conditions, at this time, the third camera and the second camera are turned on and the first camera is turned off.

[0181] It should also be understood that in the above embodiment, the resolution of the output image is the same as the resolution of the first camera or the third camera, and is lower than the resolution of the second camera. In fact, the resolution of the output image should meet the user's requirements for clarity, and there is no strict equality between the first camera and the third camera. Usually, the first camera or the third camera represents the most basic imaging performance of the shooting terminal in different shooting modes; therefore, the maximum resolution of the output image is roughly equal to the resolution of the first camera or the third camera. Usually, when the terminal leaves the factory, the maximum resolution of the output image is basically determined, and the user can set the resolution of the output image in the camera system according to their needs.

[0182] In addition, during the specific implementation process, the camera system is also used to adjust the imaging parameters of the optical zoom module according to the zoom mode of the target scene, and the imaging parameters include at least one of the following parameters: noise reduction parameters, sharpening parameters, or contrast. To reduce noise, sharpen, modulate the contrast and dynamic range of the image in the intermediate process. For example, in a high-brightness scene, the ISP module will be controlled to turn off the noise reduction and sharpening modules, and in a low-light scene, the ISP module will be controlled to turn on the noise reduction and sharpening modules, and adjust the parameters to appropriate levels. In addition, since the contrast and dynamic range parameters in the zoom mode are different from those in the ordinary camera mode, the contrast and dynamic range parameters can also be adjusted in a targeted manner in different zoom modes. Therefore, the method of the embodiment of the present application can configure imaging parameters according to different scenes to ensure the imaging quality of the final image.

[0183] Through the present invention, a lossless zoom effect of approximately 5x can be achieved on a smart phone, and even in a dark environment, a good balance can be achieved between resolution and noise. A combination of multiple fixed-focus lenses is used without using bulky functional devices, so the thickness of the terminal will not be significantly increased, ensuring the beauty of the terminal, especially for smart handheld devices such as mobile phones; at the same time, it meets the user's requirements for a small, light and thin terminal and lossless imaging under large zoom, and improves the user experience.

[0184] Based on the shooting method provided in the above embodiment, the embodiment of the present invention provides a shooting device 700, which can be applied to various types of shooting equipment, such as Fig.10As shown, the device 700 includes an acquisition module 701, a determination module 702, a de-collection module 703, an image processing module 704, and a first camera, a second camera and a third camera; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, the resolution of the second camera is greater than the resolution of the first camera and the third camera, the first camera, the second camera and the third camera are all fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera; the relevant characteristics can refer to the description in the aforementioned method embodiment.

[0185] The acquisition module 701 is used to acquire the target zoom ratio. The acquisition module 701 can be acquired by the processor calling corresponding program instructions according to external input.

[0186] The determination module 702 is used to determine the target lens from the first camera, the second camera and the third camera according to the target zoom ratio. The determination module 702 can be called by the processor to selectively enable the control of the above three cameras through program instructions in the memory.

[0187] The acquisition module 703 is used to acquire an image containing a target scene using a target lens; the acquisition module 703 can be implemented by a processor, and the acquired image is stored in a memory.

[0188] The image processing module 704 is used to obtain an output image of the target scene based on the collected image containing the target scene. The image processing module 704 can be implemented by a processor, and can be implemented by calling data and algorithms in a local memory or a cloud server, performing corresponding calculations, and outputting a picture of the target scene that the end user expects to obtain.

[0189] In the specific implementation process, the acquisition module 701 is specifically used to execute the method mentioned in step 21 and the method that can be equivalently replaced; the determination module 702 is specifically used to execute the method mentioned in step 22 and the method that can be equivalently replaced; the acquisition module 703 is specifically used to execute the method mentioned in step 23 and the method that can be equivalently replaced; the image processing module 704 is specifically used to execute the method mentioned in step 24 and the method that can be equivalently replaced.

[0190] More specifically, at different target magnifications;

[0191] The acquisition module 701 and the determination module 702 may cooperate to execute the above method of S101, S201, S301, S401, or S501;

[0192] The acquisition module 703 may execute the above method of S102, S202, S302, S402, or S502;

[0193] The image processing module 704 may execute the above-mentioned methods of S103 - S105 , S203 - S204 , S303 - S305 , S403 - S405 , or S503 - S506 .

[0194] Among them, the above-mentioned specific method embodiments and the explanations, descriptions, and expansions of various implementation forms of the technical features in the embodiments are also applicable to the execution of the method in the device, and are not repeated in the device embodiments.

[0195] The present invention provides an image processing device 700. According to different zoom requirements, different camera combinations can be used for photo taking and image processing to achieve a lossless zoom effect of approximately 5x without using bulky components, thereby improving the user's aesthetic feeling for the terminal and the image quality requirements.

[0196] It should be understood that the division of the modules in the above device 700 is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. For example, the above modules can be separately established processing elements, or they can be integrated into a chip of the terminal for implementation. In addition, they can also be stored in the storage element of the controller in the form of program code, and called and executed by a processing element of the processor. The functions of the above modules. In addition, the modules can be integrated together or implemented independently. The processing element described here can be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method or the above modules can be completed by hardware integrated logic circuits in the processor element or software instructions. The processing element can be a general-purpose processor, such as a central processing unit (CPU), or one or more integrated circuits configured to implement the above method, such as one or more application-specific integrated circuits (ASICs), or one or more microprocessors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0197] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0201] Although some embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the enumerated embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. If these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also includes these modifications and variations.

[0202] Image super-resolution reconstruction is an image processing technology that reconstructs a high-resolution image based on one or more low-resolution images. It is widely used in satellite images, medical imaging, video surveillance and other fields. Image super-resolution algorithms can be used as an image preprocessing technology to improve subsequent image recognition, detection or higher-level semantic image analysis; they can also be used as an image enhancement technology to improve image resolution or more appropriate contrast.

[0203] Unlike ordinary image super-resolution, face super-resolution image enhancement technology can use the statistical characteristics of the image to perform differentiated image enhancement processing. Face statistical characteristics include: the position structure of organs such as eyebrows, eyes, nose, mouth, ears, skin, etc. on the face, as well as the gray value distribution, color value distribution, texture characteristics, etc. of each organ.

[0204] Another aspect of the present invention provides a method and apparatus for image enhancement based on a neural network.

[0205] The method comprises:

[0206] The terminal device obtains the initial image;

[0207] In a possible implementation, the terminal device may be a smart phone, a tablet computer, or a wearable device.

[0208] In a possible implementation, a camera of a terminal device may be used to capture images, or the terminal device may receive images from other devices.

[0209] Performing face analysis on the initial image obtained by the terminal device to obtain a face image;

[0210] In a possible implementation, the obtained initial image is upsampled to increase the resolution of the initial image and obtain an upsampled image.

[0211] Perform face detection on the upsampled image;

[0212] In a possible implementation, the face position can be determined through face detection, and thus the face image can be extracted. Detecting the face in the image to obtain the face image can refer to the relevant regulations in the prior art, which will not be elaborated here.

[0213] Obtain a face mask according to the face image;

[0214] In a possible implementation, a face mask composed of various organs can be formed by parsing each face organ on the face image. In one example, the face mask can be as Fig.11 shown.

[0215] Enhance the resolution of the face image according to the face mask;

[0216] In a possible implementation, extract the Y-channel map of the face image, input the face mask and the Y-channel map of the face image into a pre-trained deep neural network, and obtain the image with enhanced resolution.

[0217] In a possible implementation, extract the face Y-channel map in the detected face image, linearly map the face Y-channel map with the value range [0, 255] to the range between [-1, 1], and perform face super-resolution image enhancement on the Y-channel map and the face mask map based on the learned deep neural network to obtain the result with enhanced resolution, and then linearly map the result map with the value range [-1, 1] to the range between [0, 255].

[0218] In a possible implementation, as Fig.12 shown, the trained deep neural network can be obtained through the following method:

[0219] Perform degradation processing on the high-definition face image to obtain a high-definition - low-definition image pair for training,

[0220] Downsample the high-definition face to obtain a low-resolution face image,

[0221] Apply degradation processing and then upsample to obtain a low-definition face image,

[0222] Parse the high-definition face image to obtain a face mask composed of various organs,

[0223] Train a face super-resolution image enhancement model based on the high-definition - low-definition face pair and the face mask,

[0224] Design a generator network G and a discriminator network D for confrontation,

[0225] Alternately train G and D, and finally obtain the generator model.

[0226] A final output image is obtained according to the initial image, the face image, and the image after resolution enhancement.

[0227] In one possible implementation, the original face image and the network result are fused to obtain a final result, which is then incorporated into the original image captured by the terminal.

[0228] In one possible implementation, after fusing the original image and the network result, pseudo-textures such as checkered patterns are suppressed, and brightness correction can be performed on the suppressed image to eliminate brightness changes in the result, and then the brightness-corrected result is merged into the original image captured by the terminal.

[0229] In a possible implementation, specifically, the original image is used as a guide graph for the network result, and a guide filter is performed on the network result to eliminate the checkered pattern in the network result. The brightness histogram of the result graph of the previous step is mapped to the brightness histogram of the input Y channel to ensure that the brightness of the result and the input are consistent, and the result of the previous step is merged into the original image captured by the terminal.

[0230] In a possible implementation, the neural network-based image enhancement method described in the embodiment of the present invention can be as follows: Fig.13 Process as shown.

[0231] The embodiment of the present invention provides a terminal super-resolution image enhancement method and device based on a deep neural network, and a neural network designed for super-resolution image enhancement based on a face mask and a Y-channel face image: a network trained using a face mask, a Y-channel face image, and high-definition and low-definition image pairs can utilize the prior characteristics of the face image to perform specific super-resolution and other image enhancements on various parts of the face; and an image fusion mechanism for suppressing pseudo-textures: utilizing the characteristic of the original image without pseudo-textures to fuse with the network results to suppress the pseudo-textures of the network.

[0232] While the resolution of facial images is enhanced, the noise in the main organs of the face is suppressed, the details are increased, the contrast and sharpness are improved, and the pseudo-texture is also suppressed.

[0233] Because a network is set up to input low-definition faces and face masks in the Y channel and output high-definition face images, this setting allows the network to learn the different change processes of each face area under the guidance of the face mask, so that regional and targeted enhancement of face images can be achieved on the terminal device; because a weighted fusion mechanism of network results, input faces, and original images is set up, this processing can take into account multi-dimensional information at the same time, so that enhancement effects such as image detail enhancement, contrast enhancement, and blur suppression can be achieved while effectively suppressing pseudo textures.

[0234] The neural network-based image enhancement device of the embodiment of the present invention includes a processor and a storage medium, and can execute the method described in the above embodiment, which will not be described in detail here.

[0235] Another aspect of the present invention discloses a method for image processing based on a deep neural network, the method comprising:

[0236] The deep neural interpolation network is used to construct the low-frequency structural information of multiple frames, and then the original image structural information is combined with the low-definition and high-definition image pairs to learn rich high-frequency information. By adding low-frequency structures, the super-resolution neural network focuses on the learning of high-frequency sparse information, so that the network's restoration results have achieved very good results in sharpness, contrast, resolution and noise, while minimizing the generation of interference.

[0237] In one possible implementation, a deep interpolation network is trained by constructing low-resolution-high-resolution image pairs.

[0238] In a possible implementation, the ghost mask of multiple frames is randomly moved by 1 pixel, the pixel information of the reference image is obtained to replace the pixel information of the current frame image, and a noise random number that conforms to the Gaussian distribution is superimposed on the pixel.

[0239] In a possible implementation, for extracting structural information of multiple frames of original images: 1) using the Laplacian operator to extract edge structural information; 2) using wavelet transform to extract high-frequency structural information.

[0240] Multi-frame super-resolution restoration method based on deep neural network.

[0241] High frequency detail extraction.

[0242] By simulating the terminal multi-frame imaging mechanism, the generated training data set can effectively cover the actual data distribution and keep the data form of training and testing consistent, so the deep neural network can very effectively learn the mapping relationship between data, thereby enhancing image details and reducing noise. Because the mean image path is added to the network, the entire network focuses more on the high-frequency differences between training images during the learning mapping process, so the entire network can learn more efficiently. Because the high-frequency information is extracted from the mean image and fused with the network output, the loss of high-frequency information can be effectively avoided, making the output result more detailed.

Claims

1. A shooting method, characterized in that, the method is applied to a terminal, and the terminal includes a first camera, a second camera, and a third camera; the method includes: obtaining a target zoom ratio; when the target zoom ratio is lower than a preset zoom ratio, using the first camera and the second camera to respectively capture a color image and a black-and-white image including a target scene; obtaining a target image according to the color image and the black-and-white image respectively captured by the first camera and the second camera; when the target zoom ratio is not lower than the preset zoom ratio, at least using the third camera to capture a color image including the target scene; obtaining a target image at least according to the color image captured by the third camera; wherein, the first camera and the third camera are color cameras, the second camera is a black-and-white camera, and the first camera, the second camera, and the third camera are all cameras with fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera.

2. The method according to claim 1, characterized in that, when the target zoom ratio is not lower than the preset zoom ratio, at least using the image of the third camera including the target scene includes: when the target zoom ratio is not lower than the preset zoom ratio and when the illuminance of the target scene is lower than a preset threshold, using the third camera and the second camera to respectively capture a color image and a black-and-white image including the target scene; the obtaining a target image at least according to the image captured by the third camera includes: obtaining the target image according to the color image and the black-and-white image respectively captured by the third camera and the second camera.

3. The method according to claim 1, characterized in that, the obtaining a target image according to the color image and the black-and-white image respectively captured by the first camera and the second camera includes: for the color image and the black-and-white image including the target scene, performing central cropping, multi-frame zooming, and image fusion to obtain an output image of the target scene.

4. The method according to claim 2, characterized in that, the obtaining a target image according to the color image and the black-and-white image respectively captured by the third camera and the second camera includes: for the color image and the black-and-white image including the target scene, performing central cropping, multi-frame zooming, and image fusion to obtain an output image of the target scene.

5. The method according to claim 2, characterized in that, the obtaining a target image according to the color image and the black-and-white image respectively captured by the third camera and the second camera includes: for the color image and the black-and-white image including the target scene, performing central cropping, multi-frame zooming, digital zooming, and image fusion to obtain an output image of the target scene.

6. The method according to any one of claims 1-5, characterized in that, the aperture value of the second camera is 1.55 or 1.65, and the aperture value of the third camera is 2.4 or 2.

2.

7. The method according to any one of claims 1-5, characterized in that, The resolution of the second camera is 20M or 24M, and the resolution of the third camera is 8M, 10M or 12M.

8. The method according to any one of claims 1 to 5, It is characterized in that The resolution of the second camera is greater than the resolution of the third camera.

9. The method according to any one of claims 1 to 5, It is characterized in that The aperture of the second camera is larger than the aperture of the third camera.

10. The method according to any one of claims 1 to 5, It is characterized in that The equivalent focal length of the third camera is 2-4 times the equivalent focal length of the second camera.

11. A photographing device, It is characterized in that The device is applied to a terminal, the terminal includes a first camera, a second camera and a third camera; the device includes: An acquisition module is used to obtain a target zoom ratio; A collection module, used for collecting a color image and a black-and-white image containing a target scene using the first camera and the second camera respectively when the target zoom ratio is lower than a preset zoom ratio; An image processing module, used to obtain a target image based on the color image and the black-and-white image collected by the first camera and the second camera respectively; The acquisition module is further configured to acquire a color image containing a target scene using at least the third camera when the target zoom ratio is not less than the preset zoom ratio; the image processing module is further configured to obtain a target image based on at least the color image acquired by the third camera; Among them, the first camera and the third camera are color cameras, the second camera is a black and white camera, the first camera, the second camera and the third camera are all cameras using fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera.

12. The device according to claim 11, It is characterized in that When the target zoom ratio is not lower than the preset zoom ratio, and when the illumination of the target scene is lower than a preset threshold, the acquisition module is specifically used to use the third camera and the second camera to respectively acquire a color image and a black-and-white image containing the target scene; the image processing module is also used to obtain the target image based on the color image and the black-and-white image acquired by the third camera and the second camera respectively.

13. The device according to claim 11, It is characterized in that The image processing module is specifically used to obtain the output image of the target scene by adopting center cropping, multi-frame zooming, and image fusion for the color image and the black-and-white image respectively collected by the first camera and the second camera.

14. The device according to claim 12, It is characterized in that The image processing module is specifically used to obtain the output image of the target scene by adopting center cropping, multi-frame zooming and image fusion for the color image and the black-and-white image respectively collected by the third camera and the second camera.

15. The device according to claim 12, It is characterized in that The image processing module is specifically used to obtain the output image of the target scene by adopting center cropping, multi-frame zoom, digital zoom, and image fusion for the color image and black-and-white image respectively collected by the third camera and the second camera.

16. The device according to any one of claims 11 to 15, It is characterized in that The aperture value of the second camera is 1.55 or 1.65, and the aperture value of the third camera is 2.4 or 2.

2.

17. The device according to any one of claims 11 to 15, It is characterized in that The resolution of the second camera is 20M or 24M, and the resolution of the third camera is 8M, 10M or 12M.

18. The device according to any one of claims 11 to 15, It is characterized in that The resolution of the second camera is greater than the resolution of the third camera.

19. The device according to any one of claims 11 to 15, It is characterized in that The aperture of the second camera is larger than the aperture of the third camera.

20. The device according to any one of claims 11 to 15, It is characterized in that The equivalent focal length of the third camera is 2-4 times the equivalent focal length of the second camera.

21. A terminal device, It is characterized in that The terminal device includes a memory, a processor, a bus, a first camera, a second camera and a third camera; the memory, the first camera, the second camera, the third camera and the processor are connected via the bus; wherein the first camera and the third camera are color cameras, the second camera is a black and white camera, and the first camera, the second camera and the third camera are cameras using fixed-focus lenses; and the equivalent focal length of the third camera is greater than the equivalent focal lengths of the first camera and the second camera; The third camera is used to collect color image signals under the control of the processor; The second camera is used to collect black and white image signals under the control of the processor; The memory is used to store computer programs and instructions; The processor is used to call the computer program and instructions stored in the memory to execute the method according to any one of claims 1-10.

22. The terminal device as described in claim 21 further comprises an antenna system, and the antenna system, under the control of the processor, sends and receives wireless communication signals to realize wireless communication with a mobile communication network; the mobile communication network comprises one or more of the following: 3G network, 4G network, 5G network, WIFI network.

Images