Shooting control method and device, equipment and storage medium

Abstract

The invention discloses a shooting control method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining the depth information of a current scene under the condition that a first camera is started for preview or recording; performing image region division based on the depth information of the current scene to obtain position information of a plurality of image regions; based on the position information of the plurality of image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, performing differentiated multi-camera consistency processing on the first camera, and determining 3A control parameters of the first camera; and performing shooting control on the first camera based on the 3A control parameter of the first camera. Therefore, the interested area and the non-interested area of the user are identified based on the depth information, reasonable processing resource allocation is performed according to different image areas, the consistency processing effect of the interested area is improved, the processing power consumption of the non-interested area is reduced, and the effect and the power consumption of a multi-shooting consistency scheme are better balanced.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a shooting control method and apparatus, equipment, storage medium, and computer program product. Background Art

[0002] Multi-camera consistency refers to the ability of two or more cameras in a multi-camera system to maintain consistency in terms of image quality, color, exposure, etc. At present, the implementation of multi-camera consistency is mainly to use the 3A statistics (3Astatistics or 3A stats) of the secondary camera to correct the 3A statistics of the main camera, reduce the differences between different cameras, and improve the consistency of imaging effects. However, since consistency processing will increase system power consumption, some consistency effects are often sacrificed to avoid the terminal battery dropping too quickly. Therefore, how to better balance the effect and power consumption has become an important development direction for multi-camera consistency solutions. Summary of the invention

[0003] The embodiments of the present application are intended to provide a shooting control method and apparatus, device, storage medium, and computer program product.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, a shooting control method is provided, comprising:

[0006] When the first camera is started for preview or recording, depth information of the current scene is obtained;

[0007] Divide the image area based on the depth information of the current scene to obtain position information of multiple image areas;

[0008] Get the 3A statistics information of the first camera and the 3A statistics information of the second camera;

[0009] Based on the position information of the multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, performing differentiated multi-camera consistency processing on the first camera to determine the 3A control parameters of the first camera;

[0010] Based on the 3A control parameter of the first camera, the first camera is controlled to shoot.

[0011] In a second aspect, a shooting control device is provided, comprising:

[0012] An acquisition unit, configured to acquire depth information of a current scene when the first camera is started for preview or recording;

[0013] A first processing unit, configured to divide the image region based on the depth information of the current scene to obtain position information of multiple image regions;

[0014] The acquiring unit is further used to acquire the 3A statistical information of the first camera and the 3A statistical information of the second camera;

[0015] The second processing unit is further used to perform differentiated multi-camera consistency processing on the first camera based on the position information of the multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, and determine the 3A control parameters of the first camera;

[0016] The control unit is used to control the shooting of the first camera based on the 3A control parameters of the first camera.

[0017] In a third aspect, a shooting control device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0018] Wherein, when the processor is configured to run a computer program, the steps of the aforementioned method are executed.

[0019] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0020] In a fifth aspect, a computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0021] In the embodiment of the present application, a shooting control method and device, equipment, storage medium, and computer program product are provided, the method comprising: obtaining the depth information of the current scene when the first camera is started for preview or recording; dividing the image area based on the depth information of the current scene to obtain the position information of multiple image areas; obtaining the 3A statistical information of the first camera and the 3A statistical information of the second camera; based on the position information of multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, performing differentiated multi-camera consistency processing on the first camera, and determining the 3A control parameters of the first camera; and controlling the shooting of the first camera based on the 3A control parameters of the first camera. In this way, the user's area of ​​interest and area of ​​non-interest are identified based on the depth information, and reasonable processing resource allocation is performed according to different image areas, so as to improve the consistency processing effect of the area of ​​interest, reduce the processing power consumption of the area of ​​non-interest, and better balance the effect and power consumption of the multi-camera consistency solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1This is a schematic diagram of a first process flow of a shooting control method in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of an image region division result in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of different statistical densities of an image area in an embodiment of the present application;

[0025] Figure 4 This is a second flow chart of the shooting control method in the embodiment of the present application;

[0026] Figure 5 Schematic diagram of the image processing flow of the main and secondary cameras in an embodiment of the present application;

[0027] Figure 6 Schematic diagram of the third process of the shooting control method in the embodiment of the present application;

[0028] Figure 7 This is a schematic diagram of the composition structure of the shooting control device in the embodiment of the present application;

[0029] Figure 8 It is a schematic diagram of the composition structure of the shooting control device in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0031] Before further describing the embodiments of the present disclosure in detail, the camera module is described as follows:

[0032] In the current electronic device camera lens support solution, there are generally 3 to 5 different types of lens support. For example, the camera operation page of a multi-camera mobile phone will have 0.6x, 1x, 2x, 3x, 5x, 10x, and 30x focal length switching options. The smaller the number, the wider the picture, and the larger the number, the farther the picture, which roughly corresponds to the focal lengths of the ultra-wide-angle lens (also called ultra-wide-angle camera, referred to as ultra-wide angle), the main camera lens (also called main camera, referred to as main camera), and the telephoto lens (also called telephoto camera, referred to as telephoto).

[0033] Wide-angle lens (Wide), the focal length of a wide-angle lens is close to the field of view of "what the human eye sees", what you see is what you shoot, and it is also the camera we use most frequently. Open the operation page of the mobile phone camera, you can see the 1x value display on the display page. 1x is the focal length of the main camera. This is the best image quality among all focal lengths, suitable for taking portraits, architecture, scenery, documentaries, etc.

[0034] Ultra Wide: Ultra Wide provides a wider field of view than the main camera. It should be noted that the focal length of the ultra wide angle is displayed differently on some mobile phone cameras. For example, it is displayed as 0.5x on the phone of manufacturer 1, 0.6x on the phone of manufacturer 2, and directly displayed as the text wide angle on the phone of manufacturer 3. Compared with the main camera, the ultra wide angle can capture a wider picture, which is suitable for taking pictures of scenery and architecture, and can obtain a more impactful picture.

[0035] In addition, the ultra-wide angle has a large viewing range, and you can get a wider picture at the same position, which brings great convenience to the later re-cropping. When shooting scenery, using an ultra-wide angle lens can not only allow more scenery elements to be captured, but also reflect the depth and sense of space of the picture as long as the composition is used, making the entire scenery photo look magnificent.

[0036] Telephoto, focal lengths above 1x, 2x, 3x or 5x are usually called telephoto. The higher the coefficient in front of x, the farther you can shoot. For example, the 5x optical zoom on the phone of manufacturer 3, when the focal length is switched to 5x, you can switch from the original main camera to the telephoto lens for shooting. The telephoto lens can take higher quality photos from a distance, shoot objects farther away or enlarge objects in the picture without causing a decrease in image quality like digital zoom. When it is inconvenient to move around and want to frame in a messy building, it is difficult to capture the subject with the main camera of the phone alone. At this time, using the telephoto lens of the phone to shoot can make ordinary photos more layered.

[0037] Telephoto lenses can "close" the distance between the background and the foreground, thus creating a sense of compression in the spatial distance, making the overall picture more substantial. This "sense of compression" is one of the characteristics of telephoto lenses. Telephoto lenses have little distortion and weak perspective effects, which can close the distance between the foreground and the background, enhance the relationship between the foreground and the background, and thus create some unique visual effects. This feature can also be used to use straight-extending scenery such as roads and railings as guide lines to draw the audience's attention to the subject in the depth of the picture.

[0038] Multi-camera consistency: during the manufacturing of a camera module containing multiple cameras, the imaging effect of each camera will be different due to factors such as assembly tolerance and incoming material differences. For key parameters that affect the shooting effect, such as white balance and lens shading correction, the terminal usually only saves a set of software parameters to adapt to the entire camera module. The differences in the above-mentioned individual cameras will also cause differences in the shooting effects of different cameras. For example, in the same environment, the shooting effects of different cameras may be bright or dark, and in severe cases, there may be color cast. Therefore, current multi-camera mobile phones are usually equipped with a multi-camera consistency (such as three-camera consistency) solution, which will bring the imaging effects or imaging styles (such as dynamic range, color tendency, etc.) of multiple cameras in the camera module closer to one of the cameras, so that the images captured by multiple cameras can show consistent image effects.

[0039] Spatial Alignment Transform (SAT), in current camera shooting solutions, SAT is used to switch lenses according to user needs in different shooting scenes, and continuous smooth zoom is achieved by integrating digital zoom. In other words, SAT is a processing operation to achieve smooth zoom, and its purpose is to fuse two or more images from different image sensors into one frame for output.

[0040] Fallback is a SAT zoom that is automatically triggered by the camera itself under certain conditions. SAT zoom can be triggered by user operations such as point-cut or swipe-cut, or by fallback. For example, fallback can include SAT zoom triggered according to the distance of the shooting scene and the brightness of the shooting environment. Taking 5x zoom as an example, the main camera should be on the telephoto lens at this time, but if the current environment is a macro environment (the distance between the camera and the subject is less than or equal to the preset distance), the telephoto lens cannot meet the shooting requirements of the current environment, and the system will automatically switch to other suitable cameras (such as wide-angle lenses or macro lenses). In other words, when the main camera cannot work properly due to malfunction, insufficient light, inappropriate focus distance, etc., the fallback mechanism will be triggered and switched to other cameras.

[0041] 3A is the general term for Auto Exposure (AE), Auto White Balance (AWB) and Auto Focus (AF).

[0042] At present, multi-lens functional configuration (such as ultra-wide angle + wide angle + telephoto) has become the mainstream of camera development. How to achieve smooth switching between multiple lenses and consistency of imaging effects has become an important development direction.

[0043] In the consistency solution, the 3A statistical information of the images collected by the secondary camera is extracted from the entire area, and then combined with the 3A statistical information of the main camera for multi-camera consistency processing. Although this can meet the consistency processing requirements of the 3A effect, it has great limitations on the processing time and power consumption of each frame (generally speaking, the larger the amount of data, the longer the processing time and the higher the power consumption). Due to the limitations of power consumption and performance, some effect experience can only be sacrificed to avoid rapid battery drain and overheating of the phone.

[0044] Based on this, the embodiment of the present application provides a shooting control method, which may specifically refer to a differentiated multi-camera consistency processing method based on depth information. In a shooting mode or focal length with a multi-camera consistency function, the user's area of ​​interest and area of ​​non-interest are identified based on depth information, and reasonable processing resources are allocated according to different image areas to improve the consistency processing effect of the area of ​​interest, reduce the processing power consumption of the area of ​​non-interest, and better balance the effect and power consumption of the multi-camera consistency solution.

[0045] In order to facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0046] Figure 1 Schematic diagram of the first process of the shooting control method in the embodiment of the present application. Figure 1 As shown, the method may specifically include:

[0047] Step 101: When the first camera is started for preview or recording, depth information of the current scene is obtained;

[0048] In a shooting mode or focal length with a multi-camera consistency function, the first camera is started for preview or recording, and the image data collected by the first camera is processed to generate a display image. The first camera can be used as a main camera (referred to as the main camera), for example, the first camera can be an ultra-wide angle or a telephoto.

[0049] In a shooting mode or focal length with a multi-camera consistency function, the second camera acts as a reference camera and provides standard 3A statistical information (including one or more of AE statistical information, AWB statistical information, and AF statistical information). Based on the 3A algorithm, the 3A statistical information of the first camera is brought closer to the reference camera, so that the images captured by multiple cameras reflect a consistent image effect. For example, when previewing or recording, the first camera acts as the main camera, the second camera acts as the secondary camera (referred to as the secondary camera), and the second camera provides standard 3A statistical information to perform multi-camera consistency processing on the first camera. For another example, when performing SAT or fallback zoom, the first camera acts as the main camera, the second camera acts as the secondary camera, and the second camera provides standard 3A statistical information to perform multi-camera consistency processing, and at the same time provides image data to be fused with the image data of the second camera to generate a display image.

[0050] That is to say, in a shooting mode or focal length with a multi-camera consistency function, the first camera and the second camera stream simultaneously, but for image display, only the image data output by the first camera can be processed and displayed, while the image data output by the second camera can be ignored, and only the 3A statistical information output by the second camera is used to perform multi-camera consistency processing on the first camera, thereby reducing the difference in image effects between different cameras and improving the consistency of effects. The image data output by the first camera and the image data output by the second camera can also be processed and displayed to achieve smooth zooming.

[0051] Step 102: Divide the image area based on the depth information of the current scene to obtain position information of multiple image areas;

[0052] The depth information of the current scene represents the depth (distance) between the camera and the visible surface of the object in the current scene. The depth information of the current scene can be the pixel values ​​contained in the depth image of the current scene, and each pixel value represents the depth from the camera to each point in the current scene. According to the difference in the depth from the camera to each point in the current scene, the image of the current scene is divided into image areas with different depth ranges, and then different image areas are processed differently to better balance the effect and power consumption.

[0053] In some embodiments, multiple image areas include at least a user-interested area and a non-interested area. The interested area refers to an area in an image or video that is designated as requiring special attention or processing. The interested area may include one or more areas, and the non-interested area may include one or more areas.

[0054] In some embodiments, the plurality of image regions are regular regions or irregular regions.

[0055] Exemplarily, if the multiple image areas are rectangular image areas, the image area division method may include: determining the area of ​​interest of the current scene based on the depth information of the current scene and a preset depth range; dividing the image area based on the area of ​​interest to determine the position information of the multiple image areas.

[0056] The preset depth range is used to characterize the depth characteristics of the region of interest. The preset depth range may specifically be a value range of the depth of the region of interest or a depth variation range.

[0057] The position information is used to indicate the image segmentation result, and the position information at least includes coordinate information of one or more specific pixels of the image area, and the specific pixel is used to locate the corresponding image area. The specific pixel may include multiple edge pixels of the image area, and the specific pixel may also include the upper left corner pixel or the center pixel of the image area. The position information may also include the size of the image area.

[0058] The region of interest can be understood as the area in the current scene that the user focuses on. These areas are usually the parts of the image that the user is most interested in or most likely to contain important information, and the image quality of these areas needs to be prioritized. Exemplarily, the region of interest may include a focal plane area or an area where a specific photographed object is located.

[0059] In some embodiments, performing image region division based on the region of interest and determining the position information of the multiple image regions may include: taking the bounding rectangle of the region of interest as the first image region and taking the region outside the first image region as the second image region; or, dividing the region outside the first image region into two or more image regions. The position information may include the coordinates and size of the upper left corner pixel of the image region.

[0060] In some embodiments, the plurality of image regions include: a focal plane region, a secondary focal plane region, and a background region.

[0061] The focal plane area refers to the area on the imaging plane where a clear image is formed after adjusting the lens focal length and aperture size. It specifically refers to the plane where the camera's photosensitive device (such as CMOS or CCD image sensors) is located, and is the key position for recording images. Objects on the focal plane area can obtain the clearest image in the image, and objects within a certain depth of field before and after the focal plane can also be clearly imaged, while objects in other positions may appear blurred. The focal plane area can be used as a user's area of ​​interest.

[0062] Methods for determining the focal plane area may include: (1) focal plane estimation based on depth distribution, analyzing the depth distribution in the depth image to find areas where the depth values ​​are relatively concentrated or have small variations. These areas are defined as focal plane areas, i.e., areas of interest to the user. (2) using depth threshold estimation, the focal plane area may correspond to a set of pixels whose depth values ​​fall within a certain range.

[0063] Figure 2 is a schematic diagram of an image region division result in an embodiment of the present application, such as Figure 2 As shown, the image of the current scene is divided into a focal plane area, a secondary focal plane area, and a background area, wherein the focal plane area refers to the circumscribed rectangular area of ​​the actual focal plane area, the position information of the focal plane area includes the coordinates of point A (x1, y1) and H1×W1, the position information of the secondary focal plane area includes the coordinates of point B (x2, y2) and H2×W2, and the position information of the background area includes the coordinates of point C (0, 0) and H3×W3. Since the background area is the area of ​​the entire image except the focal plane area and the secondary focal plane area, only the position information of the focal plane area and the position information of the secondary focal plane area can be provided to divide the entire image into three image areas.

[0064] Step 103: Obtain 3A statistical information of the first camera and 3A statistical information of the second camera;

[0065] In some embodiments, obtaining the 3A statistical information of the first camera and the 3A statistical information of the second camera may include: performing differentiated 3A statistical information processing on the first camera based on the position information of the multiple image areas to determine the 3A statistical information of the first camera; performing differentiated 3A statistical information processing on the second camera based on the position information of the multiple image areas to determine the 3A statistical information of the second camera.

[0066] In some embodiments, obtaining the 3A statistical information of the first camera and the 3A statistical information of the second camera may include: performing full-area 3A statistical information processing on the first camera to determine the 3A statistical information of the first camera; and performing differentiated 3A statistical information processing on the second camera based on position information of the multiple image areas to determine the 3A statistical information of the second camera.

[0067] In some embodiments, obtaining the 3A statistical information of the first camera and the 3A statistical information of the second camera may include: performing differentiated 3A statistical information processing on the first camera based on the position information of the multiple image areas to determine the 3A statistical information of the first camera; performing full-area 3A statistical information processing on the second camera to determine the 3A statistical information of the second camera.

[0068] In some embodiments, obtaining the 3A statistical information of the first camera and the 3A statistical information of the second camera may include: performing 3A statistical information processing on the entire area of ​​the first camera to determine the 3A statistical information of the first camera; and performing 3A statistical information processing on the entire area of ​​the second camera based on the position information of the multiple image areas to determine the 3A statistical information of the second camera.

[0069] Differentiated 3A statistical information processing refers to using different processing methods for 3A statistical information of different image areas.

[0070] In some embodiments, the multiple image areas include at least a first image area and a second image area. Differentiated 3A statistical information processing may include: performing statistical density allocation based on the position information of the multiple image areas, determining a first statistical density of the first image area, and a second statistical density of the second image area, wherein the first statistical density is greater than the second statistical density; using a target acquisition process, based on the first statistical density, acquiring 3A statistical information of a target camera in the first image area, wherein the target camera is the first camera or the second camera; using a target acquisition process, based on the second statistical density, acquiring 3A statistical information of a target camera in the second image area.

[0071] The first image area is an area that the user is more interested in than the second image area, that is, the user has a higher requirement for the consistency effect of the first image area. For example, the first image area is a focal plane area, and the second image area is a non-focal plane area; for another example, the first image area is a focal plane area, and the second image area is a secondary focal plane area; for another example, the first image area is a secondary focal plane area, and the second image area is a background area.

[0072] Statistical density can be understood as the amount of 3A statistical information collected in a unit area. The greater the statistical density, the more 3A statistical information is collected, that is, the amount of data for subsequent consistency processing is greater, and accordingly, the processing effect is better. The smaller the statistical density, the less 3A statistical information is collected, that is, the amount of data for subsequent consistency processing is less, and accordingly, the processing speed is faster and the processing power consumption is lower. Statistical density can also be understood as the number of statistical areas into which a specific image area is divided. The greater the statistical density, the more statistical areas there are, and the smaller the statistical density, the fewer statistical areas there are.

[0073] Exemplarily, the statistical density includes 16*16, 32*32, 64*64, 128*128, etc. Figure 3 is a schematic diagram of different statistical densities of an image region in an embodiment of the present application, such as Figure 3As shown, different image regions are divided into multiple statistical regions according to different statistical densities, and the 3A statistical information of a specific pixel (such as a central pixel) is extracted in each statistical region as the 3A statistical information of the statistical region, and the 3A statistical information of all statistical regions constitutes the 3A statistical information of the image region. The statistical density is related to the size of the statistical region. The larger the statistical density, the smaller the size of the statistical region, and the more 3A statistical information is extracted in the unit region. The smaller the statistical density, the larger the size of the statistical region, and the less 3A statistical information is extracted in the unit region. For example, the statistical density of the focal plane region is 128*128, the statistical density of the secondary focal plane region is 64*64, and the statistical density of the background region is 16*16 or 32*32.

[0074] In some embodiments, the differentiated 3A statistical information processing may include: when the current image captured by the target camera for the current scene meets the first condition, using the target acquisition process, based on the first statistical density, to obtain the 3A statistical information of the target camera in the first image area; using the 3A statistical information of the second image area in the reference image of the current image as the 3A statistical information of the target camera in the second image area. The reference image may be an image that performs full-area statistics before the current image or an image that performs differentiated statistics on the second image area.

[0075] It should be noted that when the differentiated 3A statistical information processing method is used to perform differentiated 3A statistical information processing on the first camera, the target camera is the first camera, and when the differentiated 3A statistical information processing method is used to perform differentiated 3A statistical information processing on the second camera, the target camera is the second camera.

[0076] In some embodiments, differentiated 3A statistical information processing may include: when the current image satisfies the second condition, using a target acquisition process to acquire 3A statistical information of the target camera in the entire image area based on a third statistical density.

[0077] In some embodiments, differentiated 3A statistical information processing may include: when the current image satisfies the second condition, based on the first statistical density, obtaining the 3A statistical information of the target camera in the first image area; based on the second statistical density, obtaining the 3A statistical information of the target camera in the second image area.

[0078] The first condition can be understood as a condition that triggers high-density statistics only for the first image area, and the second condition can be understood as a condition that triggers full-area statistics for the entire image, or a condition that performs differentiated statistics for the entire image. Taking into account the image features of the current scene, the consistency requirements of different image areas, and the sensor outflow features, different 3A statistical information processing methods are accurately configured for different images by setting the first condition or the second condition, further optimizing the balance between power consumption and effect.

[0079] In some embodiments, the image satisfying the first condition is a differentially processed image, and the image satisfying the second condition is a full-area processed image. For example, the image sensor outputs an image sequence of X={x1, x2, ..., x9, x 10 ,…}, the differentially processed images include x9 to x 9+T , the full area processing image includes x 9+T+1 That is, the whole area is processed once every T frames, where T is a positive integer, such as 1, 3, 5, 7, etc. For other images, only the 3A information of the first image area is counted frame by frame, and the reference image (such as x 9+T+1 ), and then allocate a small amount of processing resources to the second image area for multi-camera consistency processing, so that more processing resources can be allocated to the area of ​​interest to the user. In this way, on the basis of obtaining objective and true 3A statistical information of the current scene, the consistency effect of the first image area is guaranteed, and the power consumption of the system for differential processing of the second image area is reduced, further optimizing the balance between power consumption and effect.

[0080] In some embodiments, the third statistical density is different from the first statistical density and the second statistical density. In some embodiments, the third statistical density is the same as the first statistical density, and using a high statistical density for full-area statistics can obtain more 3A statistical information and improve the subsequent consistency processing effect.

[0081] In some embodiments, the target acquisition process includes: extracting 3A statistical information of the target image area from the raw data output by the image sensor of the target camera based on the statistical density of the target image area; or, controlling the image sensor of the target camera to generate 3A statistical information of the target image area based on the statistical density of the target image area; extracting 3A statistical information of the target image area from the raw data output by the image sensor of the target camera; wherein the target image area includes one or more of the multiple image areas.

[0082] Exemplarily, when the target image area is the first image area, based on the first statistical density of the first image area, 3A statistical information of the first image area is extracted from the original data output by the image sensor of the target camera; or, based on the first statistical density of the first image area, the image sensor of the target camera is controlled to generate 3A statistical information of the target image area.

[0083] When the target image area is the second image area, based on the second statistical density of the second image area, extracting 3A statistical information of the second image area from raw data output by the image sensor of the target camera; or, based on the second statistical density, controlling the image sensor of the target camera to generate 3A statistical information of the second image area;

[0084] When the target image area includes the entire image area, based on the third statistical density, 3A statistical information of the entire area is extracted from the original data output by the image sensor of the target camera; or, based on the third statistical density, the image sensor of the target camera is controlled to generate 3A statistical information of the entire area.

[0085] Step 104: Based on the position information of the multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, performing differentiated multi-camera consistency processing on the first camera to determine the 3A control parameters of the first camera;

[0086] Differentiated multi-camera consistency processing refers to using different multi-camera consistency processing processes to process different image areas, thereby improving the consistency processing effect of the area of ​​interest and reducing the processing power consumption of the area of ​​non-interest.

[0087] In some embodiments, differentiated multi-camera consistency processing may include: allocating resources based on location information of multiple image areas, determining a first multi-camera consistency processing flow for the first image area, and a second multi-camera consistency processing flow for the second image area, wherein the performance of the first multi-camera consistency processing flow is better than the performance of the second multi-camera consistency processing flow; using the first multi-camera consistency processing flow, based on the 3A statistical information of the first camera in the first image area and the 3A statistical information of the second camera in the first image area, performing multi-camera consistency processing on the first camera in the first image area, and determining the 3A control parameters of the first camera in the first image area; using the second multi-camera consistency processing flow, based on the 3A statistical information of the first camera in the second image area and the 3A statistical information of the second camera in the second image area, performing multi-camera consistency processing on the first camera in the second image area, and determining the 3A control parameters of the first camera in the second image area.

[0088] The multi-camera consistency processing flow can be understood as the multi-camera consistency processing flow. Different multi-camera consistency processing flows have different processing resources, processing time, and processing effects.

[0089] Exemplarily, the first multi-camera consistency processing flow includes a first model and / or a first number of iterations, and the second multi-camera consistency processing flow includes a second model and / or a second number of iterations; the resource consumption when the first model is running is higher than the resource consumption when the second model is running, and the first number of iterations is greater than or equal to the second number of iterations. The first model can be a large model for realizing the multi-camera consistency function, and the second model can be a small model for realizing the multi-camera consistency function. The large model and the small model differ in parameter scale, data requirements, computing resources, and performance. A large model is allocated to the user's area of ​​interest to improve the consistency processing effect of the area of ​​interest, and a small model is allocated to the non-sensory area to reduce the processing power consumption and processing time of the non-interest area.

[0090] In some embodiments, the method may also include: performing differentiated 3A statistical information processing on the first camera based on location information of multiple image areas to determine the 3A statistical information of the first camera; or, performing full-area 3A statistical information processing on the first camera to determine the 3A statistical information of the first camera.

[0091] That is to say, the first camera can also perform differentiated 3A statistical information processing. The first camera can also perform 3A statistical information processing for the entire area. Exemplarily, based on the third statistical density, the first camera performs 3A statistical information processing for the entire area.

[0092] Step 105: Based on the 3A control parameters of the first camera, control the shooting of the first camera.

[0093] The image effect of the first camera under the determined 3A control parameters is close to the image effect of the second camera, and the multi-camera consistency processing is continuously performed in the shooting mode or focal length with the multi-camera consistency function.

[0094] By adopting the above technical solution, the user's areas of interest and areas of no interest are identified based on depth information, and reasonable processing resources are allocated according to different image areas. The consistency processing effect of the area of ​​interest is improved, and the processing power consumption of the area of ​​no interest is reduced, so as to better balance the effect and power consumption of the multi-camera consistency solution.

[0095] In order to better reflect the purpose of the present application, based on the above embodiments of the present application, the shooting control method is further illustrated by examples. Figure 4 As shown, the method specifically includes:

[0096] Step 401: When the first camera is started for preview or recording, the image sensor of the second camera is controlled to output a stream;

[0097] In some embodiments, the method may also include: when the first camera is started for preview or recording, if it is a smooth switching scene, the first image processing process is used to control the image sensor outflow of the second camera; if it is a stable preview or recording scene, the first image processing process is switched to the second image processing process, and the second image processing process is used to control the image sensor outflow of the second camera; wherein the first image processing process is used to process the image data collected by the second camera to generate a display image, and the second image processing process is used to obtain 3A statistical information of the second camera.

[0098] The image processing process can be understood as the process from the image sensor output flow of the control camera to the image display. In some embodiments, in a zoom scenario where the first camera is the main camera and the second camera is the auxiliary camera, the first image processing process can be a normal processing process (pipeline) or a lightweight processing process (pipeline). The raw data collected by the second camera is processed by the first image processing process to obtain standard 3A statistical information. The image data may not be discarded, but the image data and the image collected by the first camera are smoothed to obtain a fused image for display to achieve smooth zoom. The resolution and frame rate of the second camera image may be equal to or lower than the resolution or frame rate of the first camera image.

[0099] In some embodiments, in a stable preview or recording scenario where the first camera is the primary camera and the second camera is the secondary camera, the second image processing process can be a lightweight processing process (pipeline), and different from the first image processing process, the second camera switches from the first image processing process to the second image processing process, and processes the raw data collected by the second camera through the second image processing process. The raw data collected by the second camera is processed through the second image processing process to obtain standard 3A statistical information, and the image data can be discarded, and only the standard 3A statistical information is used to perform multi-camera consistency processing on the first camera.

[0100] In some embodiments, when the first camera is started for preview or recording, a third image processing flow is used to control the image sensor outflow of the first camera. The third image processing flow is a normal processing flow (pipeline) when the first camera is used as a main camera.

[0101] Figure 5 Schematic diagram of the image processing flow of the main and secondary cameras in the embodiment of the present application. Figure 5As shown in the figure, the processing flow of the main camera (telephoto) includes: the main camera sensor outputs raw data (including image data and 3A statistical information) with an image resolution of 12M and a frame rate of 30fps, the raw data is sequentially processed by the sensor front end (SFE) to extract 3A statistical information, the SFE sends the 3A statistical information of the main camera to the 3A module to realize the multi-camera consistency function, the SFE transmits the image data of the main camera backward, and is processed by the image front end (IFE) in turn, and then sent to the display for display after being processed by the image processing engine (IPE).

[0102] The processing flow of the secondary camera (wide-angle) includes: the secondary camera sensor outputs raw data with an image resolution of 3M and a frame rate of 15fps. The SFE of the secondary camera path only extracts the 3A statistical information of the secondary camera and sends it to the 3A module. The 3A module aligns the 3A statistical information of the primary and secondary cameras with the depth information of the current scene based on the timestamp, and then performs multi-camera consistency processing based on the 3A algorithm to determine the 3A control parameters of the primary camera.

[0103] Different from the normal dual-camera scene (SAT and fallback), the three-camera consistency processing is mainly aimed at the 3A multi-camera consistency, so it is only necessary to obtain the normal 3A statistical information of the secondary camera path, and the image data is not concerned. Therefore, the secondary camera (wide-angle) processing flow can discard the image data after extracting the 3A statistical information. For the normal dual-camera scene, it is necessary to process the image data of two channels, so normal image data output is also required.

[0104] After the differentiation processing is triggered, the control module for consistency processing in the 3A module will actively obtain the depth information of the current scene (depth estimation based on the image data of the main camera) through the next frame request (the upper layer will send the frame request for the next frame from top to bottom, that is, the request information); and then convert it into the position information of multiple image areas for differentiation processing of the auxiliary camera (Aux).

[0105] The 3A module sends a request to the 3A statistics module (such as SFE) of the Aux (auxiliary camera) to send the position information of multiple image areas that are differentially processed for the auxiliary camera (Auxiliary Camera), so that differentiated 3A statistics can be calculated based on the depth information of the current scene. For differentiated statistics, you can choose to update only the 3A statistics of some areas or change the statistical density of some areas (for example, the statistical density of the focal plane area is higher, and the statistical density of the non-focal plane area is lower. The specific density can be determined based on the distance of each area relative to the focal plane.

[0106] It can be seen that the data flow of the secondary camera and the main camera is obviously different. In fact, the two belong to different image processing processes and there is no direct information synchronization mechanism. Therefore, the synchronization mechanism between the two is not very perfect. The information intersection point in the two image processing processes is the 3A module. For the depth information of the current scene, in one example, it is obtained based on the processing flow of the main camera, and this information is obtained based on the current image of the main camera. Therefore, it can only be synchronized in the common module (i.e., 3A module) passed by the two processing flows. Of course, the main camera can also choose to directly synchronize the depth information to the module of the secondary camera to convert the position information, but since the subsequent differentiated consistency processing also requires position information, the depth information can be first given to the 3A module to convert the position information. In this way, on the one hand, the differentiated processing of 3A statistical information can be directly performed on the secondary camera, and the position information of multiple image areas can also be directly synchronized to the consistency processing algorithm for differentiated multi-camera consistency processing.

[0107] The main camera module can also convert the depth information into the position information of the image area and then synchronize it to the 3A module.

[0108] Step 402: Determine whether the number of frames output by the image sensor of the second camera is greater than or equal to the first threshold; if not, execute step 403; if yes, execute step 405;

[0109] Step 403: If the number of frames output by the image sensor of the second camera is less than the first threshold, perform 3A statistical information processing on the entire area of ​​the second camera to determine the 3A statistical information of the second camera;

[0110] When the second camera is used as the secondary camera, there is no 3A statistical information in the first three frames when the sensor is started to stream. In addition, the processing process will be delayed by about three frames. Therefore, the consistency processing will take effect after about eight frames. Considering the gradual change of the consistency effect, it is chosen to use the depth information of the current scene for differentiated processing after eight frames to avoid the problem of consistency effect jump caused by directly using this method, achieve smooth transition, and improve the consistency processing effect of multiple cameras.

[0111] In some embodiments, based on the third statistical density, 3A statistical information processing is performed on the entire area of ​​the second camera to determine the 3A statistical information of the second camera.

[0112] Step 404: Based on the 3A statistical information of the first camera and the 3A statistical information of the second camera, perform multi-camera consistency processing on the first camera to determine the 3A control parameters of the first camera;

[0113] Here, the multi-camera consistency processing can be understood as a multi-camera consistency processing of the entire area, specifically referring to using a multi-camera consistency processing flow to process the entire image area to ensure a consistent effect of the entire area.

[0114] In some embodiments, a third multi-camera consistency processing flow is used to perform consistency on the first camera in the first image area based on the 3A statistical information of the first camera and the 3A statistical information of the second camera. In some embodiments, the third multi-camera consistency processing flow is the same as the first multi-camera consistency processing flow.

[0115] Step 405: If the number of frames output by the image sensor of the second camera is greater than or equal to the first threshold, obtain depth information of the current scene;

[0116] In some embodiments, obtaining the depth information of the current scene includes: determining the depth information of the current scene based on a depth estimation algorithm of monocular vision. Exemplarily, inputting the image data of the first camera into a first processing model to calculate the depth information, and the first processing model is a model constructed based on a depth estimation algorithm of monocular vision, for example, the depth estimation algorithm of monocular vision can be a neural network algorithm.

[0117] In some embodiments, obtaining the depth information of the current scene includes: determining the depth information of the current scene based on a binocular vision depth estimation algorithm. Exemplarily, the image data of the first camera, the image data of the second camera, and the camera calibration parameters are input into a second processing model to calculate the depth information, and the second processing model is a model constructed based on a binocular vision depth estimation algorithm, for example, the binocular vision depth estimation algorithm is a binocular parallax depth estimation algorithm.

[0118] Exemplarily, in order to save power consumption, a depth estimation algorithm based on monocular vision may be selected. Exemplarily, in order to improve the depth estimation accuracy, a depth estimation algorithm based on binocular vision may be selected.

[0119] Step 406: Divide the image area based on the depth information of the current scene to obtain position information of multiple image areas;

[0120] Step 407: Obtain 3A statistical information of the first camera and 3A statistical information of the second camera;

[0121] Exemplarily, based on the position information of multiple image areas, differentiated 3A statistical information processing is performed on the first camera to determine the 3A statistical information of the first camera; based on the position information of multiple image areas, differentiated 3A statistical information processing is performed on the second camera to determine the 3A statistical information of the second camera.

[0122] Step 408: Based on the position information of the multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, performing differentiated multi-camera consistency processing on the first camera to determine the 3A control parameters of the first camera;

[0123] Step 409: Based on the 3A control parameters of the first camera, control the shooting of the first camera.

[0124] In order to better reflect the purpose of this application, further examples are given based on the above embodiments of this application. Figure 6 As shown, the method specifically includes:

[0125] Step 601: Enter a preview or recording scene that supports multi-camera consistency;

[0126] It should be noted that, in addition to the preview or recording scenes, other shooting modes or focal lengths with multi-camera consistency functions can adopt the shooting control method provided in the embodiments of the present application to achieve multi-camera consistency.

[0127] Step 602: Aux pipeline starts to output stream;

[0128] When entering a preview or recording scene that supports multi-camera consistency, if the reference camera (such as wide-angle W) is the main camera, there is no need to enable the secondary camera. If other cameras (such as telephoto T) are the main cameras, the reference camera needs to be used as the secondary camera, and multi-camera consistency processing is performed on other cameras. At this time, the lightweight processing flow (lightweight pipeline) of W is initialized, and W is configured as a low-resolution configuration, and W starts to output the stream as the secondary camera.

[0129] Step 603: Determine whether the number of frames captured by the secondary camera is greater than 8. If not, execute step 604; if yes, execute step 606;

[0130] For the first few frames from the secondary camera, 3A statistical information processing and consistency processing will be directly performed on the entire area. Only after the number of frames from the secondary camera reaches 8 will the differentiated processing flow begin.

[0131] When W is used as the secondary camera, there is no 3A statistical information in the first three frames when the sensor is started to stream. In addition, the processing process will be delayed by about three frames. Therefore, the consistency processing will take effect after about eight frames. Considering the gradual change of the consistency effect, it is chosen to use the depth information of the current scene for differentiation processing after eight frames to avoid the problem of consistency effect jump caused by directly using this method, achieve smooth transition, and improve the consistency processing effect of multiple cameras.

[0132] Step 604: Processing 3A statistical information of the entire secondary camera area;

[0133] Step 605: multi-camera consistency processing for the entire area;

[0134] Step 606: Obtain depth information corresponding to the main camera, and convert the depth information into position information;

[0135] After triggering the differentiation processing, the control module of the consistency processing on the 3A side will actively obtain the depth information of the current scene (based on the main camera) through the next frame request (the upper layer will send the frame request for the next frame from top to bottom, that is, the request information); and then convert it into differentiated position information for the auxiliary camera (Aux) processing.

[0136] Step 607: 3A statistical information processing of secondary camera differentiation;

[0137] The 3A statistics module determines the image area to be differentially processed based on the position information obtained in 606, and extracts 3A statistical information of different statistical densities.

[0138] Differentiated generation can also be achieved at the stage of sensor photosensitization and 3A statistical information generation; in this way, the data volume has changed when the sensor outputs data, and the system power consumption can be better saved during post-transmission and processing; at the same time, the sensor side can be directly processed based on the sensor hardware, and the generation speed of differentiated 3A statistical information is basically the same as that of normal 3A statistical information, which will not cause the processing time to increase. The frame-by-frame control of the sensor can completely control the real-time control of information collection by updating the information of the corresponding register, and can also ensure the real-time nature of the changes.

[0139] Step 608: differentiated multi-camera consistency processing;

[0140] After obtaining the differentiated 3A statistical information of the secondary camera, the 3A statistical information of the secondary camera and the image area division result are given to the 3A module for multi-camera consistency processing. That is, when performing multi-camera consistency processing, differentiated consistency will also be performed based on the image area division result.

[0141] Step 609: updating the 3A control parameters of the main camera based on the consistency processing result;

[0142] Step 610: Continue previewing / recording.

[0143] In addition to enhancing the accuracy of information in the focal plane area by changing the statistical density of different areas of the 3A statistical information, it is also possible to perform high-density statistics of 3A statistical information frame by frame only on the focal plane area (i.e., the first image area); and for other areas, statistics of the entire area can be performed every other frame or every few frames. In this way, only updated data can be focused on during processing and transmission of 3A statistical information, and the missing parts (the parts not counted in the current frame can be supplemented with data from historical frames). In this way, when the consistency processing algorithm is processed later, multi-camera consistency processing is still performed on the entire area, but because the information in other areas outside the focal plane area is consistent with the historical frames, the natural computing power allocation will not be too much, and more computing power and better models can be applied to the focal plane area that users are concerned about.

[0144] In some embodiments, in addition to performing differentiated processing and differentiated consistency processing on the 3A statistical information of a partial image area, the 3A control parameters of the partial image area may also be updated based on changes in depth information; for example, after the consistency processing is completed to generate new 3A control parameters, the AE / AWB control parameters of the focal plane area are updated in real time frame by frame, while the non-focal plane area (second image area) is updated every other frame. The updating every few frames can also be further distinguished based on the distance difference between the corresponding plane and the focal plane (every 1 frame or every 2 frames).

[0145] The beneficial effects of adopting the above technical solution are as follows:

[0146] 1. Based on the depth information of the current scene, differentiated control is performed to collect 3A statistical information of the current scene and perform subsequent consistency processing, so that the processing resources are reasonably allocated, so that the processed effect can highlight the area of ​​interest of the user, improve the consistency effect, and reduce system power consumption;

[0147] 2. Add a depth information synchronization mechanism between the main and secondary cameras, obtain the depth information of the current main camera frame in real time, and apply it to the processing of the 3A statistical information of the secondary camera, as well as the consistency processing. Since the image processing process of the secondary camera is a lightweight pipeline, it only goes through the SFE or IFE part of the secondary camera image processing process. Therefore, there is no way to obtain the depth information of the current scene based on dual-camera parallax. Therefore, the depth information can be obtained based on the image data of a single camera, especially for the main camera (master); the depth information of the main camera is used as the basis for controlling the differentiated processing of the secondary camera 3A statistical information, that is, the 3A statistical information of the secondary camera is counted based on the depth information of the main camera. Statistics can also be performed on the 3A statistical information of the main camera based on the depth information of the main camera.

[0148] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides a shooting control device, such as Figure 7 As shown, the shooting control device 70 includes:

[0149] An acquisition unit 701 is used to acquire depth information of a current scene when the first camera is started for preview or recording;

[0150] A first processing unit 702 is used to divide the image area based on the depth information of the current scene to obtain position information of multiple image areas;

[0151] The acquisition unit 701 is further configured to acquire the 3A statistical information of the first camera and the 3A statistical information of the second camera;

[0152] The second processing unit 703 is further configured to perform differentiated multi-camera consistency processing on the first camera based on the position information of the multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, and determine the 3A control parameters of the first camera;

[0153] The control unit 704 is used to control the shooting of the first camera based on the 3A control parameters of the first camera.

[0154] In some embodiments, the first processing unit 702 is used to determine the region of interest of the current scene based on the depth information of the current scene and the preset depth range; divide the image region based on the region of interest, and determine the position information of multiple image regions. The position information includes at least the coordinate information of specific pixels in the image region, the coordinate information of multiple edge pixels in the image region, and the coordinate information may also include the coordinate information of the upper left corner pixel or the center pixel of the image region. The position information may also include the size of the image region.

[0155] In some embodiments, the plurality of image regions include: a focal plane region, a secondary focal plane region, and a background region.

[0156] In some embodiments, the acquisition unit 701 is used to perform differentiated 3A statistical information processing on the first camera based on the position information of multiple image areas to determine the 3A statistical information of the first camera; based on the position information of the multiple image areas, perform differentiated 3A statistical information processing on the second camera to determine the 3A statistical information of the second camera.

[0157] In some embodiments, the acquisition unit 701 is used to perform 3A statistical information processing on the entire area of ​​the first camera to determine the 3A statistical information of the first camera; based on the position information of the multiple image areas, perform differentiated 3A statistical information processing on the second camera to determine the 3A statistical information of the second camera.

[0158] In some embodiments, the acquisition unit 701 is used to perform differentiated 3A statistical information processing on the first camera based on the position information of the multiple image areas to determine the 3A statistical information of the first camera; and perform full-area 3A statistical information processing on the second camera to determine the 3A statistical information of the second camera.

[0159] In some embodiments, 3A statistical information of the entire area of ​​the first camera is processed to determine the 3A statistical information of the first camera; based on the position information of the multiple image areas, 3A statistical information of the entire area of ​​the second camera is processed to determine the 3A statistical information of the second camera.

[0160] In some embodiments, the multiple image areas include at least a first image area and a second image area, and the first processing unit 702 is used to perform statistical density allocation based on the position information of the multiple image areas, determine a first statistical density of the first image area, and a second statistical density of the second image area, wherein the first statistical density is greater than the second statistical density; adopt a target acquisition process, based on the first statistical density, to acquire 3A statistical information of a target camera in the first image area, wherein the target camera is the first camera or the second camera; adopt a target acquisition process, based on the second statistical density, to acquire 3A statistical information of the target camera in the second image area;

[0161] Alternatively, the first processing unit 702 is configured to, when the current image satisfies the first condition and the current image captured by the target camera for the current scene satisfies the first condition, adopt a target acquisition process to obtain 3A statistical information of the target camera in the first image area based on the first statistical density; and use the 3A statistical information of the second image area in the reference image of the current image as the 3A statistical information of the target camera in the second image area;

[0162] Alternatively, the first processing unit 702 is configured to adopt a target acquisition process and obtain 3A statistical information of the target camera in the entire image area based on the third statistical density.

[0163] In some embodiments, the target acquisition process includes: extracting 3A statistical information of the target image area from the raw data output by the image sensor of the target camera based on the statistical density of the target image area; or, controlling the image sensor of the target camera to generate 3A statistical information of the target image area based on the statistical density of the target image area; extracting 3A statistical information of the target image area from the raw data output by the image sensor of the target camera; wherein the target image area includes one or more of the multiple image areas.

[0164] Exemplarily, when the target image area is the first image area, based on the first statistical density of the first image area, 3A statistical information of the first image area is extracted from the original data output by the image sensor of the target camera; or, based on the first statistical density of the first image area, the image sensor of the target camera is controlled to generate 3A statistical information of the target image area.

[0165] When the target image area is the second image area, based on the second statistical density of the second image area, extracting 3A statistical information of the second image area from raw data output by the image sensor of the target camera; or, based on the second statistical density, controlling the image sensor of the target camera to generate 3A statistical information of the second image area;

[0166] When the target image area includes the entire image area, based on the third statistical density, 3A statistical information of the entire area is extracted from the original data output by the image sensor of the target camera; or, based on the third statistical density, the image sensor of the target camera is controlled to generate 3A statistical information of the entire area.

[0167] In some embodiments, the multiple image areas include at least a first image area and a second image area, and the second processing unit 703 is used to allocate resources based on the location information of the multiple image areas, determine a first multi-camera consistency processing flow for the first image area, and a second multi-camera consistency processing flow for the second image area, wherein the performance of the first multi-camera consistency processing flow is better than the performance of the second multi-camera consistency processing flow; using the first multi-camera consistency processing flow, based on the 3A statistical information of the first camera in the first image area and the 3A statistical information of the second camera in the first image area, multi-camera consistency processing is performed on the first camera in the first image area, and the 3A control parameters of the first camera in the first image area are determined; using the second multi-camera consistency processing flow, based on the 3A statistical information of the first camera in the second image area and the 3A statistical information of the second camera in the second image area, multi-camera consistency processing is performed on the first camera in the second image area, and the 3A control parameters of the first camera in the second image area are determined.

[0168] In some embodiments, the first multi-shot consistency processing flow includes a first model and / or a first number of iterations, and the second multi-shot consistency processing flow includes a second model and / or a second number of iterations;

[0169] The resource consumption when the first model is run is higher than the resource consumption when the second model is run, and the first iteration number is greater than or equal to the second iteration number.

[0170] In some embodiments, the acquisition unit 701 is used to control the image sensor outflow of the second camera when the first camera is started for preview or recording; determine whether the number of frames output by the image sensor of the second camera is greater than or equal to a first threshold; if the number of frames output by the image sensor of the second camera is greater than or equal to the first threshold, execute acquisition of depth information of the current scene.

[0171] In some embodiments, the first processing unit 702 is further configured to perform 3A statistical information processing of the entire area of ​​the second camera to determine the 3A statistical information of the second camera if the number of frames output by the image sensor of the second camera is less than the first threshold value;

[0172] The second processing unit 703 is further used to perform full-area consistency processing on the first camera based on the 3A statistical information of the first camera and the 3A statistical information of the second camera, and determine the 3A control parameters of the first camera.

[0173] In some embodiments, the control unit 704 is also used to, when the first camera is started for preview or recording, if it is a smooth switching scene, use the first image processing process to control the image sensor outflow of the second camera; if it is a stable preview or recording scene, switch the first image processing process to the second image processing process, and use the second image processing process to control the image sensor outflow of the second camera; wherein the first image processing process is used to process the image data collected by the second camera to generate a display image, and the second image processing process is used to obtain 3A statistical information of the second camera.

[0174] In practical applications, the shooting control device can be an electronic device that implements the shooting function, or a chip applied to an electronic device. In the present application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the shooting control method provided in any of the above embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the shooting control method, and this application will not elaborate on this one by one.

[0175] The electronic devices described in this application may include, for example, mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), wearable devices, smart bracelets, cameras, etc.

[0176] Based on the hardware implementation of each unit in the above-mentioned shooting control device, the embodiment of the present application also provides a shooting control device, such as Figure 8As shown, the shooting control device 80 includes: a processor 801 and a memory 802 configured to store a computer program that can be run on the processor; wherein the processor 801 is configured to execute the method steps in the aforementioned embodiment when running the computer program.

[0177] Of course, in practical applications, Figure 8 As shown, the various components in the shooting control device 80 are coupled together through a bus system 803. It can be understood that the bus system 803 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 803 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are marked as bus system 803 in the figure.

[0178] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0179] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0180] In an exemplary embodiment, the shooting control device further provided in the embodiment of the present application may be a chip for implementing shooting control. Optionally, the chip may also include at least one input interface. The processor may control the input interface to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips. The image data output by the image sensor of the camera device is obtained through the input interface.

[0181] Optionally, the chip may further include at least one output interface, wherein the processor may control the output interface to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0182] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of the shooting control device to complete the steps of the aforementioned method.

[0183] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0184] Optionally, the computer program product can be applied to the shooting control device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the shooting control device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0185] The embodiment of the present application also provides a computer program.

[0186] Optionally, the computer program can be applied to the shooting control device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the shooting control device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0187] It should be understood that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0188] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0189] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. The technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0192] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0193] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A shooting control method, characterized in that: The method comprises: When the first camera is started for preview or recording, depth information of the current scene is obtained; Dividing the image area based on the depth information of the current scene to obtain position information of multiple image areas; Obtaining 3A statistical information of the first camera and 3A statistical information of the second camera; Based on the position information of the multiple image areas, the 3A statistical information of the first camera and the 3A statistical information of the second camera, performing differentiated multi-camera consistency processing on the first camera to determine the 3A control parameters of the first camera; Based on the 3A control parameters of the first camera, shooting control is performed on the first camera.

2. The method according to claim 1, characterized in that The image region division based on the depth information of the current scene to obtain position information of multiple image regions includes: Determining a region of interest of the current scene based on the depth information of the current scene and a preset depth range; Image regions are divided based on the region of interest to determine position information of the multiple image regions.

3. The method according to claim 1 or 2, characterized in that: The multiple image areas include: a focal plane area, a secondary focal plane area and a background area.

4. The method according to claim 1, characterized in that The obtaining of the 3A statistical information of the first camera and the 3A statistical information of the second camera includes: Based on the position information of the multiple image areas, performing differentiated 3A statistical information processing on the first camera to determine 3A statistical information of the first camera; Based on the position information of the multiple image areas, differentiated 3A statistical information processing is performed on the second camera to determine the 3A statistical information of the second camera.

5. The method according to claim 4, characterized in that The multiple image regions include at least a first image region and a second image region, and the differentiated 3A statistical information processing includes: Performing statistical density allocation based on the position information of the multiple image regions, determining a first statistical density of the first image region and a second statistical density of the second image region, wherein the first statistical density is greater than the second statistical density; Using a target acquisition process, based on the first statistical density, acquiring 3A statistical information of a target camera in the first image area; Using a target acquisition process, based on the second statistical density, acquiring 3A statistical information of the target camera in the second image area; The differentiated 3A statistical information processing includes: When a current image captured by a target camera for a current scene satisfies a first condition, a target acquisition process is adopted to acquire 3A statistical information of the target camera in the first image area based on the first statistical density; Using the 3A statistical information of the second image area in the reference image of the current image as the 3A statistical information of the target camera in the second image area; When the current image satisfies the second condition, a target acquisition process is adopted to acquire 3A statistical information of the target camera in the entire image area based on a third statistical density; The target camera is the first camera or the second camera.

6. The method according to claim 5, characterized in that The target acquisition process includes: Extracting 3A statistical information of the target image area from raw data output by an image sensor of the target camera based on the statistical density of the target image area; or, Based on the statistical density of the target image area, controlling the image sensor of the target camera to generate 3A statistical information of the target image area; Extracting 3A statistical information of the target image area from raw data output by the image sensor of the target camera; The target image area includes one or more of the multiple image areas.

7. The method according to claim 1, characterized in that The multiple image regions include at least a first image region and a second image region, and the differentiated multi-shot consistency processing includes: Allocating resources based on the position information of the multiple image areas, determining a first multi-camera consistency processing flow for the first image area, and a second multi-camera consistency processing flow for the second image area, wherein performance of the first multi-camera consistency processing flow is better than performance of the second multi-camera consistency processing flow; Using a first multi-camera consistency processing flow, based on 3A statistical information of the first camera in the first image area and 3A statistical information of the second camera in the first image area, performing multi-camera consistency processing on the first camera in the first image area, and determining a 3A control parameter of the first camera in the first image area; A second multi-camera consistency processing process is adopted to perform multi-camera consistency processing on the first camera in the second image area based on the 3A statistical information of the first camera in the second image area and the 3A statistical information of the second camera in the second image area, and determine the 3A control parameters of the first camera in the second image area.

8. The method according to claim 7, characterized in that The first multi-shot consistency processing flow includes a first model and / or a first number of iterations, and the second multi-shot consistency processing flow includes a second model and / or a second number of iterations; The resource consumption when the first model is running is higher than the resource consumption when the second model is running, and the first number of iterations is greater than or equal to the second number of iterations.

9. The method according to claim 1, characterized in that: The step of obtaining depth information of the current scene when the first camera is started for preview or recording includes: When the first camera is started for preview or recording, controlling the image sensor outflow of the second camera; Determine whether the number of frames output by the image sensor of the second camera is greater than or equal to a first threshold; If the number of frames output by the image sensor of the second camera is greater than or equal to the first threshold, the depth information of the current scene is obtained.

10. The method according to claim 9, characterized in that The method further comprises: If the number of frames output by the image sensor of the second camera is less than the first threshold, performing 3A statistical information processing on the entire area of ​​the second camera to determine the 3A statistical information of the second camera; Based on the 3A statistical information of the first camera and the 3A statistical information of the second camera, multi-camera consistency processing is performed on the first camera to determine the 3A control parameters of the first camera.

11. The method according to claim 9, characterized in that The method further comprises: When the first camera is started for preview or recording, if the scene is switched smoothly, the first image processing process is used to control the image sensor outflow of the second camera; If it is a stable preview or recording scene, the first image processing process is switched to the second image processing process, and the second image processing process is used to control the image sensor outflow of the second camera; The first image processing process is used to process the image data collected by the second camera to generate a display image, and the second image processing process is used to obtain 3A statistical information of the second camera.

12. A shooting control device, characterized in that: The device comprises: An acquisition unit, configured to acquire depth information of a current scene when the first camera is started for preview or recording; A first processing unit, configured to divide the image region based on the depth information of the current scene to obtain position information of multiple image regions; The acquisition unit is further used to acquire the 3A statistical information of the first camera and the 3A statistical information of the second camera; The second processing unit is further configured to perform differentiated multi-camera consistency processing on the first camera based on the position information of the multiple image areas, the 3A statistical information of the first camera, and the 3A statistical information of the second camera, and determine the 3A control parameters of the first camera; A control unit is used to control the shooting of the first camera based on the 3A control parameters of the first camera.

13. A shooting control device, characterized in that: The device comprises: a processor and a memory configured to store a computer program capable of being executed on the processor, Wherein, the processor is configured to execute the steps of the method described in any one of claims 1 to 11 when running the computer program.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images