Camera occlusion identification method and device, and storage medium

Abstract

The application discloses a camera shielding identification method and device and a storage medium, and belongs to the photographic technical field.The method is applied to an electronic device comprising multiple cameras, and comprises the following steps: in the case that the electronic device switches from a first camera to a second camera for shooting, determining first statistical information of a memory frame of the first camera and second statistical information of a current frame shot by the second camera; wherein the memory frame is the last frame shot by the first camera, the first statistical information comprises a first target white point and multiple first statistical white points of the memory frame, and the second statistical information comprises a second target white point and multiple second statistical white points of the current frame; determining a shielding identification result of the first camera according to the difference between the first statistical information and the second statistical information, wherein the shielding identification result represents whether the first camera is shielded or not.The application can improve the accuracy of identifying whether the camera is shielded or not.

Description

Technical Field

[0001] This application relates to the field of camera technology, and in particular to a method, device and storage medium for camera occlusion recognition. Background Technology

[0002] As users' photography needs increase, camera modules in mobile phones and other electronic devices have evolved from single-camera to multi-camera systems. This means that electronic devices are equipped with multiple cameras, such as a main camera, a telephoto camera, and an ultra-wide-angle camera. In some scenarios, such as when an electronic device switches from one camera to another, it may be necessary to identify whether the previous camera was obstructed. Based on the identification result, appropriate processing may be performed. For example, the identification result may determine whether to use the image captured by the previous camera as a reference image to correct the image captured by the new camera.

[0003] Currently, when an electronic device switches from one camera to another for shooting, the brightness difference between the image captured by the previous camera and the image captured by the new camera can be determined, and this difference in brightness can be used to identify whether the previous camera was obstructing the view. However, if the previous camera is only partially or completely obstructing the view, the brightness difference between the two images will be very small, making it difficult to distinguish whether the previous camera was obstructing the view based solely on brightness difference, leading to incorrect obstruction identification. Therefore, the accuracy of identifying camera obstruction based on image brightness difference is relatively low. Summary of the Invention

[0004] This application provides a method, device, and storage medium for camera occlusion detection, which can improve the accuracy of detecting whether a camera is occluded. The technical solution is as follows:

[0005] Firstly, a method for occlusion recognition of a camera is provided, applied in an electronic device including multiple cameras. The method includes: after the electronic device switches from a first camera to a second camera for shooting, determining first statistical information of a memory frame from the first camera and second statistical information of the current frame captured by the second camera. Based on the difference between the first and second statistical information, an occlusion recognition result for the first camera is determined, i.e., whether the first camera is occluded.

[0006] The memory frame is the last frame captured by the first camera. The first statistical information includes the first target white point and multiple first statistical white points in the memory frame. The second statistical information includes the second target white point and multiple second statistical white points in the current frame.

[0007] Since image statistical information can more accurately reflect image features such as color temperature, and the statistical information of images captured by the corresponding camera will differ significantly depending on whether the camera is occluded or not, the difference in statistical information between images captured by the camera before and after switching can more accurately identify whether the camera was occluded before switching, thereby improving the accuracy of camera occlusion identification. Furthermore, the occlusion identification results identified in this application can more accurately correct images captured by the camera after switching, improving the accuracy of image correction. For example, it can, to some extent, avoid the severe color cast caused by using images captured by an occluded camera as reference images for color correction.

[0008] The scenario of switching from the first camera to the second camera for shooting can include: a scenario where the camera switches directly from the first camera to the second camera for shooting, or a scenario where the camera switches from the first camera to another camera first, and then switches from the other camera to the second camera. This application embodiment does not limit this. The first camera can be any one of multiple cameras configured in the electronic device, and the second camera can be any one of the multiple cameras other than the first camera.

[0009] As an example, one of the multiple cameras configured in the electronic device can be pre-set as a reference camera. This allows for color correction of images captured by other cameras based on the image parameters of the reference camera, ensuring that the colors of images captured by other cameras approximate those of the reference camera. This reduces color differences in multi-camera shots of the same scene and improves color consistency. The reference camera can be any of the multiple cameras, such as the camera that the camera application uses by default. Typically, the camera application uses the main camera by default upon startup, so the main camera can be set as the reference camera. Correspondingly, the first camera is the pre-set reference camera. That is, in this embodiment, when the electronic device switches from the reference camera to other cameras for shooting, the occlusion detection method provided in this embodiment can be used to determine whether the reference camera is occluded.

[0010] In one embodiment, a target quantity ratio can be determined based on first statistical information; if the target quantity ratio is less than a first threshold, a first ratio and a second ratio can be determined based on the first statistical information and the second statistical information; if the ratio of the first ratio to the second ratio is greater than the first ratio threshold or less than the second ratio threshold, it can be determined that the first camera is obstructed.

[0011] The target quantity ratio is the ratio of the target quantity to the total number of multiple first statistical white points. The target quantity refers to the number of first statistical white points whose coordinates fall within a radius of a preset distance centered on the coordinates of the first target white point. The preset distance can be set in advance as needed. For example, the preset distance can be an empirical value obtained by statistically analyzing the target white points and statistical white points in images captured by the first camera under multiple standard light sources.

[0012] It should be noted that when the main camera is obstructed, the captured image is essentially a solid color. For example, when the camera is obstructed by a hand, the captured image is reddish, and the B / G values ​​of the statistical white points are similar. The statistical white points are generally distributed horizontally in a coordinate system with R / G as the x-axis and B / G as the y-axis. This distribution results in the statistical white points being far from the target white point, and the number of statistical white points falling within a circle centered on the target white point with a preset distance r as the radius is small. Therefore, this embodiment can determine whether the first camera is obstructed based on the target number ratio.

[0013] For example, when the target quantity ratio is less than the first threshold, it indicates that the statistical white points in the memory frame are far from the target white points, and the first camera may be obstructing the view. When the target quantity ratio is greater than or equal to the first threshold, it indicates that the statistical white points in the memory frame are close to the target white points, and the first camera is not obstructing the view.

[0014] Wherein, the first ratio is the ratio of the mean of the first color statistical value of multiple first statistical white points to the mean of the second color statistical value, the second ratio is the ratio of the mean of the first color statistical value of multiple second statistical white points to the mean of the second color statistical value, the first color statistical value is the ratio of the R value to the G value of the corresponding statistical white point, and the second color statistical value is the ratio of the B value to the G value of the corresponding statistical white point.

[0015] In this embodiment, when the target number ratio is less than a first threshold, it can be preliminarily determined that the first camera may be obstructed. Then, other information can be combined to further determine whether the first camera is obstructed. For example, a first ratio and a second ratio can be determined based on the first statistical information and the second statistical information, and the obstruction of the first camera can be determined based on the first ratio and the second ratio.

[0016] As an example, after determining that the target quantity ratio is less than a first threshold, it can be further determined whether the target quantity ratio is less than or equal to a second threshold. If so, it is determined that the first camera is obstructed; if not, a first ratio and a second ratio are further determined based on the first and second statistical information, and the ratio of the first ratio to the second ratio is used to determine whether the first camera is obstructed. This can improve recognition efficiency.

[0017] In one embodiment, after determining the occlusion recognition result of the first camera, it is also possible to determine whether to perform an image correction process based on the camera recognition result, that is, whether to use the memory frame of the first camera as a reference image to correct the image frame captured by the second camera. Since the camera occlusion recognition method provided in this application embodiment can more accurately identify whether the camera is occluded, the occlusion recognition result identified by this application can more accurately correct the image captured by the switched camera, improving the accuracy of image correction. For example, it can, to a certain extent, avoid the serious color cast caused by using the image captured by the occluded camera as a reference image for color correction.

[0018] As an example, if the occlusion recognition result indicates that the first camera is not occluded, then the white point information of the first target white point and the white point information of the second target white point in the memory frame are fused, and the white balance is corrected for the current frame based on the fusion result; if the occlusion recognition result indicates that the first camera is occluded, then the white balance is corrected for the current frame based on the white point information of the second target white point.

[0019] As an example, the white point information of the first target white point and the white point information of the second target white point in the memory frame are fused together. Based on the fusion result, white balance correction is performed on the current frame, including: fusing the white point information of the first target white point and the white point information of the second target white point in the memory frame based on the ambient color temperature difference between the first ambient color temperature of the shooting environment corresponding to the memory frame and the second ambient color temperature of the shooting environment corresponding to the current frame to obtain the white point information of the third target white point; and performing white balance correction on the current frame based on the white point information of the third target white point.

[0020] In this way, when the difference between the ambient color temperature of the second camera and the first camera is small, it can be determined that the shooting scene is relatively similar and that the shots are likely taken in the same scene. By combining the target white point information of the memory frame of the first camera with the target white point information of the current frame, the color of the current frame is corrected by performing white balance correction, thereby reducing the color difference between the current frame and the memory frame of the first camera. This reduces the color difference between the images taken by other cameras and the first camera in the same scene and the resulting visual differences, improving the color consistency of multi-camera shooting in the same scene, and thus improving the user's visual and shooting experience.

[0021] In one embodiment, the fusion weight can be determined based on the ambient color temperature difference; based on the fusion weight, the white point information of the first target white point and the white point information of the second target white point are fused to obtain the white point information of the third target white point.

[0022] The ambient color temperature difference between the second and first ambient color temperatures represents the environmental difference between the current shooting environment and the shooting environment of the memory frame. The ambient color temperature difference is inversely proportional to the fusion weight; the smaller the ambient color temperature difference, the greater the fusion weight.

[0023] Thus, when the ambient color temperature difference is small, it means that the current shooting environment and the shooting environment of the memory frame are relatively similar. In this case, by applying a larger fusion weight to the second target white point of the memory frame, a greater degree of color correction can be applied to the current frame, making the corrected image color of the current frame much closer to the image color of the memory frame. When the ambient color temperature difference is large, it means that the current shooting environment and the shooting environment of the memory frame are significantly different. In this case, color differences between the current frame and the memory frame are normal. In this case, a smaller degree of fusion weight can be applied to the second target white point of the memory frame to apply a smaller degree of color correction to the current frame.

[0024] In one embodiment, the first target white point can be mapped to the second camera to obtain the mapped white point; then, according to the fusion weight, the white point information of the mapped white point and the white point information of the second target white point are fused to obtain the white point information of the third target white point.

[0025] By first mapping the first target white point of the memory frame to the second camera to obtain the mapped white point, and then fusing the white point information of the mapped white point with the white point information of the second target white point in the current frame, the accuracy of multi-camera color correction can be further improved.

[0026] In one embodiment, mapping a first target white point to a second camera to obtain a mapped white point includes: acquiring a first camera and its calibration data, the calibration data including white point information and color temperature under multiple standard light sources; determining at least one standard light source from the multiple standard light sources based on the color temperature difference between the estimated color temperature of the first camera and the estimated color temperature of the second camera; determining a white point mapping matrix between the first camera and the second camera based on the white point information of the first camera under the at least one standard light source and the white point information of the second camera under the at least one standard light source; and mapping the first target white point to the second camera based on the white point mapping matrix to obtain the mapped white point. This improves the accuracy of white point mapping.

[0027] In one embodiment, determining the fusion weights based on the ambient color temperature difference includes: determining a first sub-weight based on the ambient color temperature difference; determining a second sub-weight based on the estimated color temperature difference between a first estimated color temperature of the memory frame and a second estimated color temperature of the current frame; and determining the fusion weights based on the first sub-weights and the second sub-weights.

[0028] Among them, the smaller the ambient color temperature difference, the greater the weight of the first sub-sub ...

[0029] The estimated color temperature difference between the second and first estimated color temperatures indicates the difference in shooting effects between the second and first cameras. The estimated color temperature difference is proportional to the second sub-weight; the larger the estimated color temperature difference, the larger the second sub-weight.

[0030] Thus, when the estimated color temperature difference is small, it indicates that the shooting effects of the second camera and the first camera are not significantly different. In this case, a smaller degree of color correction can be applied to the current frame based on the smaller second sub-weight. Conversely, when the estimated color temperature difference is small, it indicates that the shooting effects of the second camera and the first camera are significantly different. In this case, a larger degree of color correction can be applied to the current frame based on the larger second sub-weight to mitigate the difference in shooting effects.

[0031] In one embodiment, the third sub-weight can be determined first based on the ambient brightness information of the shooting environment corresponding to the memory frame. Then, the fusion weight is determined based on the first, second, and third sub-weights. For example, the product of the first, second, and third sub-weights can be used as the fusion weight. For instance, the fusion weight, the white point information of the first target white point, and the white point information of the mapped white point satisfy the following formula: the white point information of the third target white point = the white point information of the second target white point * (1-W) + the white point information of the mapped white point * W; where W is the fusion weight.

[0032] In this context, the greater the brightness indicated by the ambient light information, the greater the third sub-weight. The ambient light information of the shooting environment corresponding to the memory frame is used to indicate the ambient brightness of the shooting environment corresponding to the memory frame, such as brightness parameters like LV. Generally, the lower the ambient brightness, the more complex the shooting environment, and the worse the shooting effect. The greater the ambient brightness, the better the shooting effect. The brightness indicated by the ambient light information is directly proportional to the third sub-weight; the greater the brightness indicated by the ambient light information, the greater the third sub-weight.

[0033] Thus, the lower the brightness indicated by the ambient light information, the worse the image quality of the first camera's memory frame. In this case, a smaller degree of color correction can be applied to the current frame based on a smaller third sub-weight. Conversely, the higher the brightness indicated by the ambient light information, the better the image quality of the first camera's memory frame. In this case, a larger degree of color correction can be applied to the current frame based on a larger third sub-weight. This strategy ensures the accuracy of color correction.

[0034] In one embodiment, white balance correction of the current frame is performed based on the white point information of the third target white point, including: determining the white balance gain based on the white point information of the third target white point; and correcting the current frame based on the white balance gain.

[0035] In one embodiment, the first estimated color temperature of the memory frame and the second estimated color temperature of the current frame can be fused to obtain a third estimated color temperature; based on the third estimated color temperature, color correction is performed on the current frame after white balance adjustment. This improves the accuracy of color correction, thereby further enhancing the color consistency of multi-camera images.

[0036] Secondly, a camera occlusion recognition device is provided, which has the function of implementing the camera occlusion recognition method described in the first aspect above. The camera occlusion recognition device includes at least one module, which is used to implement the camera occlusion recognition method provided in the first aspect above.

[0037] Thirdly, a camera occlusion recognition device is provided. The device includes a processor and a memory. The memory stores a program that enables the camera occlusion recognition device to execute the camera occlusion recognition method provided in the first aspect, and stores data related to implementing the camera occlusion recognition method described in the first aspect. The processor is configured to execute the program stored in the memory. The camera occlusion recognition device may further include a communication bus for establishing a connection between the processor and the memory.

[0038] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the camera occlusion recognition method described in the first aspect.

[0039] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the camera occlusion recognition method described in the first or second aspect above.

[0040] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the spatial distribution of a camera in an electronic device provided in an embodiment of this application;

[0042] Figure 2 This is a comparative diagram of images captured before and after camera switching, provided in an embodiment of this application.

[0043] Figure 3This is a schematic diagram illustrating the statistical differences in images captured before and after camera switching, as provided in an embodiment of this application.

[0044] Figure 4 This is a schematic diagram illustrating the statistical differences in images captured before and after camera switching, as provided in another embodiment of this application.

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0046] Figure 6 This is a block diagram of a software system for an electronic device provided in an embodiment of this application;

[0047] Figure 7 This is a logical schematic diagram of an image processing procedure provided in an embodiment of this application;

[0048] Figure 8 This is a schematic flowchart of a camera occlusion recognition method provided in an embodiment of this application;

[0049] Figure 9 This is a comparative schematic diagram showing the distribution of white dots in images captured by two main cameras and an image captured by a telephoto camera, as provided in the embodiments of this application.

[0050] Figure 10 This is a logical schematic diagram of a camera occlusion recognition method provided in an embodiment of this application;

[0051] Figure 11 This is a flowchart illustrating a multi-camera color correction algorithm provided in an embodiment of this application;

[0052] Figure 12 This is a schematic diagram showing the effect comparison before and after color correction of an image frame, provided in an embodiment of this application.

[0053] Figure 13 This is a schematic diagram of a process for image correction based on occlusion recognition results provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0055] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0056] To facilitate understanding, the names involved in the embodiments of this application will be explained first.

[0057] White dot: A dot on a white object. White refers to the visual perception formed by the equal proportions of blue, green, and red light reflected into the human eye, all possessing a certain brightness. A white object is any object that the human eye perceives as white in any scene, such as a white wall, a gray tabletop, or white paper. However, it should be noted that these white objects generally have color when photographed (i.e., their corresponding red (R), green (G), and blue (B) values ​​are not equal). Their color values ​​are determined by the light source, so what appears as white objects in a scene (to the human eye) is generally not white (image value) in the image. Like white objects, white dots also actually have color.

[0058] White point information: White point information is used to indicate the color information of the corresponding white point. For example, white point information can be the white point coordinates of the corresponding white point. In this application, the white point coordinates refer to the coordinates of the white point in a coordinate system with R / G as the abscissa and B / G as the ordinate, that is, the white point coordinates are (R / G, B / G).

[0059] Target white point: In this application, the target white point refers to the white point obtained by estimating the white point of an image using a white balance (WB) algorithm. For example, the WB algorithm can be an automatic white balance (AWB) algorithm, and correspondingly, the target white point can be called an AWB white point. For example, the AWB algorithm can be a gray-world algorithm, a perfect reflection algorithm, or a dynamic thresholding algorithm, etc., and this application does not limit this. The number of target white points in an image is generally one.

[0060] Statistical white points: Statistical white points refer to pixels in an image that meet the criteria for white points, identified by statistically analyzing the RGB values ​​of each pixel. For example, white points identified using the WB algorithm. The number of statistical white points in an image is typically multiple.

[0061] Color temperature: Color temperature is a measure of the color of light from a light source, measured in Kelvin (K). It is defined based on a blackbody; when the radiation from a light source in the visible region is identical to that of a blackbody, the temperature of the blackbody is called the color temperature of the light source. The higher the color temperature, the more bluish the white light appears; the lower the color temperature, the more yellow the white light appears.

[0062] White balance: Because the sensors in a camera cannot change their light-sensing characteristics according to changes in ambient light like the human eye, white will appear bluish or reddish in the camera's sensor under different color temperature light sources. White balance is the process of restoring the white image captured under different color temperature ambient light to true white (usually the white that the human eye perceives under natural sunlight).

[0063] White balance algorithms adjust the intensity of the R, G, and B color channels to make white appear more realistic. For example, for an image to be processed, the white points in the image can be calculated first, and then the white balance gain can be determined based on the white point information. The values ​​of the R, G, and B color channels of the image can then be adjusted according to the white balance gain. Additionally, white balance algorithms can also estimate the color temperature of the image, outputting the estimated color temperature, for example, by estimating the color temperature based on white point information.

[0064] The image processing method provided in this application is applicable to any electronic device with shooting capabilities and equipped with multiple cameras. This electronic device can be a mobile phone, tablet computer, camera, smart wearable device, or similar terminal; this application does not limit its application to this type. Furthermore, this application is primarily applied to single-camera shooting scenarios of electronic devices, i.e., scenarios where only one of the multiple cameras is used for shooting, such as switching from one camera to another for shooting.

[0065] As an example, different cameras among the multiple cameras configured in an electronic device have different shooting capabilities. For instance, the electronic device may be equipped with, but is not limited to, wide-angle cameras, telephoto cameras (such as periscope telephoto cameras), ultra-wide-angle cameras, and depth cameras.

[0066] As an example, an electronic device may be equipped with multiple cameras, including cameras located on different sides, or cameras located on the same side. That is, the embodiments of this application can be applied to scenarios where any camera (such as a front-facing camera or a rear-facing camera) on different sides is used for shooting, or to scenarios where any camera (such as a rear-facing camera) on the same side is used for shooting.

[0067] As an example, an electronic device can have multiple cameras on the same side, allowing any one of those cameras to be used for taking a picture. Similarly, an electronic device with multiple rear cameras on the back can use any one of them. Or, an electronic device with multiple front cameras on the front can use any one of them individually.

[0068] Typically, electronic devices are equipped with one main camera and at least one secondary camera. For example, please refer to... Figure 1 The spatial distribution of multiple cameras can be as follows: Figure 1 As shown in Figure (a), or, the spatial distribution of multiple cameras can also be as shown in Figure (a). Figure 1 As shown in Figure (b), the multiple cameras are camera 00, camera 01, camera 02 and camera 03. For example, camera 00 is the main camera and the others are auxiliary cameras.

[0069] When an electronic device launches its camera application, it typically defaults to using the main camera for shooting. It can then automatically switch to a secondary camera based on shooting needs or user input. For example, please refer to... Figure 1 By default, it takes pictures through camera 00, and then switches to camera 01, camera 02 or camera 03 to take pictures according to the user's switching operation.

[0070] When an electronic device switches from one camera to another for shooting, the camera used before the switch may be obstructed by the user's hand or other objects, such as fingers or palms. If the camera before the switch is obstructed, its image may not accurately reflect the scene and therefore may not meet the standard for correcting the image taken by the switched camera. For example, if the image taken by the previous camera is used as a reference image for color correction of the image taken by the switched camera, the corrected image may suffer from severe color cast.

[0071] To identify camera occlusion, some technologies propose a method based on the difference in image brightness between images captured before and after a camera switch. For example, a significant brightness difference indicates occlusion by the camera before the switch. However, if the camera before the switch is only partially or completely obstructed, the brightness difference between the images captured by the two cameras will be minimal. This makes it difficult to distinguish whether the camera before the switch was obstructing the view based solely on brightness differences, leading to incorrect occlusion identification. For example, please refer to... Figure 2 , Figure 2This is a comparative diagram showing images captured before and after camera switching, provided in an embodiment of this application. Figure 2 As shown, the electronic device first uses the main camera to take a picture, and then switches to the telephoto camera to take a picture. Figure 2 Image (a) is an image taken by the main camera when it is blocked (by the user's finger), with a light value (LV) of 40. Figure 2 Image (b) in the image is taken by the switched telephoto camera, with LV=56. Figure 2 Figure (a) in the middle and Figure 2 As can be seen from the comparison of Figure (b), the difference in brightness values ​​between the two is small. If the occlusion is identified based on the difference in image brightness before and after the camera switching, as is done in the relevant technology, it may be mistakenly judged that the main camera is not occluded, resulting in occlusion identification error.

[0072] To improve the accuracy of identifying whether a camera is occluded, this application provides a camera occlusion identification method. When an electronic device switches from a first camera to a second camera for shooting, it determines first statistical information of the memory frame of the first camera (the last frame captured by the first camera) and second statistical information of the current frame captured by the second camera. Then, based on the difference between the first and second statistical information, it determines the occlusion identification result of the first camera, which characterizes whether the first camera is occluded.

[0073] The first statistical information includes the target white point and multiple statistical white points in the current frame, and the second statistical information includes the target white point and multiple statistical white points in the memory frame. For ease of distinction, in this embodiment, the target white point in the current frame is referred to as the first target white point, and the target white point in the memory frame is referred to as the second white point; the statistical white point in the memory frame is referred to as the first statistical white point, and the statistical white point in the current frame is referred to as the second statistical white point.

[0074] Since image statistical information can more accurately reflect image features such as color temperature, and the statistical information of images captured by the corresponding camera will differ significantly depending on whether the camera is occluded or not, the difference in statistical information between images captured by the camera before and after switching can more accurately identify whether the camera was occluded before switching, thereby improving the accuracy of camera occlusion identification. Furthermore, the occlusion identification results identified in this application can more accurately correct images captured by the camera after switching, improving the accuracy of image correction. For example, it can, to some extent, avoid the severe color cast caused by using images captured by an occluded camera as reference images for color correction.

[0075] Please refer to Figure 3 , Figure 3This is a schematic diagram illustrating the statistical differences in images captured before and after a camera switch, as provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device first uses the main camera to take a picture, and then switches to the telephoto camera to take a close-up picture. Figure 3 Image (a) is an image taken by the main camera when there is no obstruction. Figure 3 Figure (b) shows the image taken by the switched telephoto camera; Figure 3 Figure (c) in the figure is a schematic diagram of the distribution of statistical white points in the image captured by the main camera. Figure 3 Figure (d) in the diagram shows the distribution of statistical white points in an image captured by a telephoto camera. The distribution diagram of statistical white points refers to the coordinate diagram of multiple statistical white points in the corresponding image in a coordinate system with rg(R / G) as the abscissa and bg(B / G) as the ordinate. For example... Figure 3 Figure (c) in the middle and Figure 3 As shown in Figure (d), when the main camera is not obstructed, the distribution of statistical white points in the images captured by the main camera and the images captured by the telephoto camera is not significantly different.

[0076] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the statistical differences in images captured before and after camera switching, as provided in another embodiment of this application. For example... Figure 4 As shown, the electronic device first uses the main camera to take a picture, and then switches to the telephoto camera to take a picture. Figure 4 Image (a) is an image taken by the main camera when the image is obstructed; Figure 4 Figure (b) shows an image taken by the telephoto camera after switching. Figure 4 Figure (c) in the figure is a schematic diagram of the distribution of statistical white points in the image captured by the main camera. Figure 4 Figure (d) in the diagram is a schematic diagram of the distribution of statistical white points in an image captured by a telephoto camera. For example... Figure 4 As shown in Figure (a), when the main camera is obstructed, the captured image is close to a solid color scene; correspondingly, as Figure 4 As shown in Figure (c), the vertical coordinates of the statistical white points in the captured image do not change much in a coordinate system with rg as the abscissa and bg as the ordinate, meaning that the B / G values ​​of each statistical white point do not change much, and the image color is close to red (reddish). Figure 4 As shown in Figure (d), the distribution of statistical white points in images captured by the telephoto camera is the normal distribution under conditions where the camera is not obstructed. Figure 4 Figure (c) in the middle and Figure 4As can be seen from Figure (d) in the diagram, when the main camera is obstructed, there is a significant difference in the distribution of statistical white points between the images captured by the main camera and those captured by the telephoto camera. In other words, the statistical information of the images captured before and after the camera switch is significantly different. Therefore, this application uses the difference in statistical information between the images captured before and after the camera switch to identify the obstruction of the camera before the switch, thus providing a more accurate way to determine whether the camera was obstructed before the switch.

[0077] It should be noted that the camera occlusion recognition method provided in this application will be explained below. Figure 8 The embodiments are described in detail below, but the embodiments of this application will not be repeated here.

[0078] For ease of explanation, the following will use an electronic device with multiple cameras as an example.

[0079] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 5 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0080] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0081] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0082] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0083] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0084] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0085] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0086] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0087] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0088] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0089] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is an integer greater than 1.

[0090] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0091] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 193.

[0092] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is an integer greater than 1.

[0093] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0094] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0095] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0096] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0097] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created by electronic device 100 during use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0098] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D and application processor.

[0099] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch operation intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than the pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0100] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch display." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0101] The software system of electronic device 100 will be described next.

[0102] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to illustrate the software system of electronic device 100.

[0103] Figure 6 This is a block diagram of a software system for an electronic device 100 provided in an embodiment of this application. See also... Figure 6 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, such as... Figure 6As shown, the system architecture of electronic device 100 includes an application layer 510, an application framework layer 520, a hardware abstraction layer (HAL) 530, and a driver layer 540.

[0104] Understandable. Figure 6 As an example only, the layers in electronic device 100 are not limited to... Figure 6 The layers shown, for example, between the application framework layer and the HAL layer, may also include the Android runtime and system library layers.

[0105] Application layer 510 may include a series of application packages. For example... Figure 6 As shown, the application package may include a camera, gallery, and other applications, including but not limited to: calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS applications.

[0106] The application framework layer 520 provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer 520 includes some predefined functions. For example, the application framework layer 520 may include a camera access interface. The camera access interface may include camera management and camera devices. Specifically, camera management can provide an access interface for managing the camera; camera devices can provide an interface for accessing the camera.

[0107] In addition, the application framework layer 520 may also include a content provider, a resource manager, a notification manager, a window manager, a view system, a phone manager, etc. Similarly, the camera application may also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs. This application embodiment does not impose any restrictions on this.

[0108] The hardware abstraction layer 530 is used to abstract hardware, such as encapsulating drivers in the driver layer and providing an interface for the application framework layer to call them, thus shielding the implementation details of the low-level hardware. For example, the hardware abstraction layer 530 can encapsulate the camera hardware abstraction layer (Camera HAL) and other hardware device abstraction layers. The camera hardware abstraction layer can connect to an algorithm library to call algorithms from that library.

[0109] For example, please refer to Figure 6The algorithm library may include a camera occlusion recognition module, which integrates the camera occlusion recognition algorithm provided in this application embodiment to identify whether a specific camera is occluded. Additionally, the algorithm library may include a color correction module, which integrates a color correction algorithm to perform color correction on the image captured by the current camera based on an image captured by a reference camera (e.g., the camera before switching). For example, the color correction module integrates the multi-camera color correction algorithm provided in this application embodiment to perform color correction on the image frame captured by the current camera. As an example, when the electronic device switches from a first camera to a second camera for shooting, the camera occlusion recognition module can trigger the color correction module to perform color correction on the image captured by the current second camera based on the image captured by the first camera before switching if it detects that the camera before switching is not occluded. However, if it detects that the camera before switching is occluded, the color correction module will not trigger to perform color correction on the image captured by the current second camera based on the image captured by the first camera before switching. For ease of explanation, the first camera before switching can also be referred to as the reference camera.

[0110] In addition, the algorithm library may also include an automatic white balance (AWB) module and other image processing modules. The AWB module integrates the AWB algorithm, which calculates statistical information of the image frame captured by the current camera and sends this statistical information to the color correction module. The color correction module then processes the statistical information of the current image frame using an appropriate color correction algorithm based on the statistical information of the memory frame (the last frame) captured by the reference camera, obtaining corrected statistical information, which is then returned to the AWB module. The AWB module can then perform white balance correction on the current image frame based on the corrected statistical information, resulting in an AWB image frame. Furthermore, the AWB module can also send the AWB image frame and the corrected statistical information to other image processing modules, allowing these modules to perform image processing on the AWB image frame based on the corrected statistical information, such as color correction or lens shading correction.

[0111] As an example, please refer to Figure 7When an electronic device switches from a reference camera to the current camera for shooting, the AWB module can use the AWB algorithm to calculate statistical information of the image frames captured by the current camera and send this statistical information to the camera occlusion recognition module. The camera occlusion recognition module uses the statistical information of the image frames captured by the current camera and the pre-stored statistical information of the reference camera's captured frames to identify whether the reference camera was occluded before the switch. If it is determined that the reference camera was not occluded before the switch, the AWB module is triggered to send specific statistical information of the current image frame calculated using the AWB algorithm (such as AWB white point and AWB color temperature) to the color correction module, so that the color correction module can perform color correction on the current image frame based on the specific statistical information of the memory frames of the reference camera's captured images before the switch. For example, the color correction module performs a fusion process based on the AWB color temperature difference between the memory frame and the current image frame. This involves fusing the AWB white points of the memory frame and the current image frame to obtain a fused white point, and fusing the AWB color temperatures of the memory frame and the current image frame to obtain a fused color temperature. The fused white point and fused color temperature are then fed back to the AWB module, allowing it to perform white balance correction on the current image frame, resulting in an AWB image frame. Furthermore, the AWB module can send the fused color temperature and AWB image frame to other image processing modules for further processing, such as color correction or lens shading correction. Additionally, if the camera occlusion detection module detects occlusion by the reference camera before the switch, it will not trigger the AWB module to send specific statistical information (such as AWB white point and AWB color temperature) of the current image frame calculated using the AWB algorithm to the color correction module, thus preventing the color correction module from performing color correction on the current image frame.

[0112] The driver layer 540 is used to provide drivers for different hardware devices. For example, the driver layer 540 may include camera device drivers, digital signal processor drivers, and graphics processor drivers.

[0113] Additionally, the hardware layer 550 includes driveable hardware modules, such as camera devices. For example, a camera device may include multiple cameras, such as camera 1, camera 2, ..., camera n. Furthermore, the camera device may also include multispectral sensors, time-of-flight (TOF) sensors, etc., but this embodiment does not limit the scope of the application.

[0114] In this application, by calling the hardware abstraction layer interface in the hardware abstraction layer 530, the application layer 510 and application framework layer 520 above the hardware abstraction layer 530 can be connected with the driver layer 550 and hardware layer 550 below, so as to realize camera data transmission and function control.

[0115] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 100.

[0116] The camera application in application layer 510 can be displayed as an icon on the screen of electronic device 100. When the user clicks the camera application icon to trigger it, electronic device 100 starts running the camera application. When the camera application is running on electronic device 100, it calls the corresponding interface of the camera application in application framework layer 520, and then calls hardware abstraction layer 530 to start the camera device driver, turning on any camera 193 on electronic device 100 to capture images. For example, camera hardware abstraction layer 530 can send a command to camera device driver to call a certain camera to capture images through the called camera.

[0117] Taking the default use of the main camera upon camera app launch as an example, after the main camera is invoked, for instance, when the main camera captures the last frame (memory frame), a multispectral sensor can be invoked to obtain the ambient color temperature at the time the memory frame was captured. Simultaneously, the image signal processor is invoked to perform white balance processing on the memory frame captured by the main camera, and statistical information (such as target white point, statistical white point, estimated color temperature, etc.) of the memory frame calculated using the white balance algorithm is obtained during the white balance processing. This statistical information and ambient color temperature information of the memory frame are then stored. Subsequently, when any camera other than the main camera in the hardware layer is invoked, i.e., when the main camera is switched to another camera, during the current camera's capture process, a multispectral sensor can be invoked to obtain the ambient color temperature at the time the current frame was captured. Simultaneously, the image signal processor is invoked to perform white balance processing on the current frame captured by that camera, and statistical information of the current frame calculated using the white balance algorithm is obtained during the white balance processing. Then, the statistical information of the memory frame is obtained, and based on the difference between the statistical information of the memory frame and the statistical information of the current frame, it is determined whether the main camera is obstructed (i.e., whether the main camera was obstructed when capturing the memory frame). If it is determined that the main camera is not obstructed, color correction is performed on the current image frame based on the memory frame. For example, based on the difference between the ambient color temperature of the memory frame and the ambient color temperature of the current frame, the white point information of the target white point in the memory frame and the target-calibrated white point information in the current frame are fused. The fused white point information is fed back to the image signal processor, so that the image signal processor can perform white balance correction and other processing on the current frame based on the fused white point information to obtain the target image. Then, the image signal processor returns the target image to the camera hardware abstraction layer via the camera device driver. The camera hardware abstraction layer further processes the target image and sends the processed image back to the camera application for display and storage via the camera access interface. In addition, if it is determined that the main camera is obstructed, the above-mentioned step of color correction of the current image frame based on the memory frame will not be performed. That is, the image signal processor can directly perform white balance correction and other processing on the current frame based on the statistical information of the current frame calculated by the white balance algorithm to obtain the target image. Then, the image signal processor returns the target image to the camera hardware abstraction layer through the camera device driver. The camera hardware abstraction layer further processes the target image and sends the processed image back to the camera application for display and storage through the camera access interface.

[0118] The execution subject of the image processing method provided in this application embodiment can be the aforementioned electronic device, or a functional module and / or functional entity in the electronic device that can implement the image processing method. Furthermore, the solution of this application can be implemented by hardware and / or software, and the specific implementation can be determined according to actual usage requirements. This application embodiment does not impose any limitations.

[0119] Next, the occlusion recognition method of the camera provided in the embodiments of this application will be described in detail.

[0120] Figure 8 This is a flowchart illustrating a camera occlusion recognition method provided in an embodiment of this application, as shown below. Figure 8 As shown, the method includes the following steps:

[0121] Step A1: When the electronic device switches from the first camera to the second camera for shooting, determine the first statistical information of the memory frame of the first camera and the second statistical information of the current frame captured by the second camera.

[0122] The scenario of switching from the first camera to the second camera for shooting can include: a scenario of switching directly from the first camera to the second camera for shooting, or a scenario of first switching from the first camera to another camera and then switching from the other camera to the second camera. This application embodiment does not limit this.

[0123] The first camera can be any one of the multiple cameras configured in the electronic device, and the second camera can be any one of the multiple cameras other than the first camera.

[0124] As an example, one of the multiple cameras configured on an electronic device can be pre-set as a reference camera. This allows for color correction of images captured by the other cameras, using the image parameters of the reference camera as a benchmark. This ensures that the colors of images captured by the other cameras approximate those of the reference camera, thereby reducing color differences and improving color consistency across multiple cameras capturing the same scene. The reference camera can be any of the multiple cameras, such as the camera that the camera application uses by default. Typically, the camera application uses the main camera by default upon startup, so the main camera can be set as the reference camera.

[0125] Accordingly, the first camera is a pre-set reference camera. That is, in this embodiment of the application, when the electronic device switches from the reference camera to other cameras for shooting, the occlusion recognition method of the camera provided in this embodiment of the application can be used to determine whether the reference camera is occluded.

[0126] The memory frame refers to the last frame captured by the first camera, such as the last frame of the first camera preview. That is, this embodiment can identify whether occlusion occurs when the first camera captures the last frame before switching to another camera. It should be understood that the memory frame can also be replaced with other image frames captured by the first camera as needed to identify whether occlusion occurs when the first camera captures other image frames; for example, the memory frame can be replaced with the first frame captured by the first camera. This embodiment does not limit this.

[0127] As an example, the current frame could be the first frame captured by the second camera, such as the first frame of a preview. That is, after the electronic device switches from the first camera to the second camera for shooting, it can identify whether the first camera is occluded based on the statistical information of the last frame captured by the first camera and the statistical information of the first frame captured by the second camera. Based on the occlusion identification result, it can determine whether to initiate the image correction process, that is, whether to use the memory frame of the first camera as a reference image to correct the image frame captured by the second camera.

[0128] The first statistical information includes a first target white point and multiple first statistical white points in the memory frame, such as the coordinates of the first target white point and multiple first statistical white points. The second statistical information includes a second target white point and multiple second statistical white points in the current frame, such as the coordinates of the second target white point and multiple second statistical white points.

[0129] After determining the first statistical information of the memory frame of the first camera and the second statistical information of the current frame captured by the second camera, the occlusion recognition result of the first camera can be determined based on the difference between the first and second statistical information. The occlusion recognition result indicates whether the first camera is occluded.

[0130] As an example, the operation of determining the occlusion recognition result of the first camera based on the difference between the first statistical information and the second statistical information can be implemented through the following steps A2-A8.

[0131] Step A2: Based on the first statistical information, determine the target quantity ratio. The target quantity ratio is the ratio of the target quantity to the total number of multiple first statistical white points. The target quantity refers to the number of first statistical white points whose coordinates are within a range centered on the coordinates of the first target white point and with a preset distance as the radius.

[0132] The preset distance can be set in advance as needed. For example, the preset distance can be an empirical value obtained by statistically analyzing the target white point and statistical white point in images captured by the first camera under multiple standard light sources. Multiple standard light sources can be specified as needed. Standard light sources are light sources defined by standards, such as those specified by the International Commission on Illumination (ICI) for standardizing color detection. These multiple standard light sources have different color temperatures. For example, these multiple standard light sources can be at least two of the following: D75, D65, D50, CWF, TL84, U30, A, and H. Other standard light sources can also be included, and this application does not limit this. The color temperatures of the standard light sources D75, D65, D50, CWF, TL84, U30, A, and H decrease sequentially: 7500K, 6500K, 5000K, 4150K, 4100K, 3000K, 2856K, and 2300K, respectively.

[0133] As an example, from multiple standard light sources, the second and second-to-last standard light sources can be selected in descending order of color temperature. The preset distance is determined based on the difference between the target white point and / or statistical white point of the first camera under these two standard light sources. For instance, the preset distance can be determined based on the coordinate distance of the target white point in the images captured by the first camera under these two standard light sources. It should be understood that other methods can also be used to determine the preset distance, and this application embodiment does not limit this approach.

[0134] The target number ratio indicates the degree of clustering of multiple first statistical white points around a first target white point. A larger target number ratio indicates a higher degree of clustering of multiple first statistical white points around the first target white point, meaning the first statistical white points are closer to the first target white point. A smaller target number ratio indicates a lower degree of clustering of multiple first statistical white points around the first target white point, meaning the first statistical white points are farther from the first target white point.

[0135] Taking the switching from the main camera to the telephoto camera as an example, please refer to... Figure 9 , Figure 9 This is a comparative schematic diagram showing the distribution of white dots in images captured by two main cameras and an image captured by a telephoto camera, as provided in the embodiments of this application. Figure 9 Figure (a) in the figure shows a comparison of the white point distribution in an image captured by the main camera without obstruction and the white point distribution in an image captured by the telephoto camera. Figure 9 As shown in Figure (a), when the main camera is not obstructed, the number of statistical white points in the captured image that fall within the circle centered on the target white point and with a preset distance r as the radius is relatively large. Figure 9 Figure (b) shows a comparison of the white point distribution in an image captured by the main camera under obstruction conditions with that captured by the telephoto camera. Figure 9 As shown in Figure (b), when the main camera is blocked, the image it captures is basically a solid color. Therefore, the B / G values ​​of the statistical white points are similar, and the statistical white points are basically distributed in a straight line. As a result, the statistical white points are far away from the target white points, and the number of statistical white points falling into the circle with the target white point as the center and a preset distance r as the radius is small.

[0136] It should be noted that when the main camera is obstructed, the captured image is essentially a solid color. For example, when the camera is obstructed by a hand, the captured image is reddish, and the B / G values ​​of the statistical white points are similar. The statistical white points are generally distributed horizontally in a coordinate system with R / G as the x-axis and B / G as the y-axis. This distribution results in the statistical white points being far from the target white point, and the number of statistical white points falling within a circle centered on the target white point with a preset distance r as the radius is small. Therefore, this embodiment can determine whether the first camera is obstructed based on the target number ratio.

[0137] For example, the target quantity ratio can be represented by refNumRatio, which is the ratio of the target quantity to the total number of multiple first statistical white points.

[0138] Step A3: Determine whether the target quantity ratio is less than the first threshold.

[0139] The first threshold can be set in advance as needed, and is generally a small value, such as 0.06, 0.08 or 0.1.

[0140] When the target quantity ratio is less than the first threshold, it indicates that the statistical white points in the memory frame are far from the target white points, and the first camera may be obstructing the view. When the target quantity ratio is greater than or equal to the first threshold, it indicates that the statistical white points in the memory frame are close to the target white points, and the first camera is not obstructing the view.

[0141] For example, if refNumRatio < 0.08, it is determined that the first camera may be obstructed; if refNumRatio ≥ 0.08, it is determined that the first camera is not obstructed.

[0142] Step A4: If the target quantity ratio is greater than or equal to the first threshold, then it is determined that the first camera is not obstructed.

[0143] Step A5: If the target quantity ratio is less than the first threshold, then determine the first ratio and the second ratio based on the first statistical information and the second statistical information. The first ratio is the ratio of the average of the first color statistical values ​​of multiple first statistical white points to the average of the second color statistical values, and the second ratio is the ratio of the average of the first color statistical values ​​of multiple second statistical white points to the average of the second color statistical values.

[0144] The first color statistical value is the ratio of the R value to the G value of the corresponding white point, and the second color statistical value is the ratio of the B value to the G value of the corresponding white point.

[0145] In this embodiment, when the target number ratio is less than a first threshold, it can be preliminarily determined that the first camera may be obstructed. Then, other information can be combined to further determine whether the first camera is obstructed. For example, a first ratio and a second ratio can be determined based on the first statistical information and the second statistical information, and the obstruction of the first camera can be determined based on the first ratio and the second ratio.

[0146] For example, determining the first ratio and the second ratio based on the first statistical information and the second statistical information may include the following steps:

[0147] 1) Calculate the R / G value and B / G value of each first statistical white point in the multiple first statistical white points, and the R / G value and B / G value of each second statistical white point in the multiple second statistical white points.

[0148] 2) Calculate the mean R / G value refRgAvg and the mean B / G value refBgAvg for multiple first statistical white points, and calculate the mean R / G value curRgAvg and the mean B / G value curBgAvg for multiple second statistical white points.

[0149] 3) Calculate the first ratio refDeltaMean = refRgAvg / refBgAvg and the second ratio curDeltaMean = curRgAvg / curBgAvg respectively.

[0150] As an example, after determining that the target quantity ratio is less than a first threshold, it can be further determined whether the target quantity ratio is less than or equal to a second threshold. If so, it is determined that the first camera is obstructed; if not, a first ratio and a second ratio are further determined based on the first and second statistical information, and the ratio of the first ratio to the second ratio is used to determine whether the first camera is obstructed. This can improve recognition efficiency.

[0151] The second threshold is less than the first threshold. For example, the second threshold can be 0.01 or 0.

[0152] Step A6: Determine whether the ratio of the first ratio to the second ratio is greater than the first ratio threshold, or whether the ratio of the first ratio to the second ratio is less than the second ratio threshold.

[0153] The first proportional threshold is greater than 1, such as 2 or 3. The second proportional threshold is less than 1, such as 0.5 or 0.3. The first and second proportional thresholds can be set as needed, and this embodiment does not limit them.

[0154] If the ratio of the first ratio to the second ratio is greater than the first ratio threshold, it indicates that the average R / G value of the statistical white points in the image captured by the first camera is too high, and the image is too red, thus confirming that the first camera is occluded. If the ratio of the first ratio to the second ratio is less than the second ratio threshold, it indicates that the average B / G value of the statistical white points in the image captured by the first camera is too high, and the image is too blue, thus confirming that the first camera is occluded. If the ratio of the first ratio to the second ratio is greater than or equal to the second ratio threshold and less than or equal to the first ratio threshold, it indicates that the RGB values ​​of the statistical white points in the image captured by the first camera are within the normal range, thus confirming that the first camera is not occluded.

[0155] Step A7: If the ratio of the first ratio to the second ratio is greater than the first ratio threshold, or if the ratio of the first ratio to the second ratio is less than the second ratio threshold, then it is determined that the first camera is obstructed.

[0156] For example, assuming the first and second proportional thresholds are 2 and 0.5 respectively, if refDeltaMean > 2*curDeltaMean or refDeltaMean < 0.5*curDeltaMean, then it is determined that the first camera is obstructed.

[0157] In addition, if the ratio of the first ratio to the second ratio is greater than or equal to the second ratio threshold and less than or equal to the first ratio threshold, then it is determined that the first camera is not obstructed.

[0158] For example, if 0.5*curDeltaMean≥refDeltaMean≤2*curDeltaMean, then it is determined that the first camera is not obstructed.

[0159] Please refer to Figure 10 , Figure 10 This is a logical schematic diagram of a camera occlusion recognition method provided in an embodiment of this application. For example... Figure 10As shown, assuming the electronic device switches from the main camera to the telephoto camera for shooting, the target quantity ratio (refNumRatio) can be calculated first based on the statistical information of the main camera's memory frames. Then, it is determined whether refNumRatio is less than 0.08. If not, it is determined that the main camera is not obstructed. If so, it is determined whether refNumRatio is equal to 0. If so, it is determined that the main camera is obstructed. If not, it is determined whether refDeltaMean is calculated for the main camera and curDeltaMean for the telephoto camera. Then, it is determined whether refDeltaMean is greater than 2*curDeltaMean or less than 0.5*curDeltaMean. If so, it is determined that the main camera is obstructed; otherwise, it is determined that the main camera is not obstructed.

[0160] It should be understood that the embodiments of this application are only used as an example to illustrate how the occlusion recognition result of the first camera is determined based on the difference between the first statistical information of the image captured by the first camera and the second statistical information of the image captured by the second camera after the first camera switches to the second camera. In other embodiments, it can be assumed that the first camera is not occluded, and the occlusion recognition result of the second camera is determined based on the difference between the first statistical information of the image captured by the first camera and the second statistical information of the image captured by the second camera. The embodiments of this application do not limit this.

[0161] In this embodiment, the difference in statistical information between images captured before and after camera switching can be used to identify whether a camera is obstructed. Since image statistical information can more accurately reflect image features such as color temperature, and the statistical information of images captured by the corresponding camera will differ significantly depending on whether the camera is obstructed or not, the difference in statistical information between images captured before and after switching can more accurately identify whether the camera was obstructed before switching, thereby improving the accuracy of identifying whether a camera is obstructed.

[0162] Furthermore, in this embodiment, the image correction process can be determined based on the camera recognition result, i.e., whether to use the memory frame of the first camera as a reference image to correct the image frame captured by the second camera. Since the camera occlusion recognition method provided in this embodiment can more accurately identify whether a camera is occluded, the occlusion recognition result identified by this application can more accurately correct the image captured by the switched camera, improving the accuracy of image correction. For example, it can, to some extent, avoid the severe color cast caused by using the image captured by the occluded camera as a reference image for color correction.

[0163] As an example, based on the color temperature difference between the first ambient color temperature of the memory frame corresponding to the shooting environment of the first camera and the second ambient color temperature of the current frame captured by the second camera, the white point information of the first target white point in the memory frame and the white point information of the second target white point in the current frame can be fused, and white balance correction can be performed on the current frame based on the fused white point information.

[0164] Next, taking the first camera as a pre-set reference camera as an example, we will illustrate the application scenarios involved in image correction provided in the embodiments of this application.

[0165] When electronic devices are equipped with multiple cameras, if the sensors of each camera respond differently to color, the colors in the images captured by different cameras in the same scene may vary. This visual difference caused by color variations in multi-camera shots of the same scene will affect the user's shooting experience. For example, when a user switches cameras to shoot in the same scene, the colors of the images captured before and after the switch may differ. The user will see the same object in inconsistent colors in the previous and subsequent shots, resulting in a poor visual experience. Furthermore, this situation can cause confusion for the user, potentially leading them to doubt their shooting operation and negatively impacting their shooting experience.

[0166] Please refer to Figure 3 In the same scene, users can first use the main camera to take pictures, and then switch to the telephoto camera to take pictures. Figure 3 Image (a) in the image is the one taken by the main camera. Figure 3 Figure (b) shows an image taken by the telephoto camera. Comparing the image taken by the main camera with the image taken by the telephoto camera, it can be seen that the field of view (FOV) of the image taken by the telephoto camera is smaller than that of the image taken by the main camera. Moreover, there is a large color difference between the two images. The same object in the shooting scene (such as a street lamp) appears in different colors in the two images, resulting in a poor visual experience for the user.

[0167] To address color differences in multi-camera shooting within the same scene, this application also provides a multi-camera color correction method. In this method, one of the multiple cameras configured in an electronic device can be pre-set as a reference camera. The white point information of the first target white point in the memory frame (i.e., the last frame captured) of the reference camera, along with the first ambient color temperature of the shooting environment corresponding to the memory frame, is stored. Based on the relevant information of the memory frame of the reference camera, color correction is performed on images captured by other cameras. For example, when the electronic device uses any other camera besides the reference camera to capture an image, the white point information of the second target white point in the current frame captured by the other camera, along with the second ambient color temperature of the shooting environment corresponding to the current frame, can be obtained. Then, based on the color temperature difference between the first and second ambient color temperatures of the shooting environment corresponding to the memory frame of the reference camera, the white point information of the first target white point in the memory frame and the white point information of the second target white point in the current frame are fused. White balance correction is then performed on the current frame based on the fused white point information.

[0168] In this way, when the difference between the ambient color temperature of other cameras and the ambient color temperature of the reference camera is small, it can be determined that the shooting scene is relatively similar and that the shots are likely taken in the same scene. By combining the target white point information of the memory frame of the reference camera and the target white point information of the current frame, the color of the current frame is corrected by performing white balance correction, thereby reducing the color difference between the current frame and the memory frame of the reference camera. This reduces the color difference between the images taken by other cameras and the reference camera in the same scene and the resulting visual differences, improving the color consistency of multi-camera shooting in the same scene, and thus improving the user's visual and shooting experience.

[0169] It should be noted that the specific algorithm of this multi-camera color correction method will be explained below. Figure 11 The embodiments are described in detail below, but the embodiments of this application will not be repeated here.

[0170] It should be understood that the above-mentioned camera occlusion recognition method can also be applied to other application scenarios, such as real-time detection of the camera's captured images, and triggering camera occlusion warnings when abnormal situations such as occlusion occur.

[0171] As an example, one of the multiple cameras configured in an electronic device can be pre-set as a reference camera. This allows for color correction of images captured by the other cameras, using the image parameters of the reference camera as a benchmark. This ensures that the colors of images captured by the other cameras approximate those of the reference camera, thereby reducing color differences and improving color consistency across multiple cameras capturing the same scene. The reference camera can be any one of the multiple cameras. In one example, the camera that the camera application launches by default can be set as the reference camera. For instance, the main camera is typically the one launched by default in a camera application, so it can be set as the reference camera.

[0172] Figure 11 This is a flowchart illustrating a multi-camera color correction algorithm provided in an embodiment of this application, as shown below. Figure 11 As shown, the method includes the following steps:

[0173] Step B1: Obtain and store the white point information of the first target white point of the memory frame of the reference camera, and the first ambient color temperature of the shooting environment corresponding to the memory frame.

[0174] In this embodiment, when the camera application detects that it has switched from a reference camera to another camera for shooting, the ambient color temperature (i.e., the first ambient color temperature) of the shooting environment corresponding to the last frame (i.e., the memory frame) captured by the reference camera, and the white point information of the white point of the captured memory frame (i.e., the white point information of the first target white point) are obtained, and the white point information of the first target white point and the first ambient color temperature of the memory frame are stored. For example, the memory frame can be the last frame of the reference camera preview.

[0175] The camera switching event, which switches from the reference camera to other cameras for shooting, can be triggered by the user's camera switching operation or by the camera application automatically according to shooting needs. This application embodiment does not limit this.

[0176] The second target white point can be the AWB white point of the memory frame, that is, the white point represented by the white point information output by the AWB module in the image signal processor after estimating the white point of the memory frame. The white point information of the second target white point is used to indicate the second target white point and can be the color information of the second target white point, such as the white point coordinates (R / G, B / G). The white point coordinates refer to the coordinates in a coordinate system with R / G as the abscissa and B / G as the ordinate. The first ambient color temperature is used to indicate the color temperature of the corresponding shooting environment and can be the correlated color temperature (CCT) of the shooting environment. For example, the first ambient color temperature is the color temperature detected by a multispectral sensor under the corresponding shooting environment.

[0177] Furthermore, other information corresponding to the memory frame can also be acquired and stored, such as one or more of the following: the first estimated color temperature of the memory frame, and the brightness information of the shooting environment corresponding to the memory frame.

[0178] The first estimated color temperature is the color temperature obtained by estimating the color temperature of the memory frame, such as the color temperature estimated based on the first target white point. For example, the first estimated color temperature can be obtained by estimating the color temperature of the memory frame using a white balance algorithm based on the first target white point. For instance, the first estimated color temperature can be the AWB color temperature, which is the color temperature output by the AWB module in the image signal processor after estimating the color temperature of the memory frame.

[0179] The brightness information indicates the ambient brightness of the corresponding shooting environment. For example, the brightness information can be a light value (LV), or other parameters used to measure ambient brightness. This brightness information can be obtained by performing brightness statistics on the memory frames. For instance, this brightness information is the output of the auto exposure (AE) module in the image signal processor, which performs brightness statistics on the memory frames.

[0180] Step B2: After the second camera, other than the reference camera, is called to take pictures, obtain the white point information of the second target white point in the current frame captured by the second camera, as well as the second ambient color temperature of the shooting environment corresponding to the current frame.

[0181] Here, the second camera is the current camera, and can be any of multiple cameras other than the reference camera. For example, the second camera can be the current camera after a camera switch. For instance, the scenarios in which the second camera is invoked can include any of the following: switching directly from the reference camera to the second camera for shooting; or first switching from the reference camera to another camera, and then switching from that camera to the second camera for shooting, etc. For instance, the current frame can be the current frame of the real-time preview from the second camera.

[0182] The second target white point can be an AWB white point, which is the white point represented by the white point information output by the AWB module in the image signal processor after estimating the white point of the memory frame. The white point information of the second target white point is used to indicate the second target white point and can be its color information, such as its white point coordinates (R / G, B / G). The second ambient color temperature is used to indicate the color temperature of the corresponding shooting environment and can be the correlated color temperature (CCT) of the shooting environment. For example, the second ambient color temperature is the color temperature detected by a multispectral sensor under the corresponding shooting environment.

[0183] Furthermore, other information corresponding to the current frame captured by the second camera can also be obtained, such as the second estimated color temperature of the current frame. The second estimated color temperature is the color temperature obtained by estimating the color temperature of the current frame, for example, it is the color temperature estimated based on the white point of the second target. For example, the second estimated color temperature can be the color temperature obtained by estimating the color temperature of the current frame using a white balance algorithm. For example, the second estimated color temperature is the AWB color temperature, that is, the color temperature output by the AWB module in the image signal processor after estimating the color temperature of the current frame.

[0184] Step B3: Map the first target white point of the memory frame of the reference camera to the second camera to obtain the mapped white point.

[0185] By first mapping the first target white point of the memory frame of the reference camera to the second camera to obtain the mapped white point, and then fusing the white point information of the mapped white point with the white point information of the second target white point in the current frame, the accuracy of multi-camera color correction can be further improved.

[0186] As an example, the operation of mapping a first target white point from a memory frame of a reference camera to a second camera may include the following steps:

[0187] 1) Obtain calibration data from the reference camera and the second camera. The calibration data for each camera includes white point information and color temperature of each camera under multiple standard light sources.

[0188] In this embodiment of the application, for multiple cameras configured in an electronic device, the white point information and color temperature of each camera under multiple standard light sources can be pre-calibrated, and the calibration data of each camera can be stored.

[0189] Multiple standard light sources can be specified as needed. Standard light sources are light sources defined by standards, such as those specified by the International Commission on Illumination (ICI) for standardizing color detection. These multiple standard light sources have different color temperatures. For example, these multiple standard light sources can be at least two of the following: D75, D65, D50, CWF, TL84, U30, A, and H. Other standard light sources can also be included, but this application does not limit this. The color temperatures of the standard light sources D75, D65, D50, CWF, TL84, U30, A, and H decrease sequentially: 7500K, 6500K, 5000K, 4150K, 4100K, 3000K, 2856K, and 2300K.

[0190] As an example, when calibrating the white point information and color temperature of a camera under multiple standard light sources, the camera can be used to take pictures under each standard light source, and the white point information of the captured images can be used as the white point information of the camera under the corresponding standard light source. The color temperature of the camera under each standard light source can be the estimated color temperature of the images captured by the camera under each standard light source, the color temperature detected by an illuminance meter under the corresponding standard light source, or the color temperature of the corresponding standard light source. This application does not limit this. For example, assuming that the color temperature of each standard light source is used as the color temperature of the camera under each standard light source, the color temperatures of the camera under the standard light sources D75, D65, D50, CWF, TL84, U30, A, and H are 7500K, 6500K, 5000K, 4150K, 4100K, 3000K, 2856K, and 2300K, respectively.

[0191] 2) Determine at least one standard light source from a plurality of standard light sources based on the color temperature difference between the first estimated color temperature of the memory frame of the reference camera and the color temperature difference between ... reference camera.

[0192] That is, at least one standard light source is determined from multiple standard light sources whose corresponding color temperature is closest to the first estimated color temperature. The number of these at least one standard light source can be preset, such as 1, 2 or 3, etc., and this application embodiment does not limit this.

[0193] For example, assuming there are at least two standard light sources, the two standard light sources ranked first can be determined from these multiple standard light sources in ascending order of the color temperature difference between their corresponding color temperatures and the first estimated color temperature. That is, the two standard light sources whose color temperatures are closest to the first estimated color temperature are determined from the multiple standard light sources.

[0194] 3) Determine the white point mapping matrix between the reference camera and the second camera based on the white point information of the reference camera under at least one standard light source and the white point information of the second camera under at least one standard light source.

[0195] As an example, the logarithm of the white point information of the reference camera under at least one standard light source and the logarithm of the white point information of the second camera under at least one standard light source can be taken. Based on the logarithm results, the white point mapping matrix between the reference camera and the second camera can be determined. The logarithm can be either the common logarithm (i.e., log) or the natural logarithm (i.e., ln), and this embodiment does not limit the choice.

[0196] For example, assuming that the number of at least one standard light source is 2, and these two standard light sources are denoted as L1 and L2, and the white point information is the white point coordinates (R / G, B / G), then the white point mapping matrix between the reference camera and the second camera can be determined by the following formula (1):

[0197]

[0198] Where T is the white point mapping matrix. W dst It is the logarithmic result of the coordinates of the white point of the reference camera under L1 and L2, that is... in, It is the logarithmic result of the white point coordinates (R / G, B / G) of the reference camera under L1; It is the logarithmic result of the white point coordinates (R / G, B / G) of the reference camera in L2.

[0199] Among them, W src It is the logarithmic result of the coordinates of the white point of the second camera under L1 and L2, that is... in, It is the logarithmic result of the white point coordinates (R / G, B / G) of the second camera under L1; It is the logarithmic result of the white point coordinates (R / G, B / G) of the second camera under L2.

[0200] 4) Based on the white point mapping matrix, map the first target white point to the second camera to obtain the mapped white point.

[0201] For example, based on the white point mapping matrix, the first target white point can be mapped to the second camera using the following formula (2), and the logarithm of the white point coordinates of the mapped white point can be obtained:

[0202]

[0203] Among them, W sync The logarithm of the white point coordinates is the result of mapping the white points, where T is the white point mapping matrix, and log(rg ref ), log(bg ref ) is the logarithmic result of the coordinates of the first target white point in the memory frame of the reference camera.

[0204] Then, the coordinates of the mapped white point can be determined based on the logarithm of the white point coordinates. For example, the negative of the logarithm of the mapped white point coordinates can be used as the coordinates of the mapped white point to transform the mapped white point from the log domain to the original domain.

[0205] Step B4: Determine the fusion weights based on the ambient color temperature difference between the second ambient color temperature and the first ambient color temperature of the shooting environment corresponding to the memory frame.

[0206] The ambient color temperature difference between the second and first ambient color temperatures represents the environmental difference between the current shooting environment and the shooting environment of the memory frame. The ambient color temperature difference is inversely proportional to the fusion weight; the smaller the ambient color temperature difference, the greater the fusion weight.

[0207] Thus, when the ambient color temperature difference is small, it means that the current shooting environment and the shooting environment of the memory frame are relatively similar. In this case, by applying a larger fusion weight to the second target white point of the memory frame, a greater degree of color correction can be applied to the current frame, making the corrected image color of the current frame much closer to the image color of the memory frame. When the ambient color temperature difference is large, it means that the current shooting environment and the shooting environment of the memory frame are significantly different. In this case, color differences between the current frame and the memory frame are normal. In this case, a smaller degree of fusion weight can be applied to the second target white point of the memory frame to apply a smaller degree of color correction to the current frame.

[0208] In one embodiment, the fusion weight can be determined based on the ambient color temperature difference and the correspondence between the ambient color temperature difference and the fusion weight. In the correspondence between the ambient color temperature difference and the fusion weight, the ambient color temperature difference is inversely proportional to the fusion weight; that is, the smaller the ambient color temperature difference, the larger the fusion weight.

[0209] In another embodiment, determining the blending weights based on the ambient color temperature difference may further include the following steps:

[0210] 1) Determine the first sub-weight based on the ambient color temperature difference. The smaller the ambient color temperature difference, the larger the first sub-weight.

[0211] For example, the first sub-weight can be determined based on the current ambient color temperature difference and the correspondence between the ambient color temperature difference and the sub-weights. In the correspondence between the ambient color temperature difference and the sub-weights, the ambient color temperature difference is inversely proportional to the sub-weight.

[0212] For example, the correspondence can include multiple ambient color temperature differences and their corresponding sub-weights. If the correspondence includes the sub-weight corresponding to the current ambient color temperature difference, then the sub-weight corresponding to the current ambient color temperature difference can be used as the first sub-weight. Alternatively, if the correspondence does not include the sub-weight corresponding to the current ambient color temperature difference, then the sub-weight corresponding to the current ambient color temperature difference can be interpolated based on the multiple sub-weights corresponding to ambient color temperature differences in the correspondence to obtain the first sub-weight.

[0213] 2) Determine the second sub-weight based on the estimated color temperature difference between the second estimated color temperature and the first estimated color temperature of the memory frame. The larger the estimated color temperature difference, the larger the second sub-weight.

[0214] The estimated color temperature difference between the second and first estimated color temperatures indicates the difference in shooting effects between the second and reference cameras. The estimated color temperature difference is proportional to the second sub-weight; the larger the estimated color temperature difference, the larger the second sub-weight.

[0215] Thus, when the estimated color temperature difference is small, it means that the shooting effect of the second camera and the reference camera is not significantly different. In this case, a small degree of color correction can be applied to the current frame based on a smaller second sub-weight. Conversely, when the estimated color temperature difference is small, it means that the shooting effect of the second camera and the reference camera is significantly different. In this case, a larger degree of color correction can be applied to the current frame based on a larger second sub-weight to improve the difference in shooting effect.

[0216] As an example, a second sub-weight can be determined based on the current estimated color temperature difference and the correspondence between the estimated color temperature difference and the sub-weights. In the correspondence between the estimated color temperature difference and the sub-weights, the estimated color temperature difference is proportional to the sub-weight. For example, this correspondence can include multiple estimated color temperature differences and their corresponding sub-weights. If the correspondence includes the sub-weight corresponding to the current estimated color temperature difference, then the sub-weight corresponding to the current estimated color temperature difference can be used as the second sub-weight. Alternatively, if the correspondence does not include the sub-weight corresponding to the current estimated color temperature difference, then the sub-weight corresponding to the current estimated color temperature difference can be interpolated based on the multiple sub-weights corresponding to estimated color temperature differences in the correspondence to obtain the second sub-weight.

[0217] 3) Determine the fusion weight based on the first sub-weight and the second sub-weight.

[0218] For example, the product of the first sub-weight and the second sub-weight can be determined as the fusion weight. It should be understood that other methods can also be used to process the first sub-weight and the second sub-weight to obtain the fusion weight, and the embodiments of this application do not limit this.

[0219] In another embodiment, determining the blending weights based on the ambient color temperature difference may further include the following steps:

[0220] 1) Determine the first sub-weight based on the ambient color temperature difference. The smaller the ambient color temperature difference, the larger the first sub-weight.

[0221] 2) Determine the second sub-weight based on the estimated color temperature difference between the second estimated color temperature and the first estimated color temperature of the memory frame. The larger the estimated color temperature difference, the larger the second sub-weight.

[0222] 3) Determine the third sub-weight based on the ambient brightness information of the shooting environment corresponding to the memory frame. The greater the brightness indicated by the ambient brightness information, the greater the third sub-weight.

[0223] The ambient brightness information of the shooting environment corresponding to the memory frame is used to indicate the ambient brightness of the shooting environment, such as LV (Level of Detail). Generally, the lower the ambient brightness, the more complex the shooting environment, and the worse the shooting effect. The higher the ambient brightness, the better the shooting effect. The brightness indicated by the ambient brightness information is proportional to the third sub-weight; the higher the brightness indicated by the ambient brightness information, the higher the third sub-weight.

[0224] Thus, the lower the brightness indicated by the ambient light information, the worse the image quality of the reference camera's memory frame. In this case, a smaller degree of color correction can be applied to the current frame based on a smaller third sub-weight. Conversely, the higher the brightness indicated by the ambient light information, the better the image quality of the reference camera's memory frame. In this case, a larger degree of color correction can be applied to the current frame based on a larger third sub-weight. This strategy ensures the accuracy of color correction.

[0225] As an example, a third sub-weight can be determined based on the target ambient brightness information of the shooting environment corresponding to the memory frame, and the correspondence between the ambient brightness information and the sub-weights. In the correspondence between ambient brightness information and sub-weights, the brightness indicated by the ambient brightness information is proportional to the sub-weight. For example, this correspondence can include multiple ambient brightness information items and their corresponding sub-weights. If the correspondence includes the sub-weight corresponding to the target ambient brightness information, then the sub-weight corresponding to the target ambient brightness information can be used as the third sub-weight. Alternatively, if the correspondence does not include the sub-weight corresponding to the target ambient brightness information, then interpolation can be performed based on the sub-weights corresponding to multiple ambient brightness information items in the correspondence to obtain the third sub-weight.

[0226] 4) Determine the fusion weight based on the first sub-weight, the second sub-weight, and the third sub-weight.

[0227] For example, the product of the first sub-weight, the second sub-weight, and the third sub-weight can be determined as the fusion weight. It should be understood that other methods can also be used to process the first sub-weight, the second sub-weight, and the third sub-weight to obtain the fusion weight, and the embodiments of this application do not limit this.

[0228] Step B5: Based on the fusion weight, fuse the white point information of the second target white point and the white point information of the mapped white point to obtain the white point information of the third target white point.

[0229] For example, based on this fusion weight, the white point information of the second target white point and the white point information of the mapped white point can be fused using the following formula (3) to obtain the logarithmic result of the white point information of the third target white point:

[0230] W cur =W cur *(1-w)+W sync *w(3)

[0231] Among them, W act W is the logarithmic result of the white point information of the third target white point. cur The logarithm of the white point information for the second target white point, where w is the fusion weight, W sync This is the logarithmic result of the white point information mapped to white points.

[0232] Then, the white point information of the third target white point can be determined based on the logarithm of the white point information of the third target white point. For example, the negative of the logarithm of the white point information of the third target white point can be used to determine the white point information of the third target white point, so as to transform the white point information of the third target white point from the log domain to the original domain.

[0233] Step B6: Perform white balance correction on the current frame based on the white point information of the third target white point.

[0234] As an example, the white balance gain can be determined based on the white point information of the third target white point, and the white balance of the current frame can be adjusted according to the white balance gain.

[0235] For example, the white point information of the third target white point is its coordinates (R / G, B / G). The white balance gain determined based on the white point coordinates of the third target white point includes: R channel gain R_Gain = G / R, and B channel gain B_Gain = G / B. Accordingly, after white balance adjustment, the R' of each pixel in the current frame is R*R_Gain; B' is B*B_Gain; and the G channel value remains unchanged. Here, R and B are the original R and B values ​​corresponding to each pixel in the current frame, respectively.

[0236] Step B7: Based on the fusion weight, the first estimated color temperature and the second estimated color temperature are fused to obtain the third estimated color temperature.

[0237] For example, based on this fusion weight, the first color temperature and the second color temperature can be fused using the following formula (4) to obtain the third color temperature:

[0238] CCT3=CCT2*(1-w)+CCT1*w(4)

[0239] Wherein, CCT3 is the third estimated color temperature, CCT1 is the first estimated color temperature, w is the fusion weight, and CCT2 is the second estimated color temperature.

[0240] Step B8: Based on the third estimated color temperature, perform image processing on the current frame after white balance correction.

[0241] The image processing may include one or more of color correction matrix (CCM) and lens shade correction (LSC).

[0242] For example, the color correction matrix of the current frame after white balance adjustment can be determined based on the third estimated color temperature, and the color correction of the current frame after white balance adjustment can be performed based on the color correction matrix.

[0243] Please refer to Figure 12 , Figure 12 This is a schematic diagram comparing the effects of color correction on image frames before and after, as provided in an embodiment of this application. It is assumed that the camera application switches from the main camera to the telephoto camera to take pictures in the same scene. Figure 12 Image (a) in the image is the memory frame captured by the main camera (i.e., the last frame captured by the main camera); Figure 12 Figure (b) in the image is an image frame taken by a telephoto camera, i.e., an image frame before color correction; Figure 12 Figure (c) shows the multi-camera color correction algorithm provided in the embodiments of this application. Figure 12 The image frame shown in Figure (b) is obtained after color correction. Figure 12 It can be seen that the color difference between the image frame before color correction and the memory frame is large, while the color difference between the image frame after color correction and the memory frame is small. This can improve the color consistency of images captured by multiple cameras.

[0244] Next, in conjunction with the above Figure 6 and Figure 7 The process of image correction based on occlusion recognition results provided in this application embodiment is illustrated by example.

[0245] Please refer to Figure 13 , Figure 13 This is a schematic flowchart illustrating an image correction process based on occlusion recognition results for an image captured by a current camera, as provided in an embodiment of this application. This method is applied in electronic devices, and this embodiment uses the default-activated main camera as a reference camera. Figure 13 As shown, the method includes the following steps:

[0246] Step 1301: The user launches the camera application.

[0247] For example, users can tap the camera app icon to launch the camera app.

[0248] Step 1302: In response to the user's startup action, the camera application is launched.

[0249] Step 1303: The camera application sends a call command 1 to the main camera.

[0250] Command 1 is used to activate the main camera. After the camera app starts, it can use the main camera by default for taking pictures.

[0251] Step 1304: The main camera performs a preview at the default magnification according to the call command 1.

[0252] For example, the default magnification can be 1X.

[0253] Step 1305: Detect the ambient color temperature using a multispectral sensor.

[0254] During the main camera preview, the multispectral sensor can detect the ambient color temperature of the shooting environment.

[0255] As an example, once the camera application is launched, it can send a launch command to the multispectral sensor so that the multispectral sensor can detect the ambient color temperature based on the launch command.

[0256] Step 1306: The main camera sends preview image frames to the ISP.

[0257] The image frames previewed by the main camera can be sent to the ISP for processing first.

[0258] Step 1307: The ISP sends the image frame to the AWB module.

[0259] For example, the ISP can call the AWB module based on the image frame to send the image frame to the AWB module for processing.

[0260] Step 1308: The AWB module uses the AWB algorithm to calculate the statistical information of the image frame. The statistical information includes AWB white point coordinates, AWB color temperature, statistical white point coordinates, and LV.

[0261] For example, the AWB module can use the AWB algorithm to estimate the white point of an image frame, obtain the AWB white point coordinates, estimate the color temperature based on the AWB white point coordinates, obtain the AWB color temperature, estimate the white point of an image frame to obtain the white point coordinates of multiple statistical white points, and estimate the brightness of an image frame to obtain the brightness information LV.

[0262] It should be noted that the embodiments in this application only illustrate the method of obtaining the LV by estimating the brightness of an image frame using the AWB module. It should be understood that the LV of an image frame can also be determined in other ways. For example, the LV can be obtained through the AE module or through a brightness sensor, and this application embodiment does not limit this method.

[0263] Step 1309: The user switches to the telephoto camera.

[0264] Users can switch the main camera to the telephoto camera as needed. For example, they can switch the main camera to telephoto mode by clicking the option corresponding to the telephoto camera displayed in the camera app's preview frame.

[0265] Step 1310: In response to the user's camera switching operation, the camera application sends call command 2 to the telephoto camera.

[0266] Instruction 2 is used to invoke the telephoto camera.

[0267] Step 1311: The camera application sends a camera switching notification to the AWB module.

[0268] The camera switching notification may carry the identifier of the camera after switching, or it may carry the identifier of the camera before switching. This application embodiment does not limit this.

[0269] Step 1312: Based on the camera switching notification, the AWB module sends the statistical information 1 of the memory frame to the camera occlusion recognition module and the color correction module, including AWB white point coordinates 1, AWB color temperature 1, statistical white point coordinates 1 and LV1.

[0270] Among them, the memory frame refers to the last frame of the main camera preview.

[0271] As an example, the AWB module can send the AWB white point coordinates 1 and statistical white point coordinates 1 of the memory frame to the camera occlusion recognition module, and send the AWB white point coordinates 1, AWB color temperature 1 and LV1 to the color correction module.

[0272] Step 1313: The AWB module sends a color temperature data acquisition request to the multispectral sensor based on the camera switching notification.

[0273] Among them, the color temperature data acquisition request is used to request the ambient color temperature of the shooting environment, such as obtaining the ambient color temperature 1 of the shooting environment corresponding to the memory frame.

[0274] Step 1314: Based on the color temperature data acquisition request, the multispectral sensor sends the detected ambient color temperature 1 to the color correction module.

[0275] Among them, ambient color temperature 1 is used to indicate the ambient color temperature 1 of the shooting environment corresponding to the memory frame of the main camera.

[0276] Step 1315: The color correction module stores the statistical information 1 of the memory frame and the ambient color temperature 1.

[0277] For example, the storage memory frame contains the AWB white point coordinates 1, AWB color temperature 1 and LV1, as well as the ambient color temperature 1.

[0278] In other words, the color correction module can store the data corresponding to the memory frame so that the image frames previewed by other cameras can be color corrected later based on the data corresponding to the memory frame.

[0279] Step 1316: The telephoto camera performs a real-time preview according to the call command 2.

[0280] Step 1317: The telephoto camera sends the current frame of the real-time preview to the ISP.

[0281] Step 1318: The ISP sends the current frame to the AWB module.

[0282] The ISP can invoke the AWB module based on the current frame to send the current frame to the AWB module for processing.

[0283] Step 1319: The AWB module uses the AWB algorithm to calculate the statistical information 2 of the current frame. The statistical information 2 includes the AWB white point coordinates 2, the AWB color temperature 2, and the statistical white point coordinates 2.

[0284] For example, the AWB module can use the AWB algorithm to estimate the white point of the image frame, obtain the AWB white point coordinate 2, estimate the color temperature based on the AWB white point coordinate 2, obtain the AWB color temperature 2, and perform white point statistics on the image frame to obtain the white point coordinates of multiple statistical white points (i.e., statistical white point coordinate 2).

[0285] Step 1320: The AWB module sends statistical information 2 to the camera occlusion recognition module.

[0286] For example, the AWB module sends the AWB white point coordinates 2 and the statistical white point coordinates 2 to the camera occlusion recognition module.

[0287] Step 1321: The camera occlusion recognition module determines the occlusion recognition result of the main camera based on the difference between the statistical information 1 of the memory frame and the statistical information 2 of the current frame. This occlusion recognition result is used to characterize whether the main camera is occluded.

[0288] The camera occlusion recognition module can be configured as described above. Figure 8 The occlusion recognition method for the camera described in the embodiment determines the occlusion recognition result of the main camera. The specific implementation process can be referred to the above. Figure 8The relevant descriptions of the embodiments are not repeated here.

[0289] Step 1322: The camera occlusion recognition module sends the occlusion recognition results of the main camera to the AWB module and the multispectral sensor.

[0290] Step 1323: If the occlusion recognition result indicates that the main camera is not occluded, the AWB module can trigger the color correction process, that is, execute the following steps 1324-1331.

[0291] If the occlusion recognition result indicates that the main camera is not occluded, the AWB module can trigger the color correction process, that is, trigger the color correction module to perform color correction on the current frame according to the relevant parameters of the memory frame.

[0292] Step 1324: The AWB module sends the AWB white point coordinates 2 and the AWB color temperature 2 to the color correction module.

[0293] Step 1325: If the occlusion recognition result indicates that the main camera is not occluded, the multispectral sensor sends the detected ambient color temperature 2 to the color correction module.

[0294] Among them, ambient color temperature 2 is used to indicate the ambient color temperature of the shooting environment corresponding to the current frame.

[0295] Step 1326: The color correction module calculates the coordinates of the fused white point and the fused color temperature using a multi-camera color correction algorithm based on the memory frame and the corresponding data of the current frame.

[0296] The specific implementation process of using a multi-camera color correction algorithm to calculate the coordinates of the fused white point and the fused color temperature can be found above. Figure 11 The relevant descriptions in the embodiments will not be repeated here.

[0297] Step 1327: The color correction module sends the coordinates of the blended white point and the blended color temperature to the AWB module.

[0298] Step 1328: The AWB module performs white balance correction on the current frame based on the coordinates of the merged white point.

[0299] Step 1329: The AWB module sends the current frame with fused color temperature and white balance correction to the image processing module.

[0300] Step 1330: The image processing module processes the current frame after white balance correction according to the fused color temperature to obtain target image frame 1.

[0301] For example, the image processing module may include one or more image processing modules such as CCM module and LSC module. For instance, based on the fused color temperature, color correction and lens shading correction can be performed on the current frame after white balance correction to obtain the processed target image frame 1.

[0302] Step 1331: The image processing module sends the target image frame 1 to the camera application.

[0303] For example, the image processing module can first return the target image frame 1 to the ISP, and then the ISP can send the target image frame 1 to other subsequent processing modules.

[0304] Step 1332: The camera application displays the target image frame 1.

[0305] For example, a camera app can display the target image frame 1 in the preview interface.

[0306] Step 1333: If the occlusion recognition result indicates that the main camera is occluded, the AWB module will not trigger the color correction process. That is, the AWB module will perform white balance correction and other subsequent processing on the current frame according to the original algorithm.

[0307] Step 1334: The AWB module performs white balance correction on the current frame based on the AWB white point coordinates 2.

[0308] Step 1335: The AWB module sends the current frame with AWB color temperature 2 and white balance correction to the image processing module.

[0309] Step 1336: The image processing module processes the current frame after white balance correction according to the AWB color temperature 2 to obtain the target image frame 2.

[0310] For example, the image processing module may include one or more image processing modules such as CCM module and LSC module. For instance, based on AWB color temperature 2, color correction and lens shading correction can be performed on the current frame after white balance correction to obtain the processed target image frame 2.

[0311] Step 1337: The image processing module sends the target image frame 2 to the camera application.

[0312] For example, the image processing module can first return the target image frame 2 to the ISP, and then the ISP can send the target image frame 2 to other subsequent processing modules.

[0313] Step 1338: The camera application displays target image frame 2.

[0314] For example, a camera app can display the target image frame 2 in the preview interface.

[0315] In this embodiment, the accuracy of identifying whether a camera is occluded is improved by analyzing the differences in statistical information between images captured before and after camera switching. Furthermore, the camera identification result can be used to determine whether to perform an image correction process, i.e., whether to use the memory frame of the camera before switching as a reference image to correct the image frame captured by the camera after switching. Since the camera occlusion identification method provided in this embodiment can more accurately identify whether a camera is occluded, the occlusion identification result can more accurately correct the image captured by the camera after switching, improving the accuracy of image correction. For example, it can, to some extent, avoid the severe color cast caused by using the image captured by the occluded camera as a reference image for color correction.

[0316] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0317] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.

[0318] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments.

[0319] This embodiment also provides a computer program product, which is a computer-readable storage medium storing program code. When the computer program product is run on a computer, it causes the computer to perform the above-described related steps to implement the method in the above embodiment.

[0320] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the methods in the above-described method embodiments.

[0321] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0322] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0323] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for occlusion recognition of a camera, characterized in that, Applied to an electronic device, the electronic device including multiple cameras, the method includes: When the electronic device switches from the first camera to the second camera to take pictures, the first statistical information of the memory frame of the first camera and the second statistical information of the current frame captured by the second camera are determined. Wherein, the memory frame is the last frame captured by the first camera, the first statistical information includes the first target white point and multiple first statistical white points of the memory frame, the second statistical information includes the second target white point and multiple second statistical white points of the current frame, the target white point refers to the white point obtained by estimating the white point of the image using a white balance algorithm, and the statistical white point refers to the pixel point that meets the white point condition obtained by statistically analyzing the RGB values ​​of each pixel point in the image. Based on the difference between the first statistical information and the second statistical information, the occlusion recognition result of the first camera is determined, and the occlusion recognition result indicates whether the first camera is occluded; The step of determining the occlusion recognition result of the first camera based on the difference between the first statistical information and the second statistical information includes: Based on the first statistical information, a target quantity ratio is determined. The target quantity ratio is the ratio of the target quantity to the total number of the plurality of first statistical white points. The target quantity refers to the number of first statistical white points whose coordinates are within a range centered on the coordinates of the first target white point and with a preset distance as the radius. If the target quantity ratio is less than the first threshold, then a first ratio and a second ratio are determined based on the first statistical information and the second statistical information. The first ratio is the ratio of the average of the first color statistical values ​​of the plurality of first statistical white points to the average of the second color statistical values. The second ratio is the ratio of the average of the first color statistical values ​​of the plurality of second statistical white points to the average of the second color statistical values. The first color statistical value is the ratio of the R value to the G value of the corresponding statistical white point, and the second color statistical value is the ratio of the B value to the G value of the corresponding statistical white point. If the ratio of the first ratio to the second ratio is greater than the first ratio threshold or less than the second ratio threshold, then it is determined that the first camera is obstructed; wherein the first ratio threshold is greater than 1 and the second ratio threshold is less than 1.

2. The method as described in claim 1, characterized in that, If the target quantity ratio is less than a first threshold, then determining the first ratio and the second ratio based on the first statistical information and the second statistical information includes: If the target quantity ratio is less than the first threshold but greater than the second threshold, then the first ratio and the second ratio are determined based on the first statistical information and the second statistical information, wherein the second threshold is less than the first threshold.

3. The method as described in claim 2, characterized in that, The method further includes: If the target quantity ratio is less than or equal to the second threshold, then it is determined that the first camera is obstructed.

4. The method according to any one of claims 1-3, characterized in that, After determining the occlusion recognition result of the first camera based on the first statistical information and the second statistical information, the method further includes: If the occlusion recognition result indicates that the first camera is not occluded, then the white point information of the first target white point and the white point information of the second target white point in the memory frame are fused together, and the white balance is corrected for the current frame based on the fusion process result. If the occlusion recognition result indicates that the first camera is occluded, then white balance correction is performed on the current frame based on the white point information of the second target white point.

5. The method as described in claim 4, characterized in that, The step of fusing the white point information of the first target white point and the white point information of the second target white point in the memory frame, and then performing white balance correction on the current frame based on the fusion processing result, includes: Based on the ambient color temperature difference between the first ambient color temperature of the shooting environment corresponding to the memory frame and the second ambient color temperature of the shooting environment corresponding to the current frame, the white point information of the first target white point and the white point information of the second target white point of the memory frame are fused to obtain the white point information of the third target white point. Based on the white point information of the third target white point, white balance correction is performed on the current frame.

6. The method as described in claim 5, characterized in that, The step of fusing the white point information of the first target white point and the white point information of the second target white point in the memory frame with the ambient color temperature difference between the first ambient color temperature of the shooting environment corresponding to the memory frame and the second ambient color temperature of the shooting environment corresponding to the current frame to obtain the white point information of the third target white point includes: The fusion weight is determined based on the ambient color temperature difference; wherein, the smaller the ambient color temperature difference, the larger the fusion weight. According to the fusion weight, the white point information of the first target white point and the white point information of the second target white point are fused to obtain the white point information of the third target white point.

7. The method as described in claim 6, characterized in that, Before fusing the white point information of the first target white point and the white point information of the second target white point according to the fusion weight, the method further includes: The first target white point is mapped onto the second camera to obtain the mapped white point; The step of fusing the white point information of the first target white point and the white point information of the second target white point according to the fusion weight to obtain the white point information of the third target white point includes: According to the fusion weight, the white point information of the mapped white point and the white point information of the second target white point are fused to obtain the white point information of the third target white point.

8. The method as described in claim 7, characterized in that, The step of mapping the first target white point to the second camera to obtain the mapped white point includes: Acquire the first camera and its calibration data, the calibration data including white point information and color temperature under multiple standard light sources; Based on the color temperature difference between the estimated color temperature of the memory frame and the color temperature of the memory frame, at least one standard light source is determined from the plurality of standard light sources; Based on the white point information of the first camera under the at least one standard light source and the white point information of the second camera under the at least one standard light source, a white point mapping matrix between the first camera and the second camera is determined; Based on the white point mapping matrix, the first target white point is mapped to the second camera to obtain the mapped white point.

9. The method as described in any one of claims 6-8, characterized in that, The step of determining the blending weight based on the ambient color temperature difference includes: A first sub-weight is determined based on the ambient color temperature difference; wherein, the smaller the ambient color temperature difference, the larger the first sub-weight. A second sub-weight is determined based on the estimated color temperature difference between the first estimated color temperature of the memory frame and the second estimated color temperature of the current frame; wherein, the larger the estimated color temperature difference, the larger the second sub-weight. The fusion weight is determined based on the first sub-weight and the second sub-weight.

10. The method as described in claim 9, characterized in that, Before determining the fusion weight based on the first sub-weight and the second sub-weight, the method further includes: A third sub-weight is determined based on the ambient brightness information of the shooting environment corresponding to the memory frame. The greater the brightness indicated by the ambient brightness information, the greater the third sub-weight. Determining the fusion weight based on the first sub-weight and the second sub-weight includes: The fusion weight is determined based on the first sub-weight, the second sub-weight, and the third sub-weight.

11. The method as described in claim 10, characterized in that, Determining the fusion weight based on the first sub-weight, the second sub-weight, and the third sub-weight includes: The product of the first sub-weight, the second sub-weight, and the third sub-weight is determined as the fusion weight.

12. The method as described in claim 7 or 8, characterized in that, The fusion weight, the white point information of the second target white point, the white point information of the mapped white point, and the white point information of the third target white point satisfy the following formula: The white point information of the third target white point = the white point information of the second target white point * (1-W) + the white point information of the mapped white point * W; where W is the fusion weight.

13. The method according to any one of claims 5-12, characterized in that, The step of performing white balance correction on the current frame based on the white point information of the third target white point includes: The white balance gain is determined based on the white point information of the third target white point; The current frame is corrected based on the white balance gain.

14. The method as described in claim 13, characterized in that, The method further includes: The first estimated color temperature of the memory frame and the second estimated color temperature of the current frame are fused to obtain a third estimated color temperature. Based on the third estimated color temperature, color correction is performed on the current frame after white balance adjustment.

15. The method according to any one of claims 1-14, characterized in that, The first camera is a reference camera pre-set among the plurality of cameras, and the second camera is any camera among the plurality of cameras other than the reference camera.

16. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-15.

Images