Image processing method and device
By adjusting the image parameters after camera switching, the mean value is the same as the reference image average, and the image is close to the original parameters by decreasing frame by frame, the problem of large image brightness differences after camera switching is solved, improving the user experience and reducing the Equipment power consumption.
Patent Information
- Application Number
- CN202110394604.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-04-13
AI Technical Summary
After switching cameras with different zoom magnifications, there is a big difference in the brightness of the images taken before switching and the images taken after switching, which affects the user experience.
By adjusting the pixel parameters of the first image of the m frame and the second image of the n frame according to the pixel parameters of the reference image, the adjusted image average is the same as the reference image average, and the image captured by the second camera is made by decreasing frame by frame. Close to the original parameters and play a transitional role.
It effectively reduces the brightness difference between the image before and after switching, improves the user experience, and reduces the power consumption of terminal devices, avoids heating and improves performance.
Smart Images

Figure CN115209062B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art
[0002] The function of camera zoom is increasingly valued by users. Therefore, some current terminal devices (such as some smart phones) are equipped with multi-camera modules. The multi-camera module usually includes multiple cameras, and different cameras have different zoom ratios. If the user applies the zoom function of the camera, when the zoom ratio reaches a certain threshold, the camera in the multi-camera module can be switched to enable cameras with different zoom ratios, so as to make full use of the advantages of the multi-camera module and meet the user's shooting needs in different scenarios. For example, if the zoom ratio of the camera currently enabled by the terminal device is 1x, and the user wants to shoot with a camera with a higher zoom ratio, the terminal device can enable a camera with a higher zoom ratio (such as 10x) in the multi-camera module for shooting.
[0003] However, cameras with different zoom ratios have different light sensitivity. Therefore, even in the same exposure environment, the brightness of the pictures taken by cameras with different zoom ratios is different. In this case, if the terminal device switches cameras with different zoom ratios during continuous shooting, there will often be obvious brightness changes in the images taken before and after the switch, that is, there will often be a large difference in brightness between the images taken before and after the switch, affecting the user experience. Summary of the invention
[0004] In order to solve the problem in the prior art that after switching cameras with different zoom magnifications, there is a large difference in brightness between the image taken before the switch and the image taken after the switch, the embodiments of the present application provide an image processing method and device.
[0005] In a first aspect, an embodiment of the present application discloses an image processing method, comprising:
[0006] According to the parameters of each pixel in the reference image, respectively adjust the parameters of each pixel in the m-frame first image, wherein the reference image is an image taken by the first camera before switching, and the first image is an image taken by the second camera after switching, the parameters at least include the brightness of the pixels, the parameters of each pixel in the reference image are the reference parameters, the parameters of each pixel in the first image after adjustment are the first parameters, and the mean value of the first parameters is the same as the mean value of the reference parameters;
[0007] According to the parameters of each pixel in the reference image, the parameters of each pixel in the n frames of the second image are adjusted respectively to obtain the adjusted second image, wherein the second image is an image taken by the second camera after switching, the shooting time of the second image is later than the shooting time of the first image, and the adjustment amplitude of the n frames of the second image decreases frame by frame.
[0008] Through the above steps, the parameters of each pixel in the first image taken by the second camera can be adjusted so that the mean value of the first parameter of the first image taken by the second camera is the same as the mean value of the reference parameter, thereby avoiding excessive brightness difference between the first image and the reference image.
[0009] In addition, through the above steps, the parameters of each pixel in the n frames of second images taken by the second camera can also be adjusted. Since the adjustment range of the n frames of second images decreases frame by frame, the parameters of the second image taken later are closer to the original parameters of the second image after adjustment. Correspondingly, the parameters of the image taken earlier in the n frames of second images are closer to the parameters of the reference image, and as the shooting time goes by, the parameters of the image taken later in the n frames of second images are closer to the parameters of the original second image taken by the second camera. In this case, the second camera also takes other images after taking the second image, and the adjusted second image can play a transitional role between the first image and the other images, so that the user can adapt to the parameters of the other images and avoid the parameter difference between the parameters of the other images and the adjusted second image being too large.
[0010] In an optional design, adjusting the parameters of each pixel in the m-frame first image respectively according to the parameters of each pixel in the reference image includes:
[0011] The parameters of each pixel in the first target image are standardized by the following formula:
[0012]
[0013] The first target image is any one of the m frames of first images, original val is the original value of the parameter of the first pixel in the first target image, average cur is the average value of the parameters of each pixel in the first target image before the standardization process, stdev cur is the standard deviation of the parameters of each pixel in the first target image before the normalization process, new val is a parameter of the first pixel in the first target image after normalization;
[0014] The first target image after the standardization process is subjected to a de-standardization process by the following formula, and the first target image after the de-standardization process is the adjusted first image:
[0015] processed val =new val *stdev ref +average ref ;
[0016] Among them, new val is the parameter of the first pixel in the first target image after normalization, stdev ref is the standard deviation of the parameters of each pixel in the reference image, average ref is the average value of the parameters of each pixel in the reference image, processed val is a parameter of the first pixel in the adjusted first target image.
[0017] By using formulas (1) and (2), the parameters of each pixel in the adjusted first image can be determined, and then the adjusted first image can be determined. Moreover, by using this scheme, the variance of the parameters of the adjusted first image is the same as the variance of the parameters of the reference image. In this case, the contrast of the adjusted first image is closer to the contrast of the reference image, and therefore, the brightness difference between the adjusted first image and the reference image is small.
[0018] In an optional design, adjusting the parameters of each pixel in the n frames of the second image respectively according to the parameters of each pixel in the reference image includes:
[0019] According to the reference parameters and the parameter threshold, respectively adjust the parameters of each pixel in the n frames of the second image, and the second image adjusted according to the reference parameters and the parameter threshold is the first image to be processed;
[0020] According to the smoothing coefficients respectively corresponding to the n frames of second images, the parameters of each pixel in the first image to be processed are adjusted respectively, and the first image to be processed adjusted according to the smoothing coefficients is the adjusted second image.
[0021] By adjusting the second image through the above steps, it is possible to avoid the adjusted parameters of the second image being too large or too small, and the adjustment amplitude of the n frames of the second image can be reduced frame by frame, so that the second image plays a transition role.
[0022] In an optional design, adjusting the parameters of each pixel in the n frames of the second image respectively according to the reference parameter and the parameter threshold includes:
[0023] Downsampling the reference image and the second target image respectively, obtaining a first small image corresponding to the reference image, and obtaining a second small image corresponding to the second target image, wherein the second target image is any one frame of the n frames of second images, and the first small image and the second small image have the same size;
[0024] Determine a first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the same coordinates;
[0025] According to a first threshold, respectively adjusting the first parameter ratio corresponding to each coordinate, wherein the adjusted first parameter ratio is not greater than a first threshold upper limit in the first threshold, or the adjusted first parameter ratio is not less than a first threshold lower limit in the first threshold, wherein the parameter threshold includes the first threshold;
[0026] According to the coordinates respectively corresponding to the adjusted first parameter ratios, bilinear interpolation processing is performed on the adjusted first parameter ratios, and each first parameter ratio after the processing corresponds to a pixel in the second target image;
[0027] Each of the first parameter ratios after the processing is dot-multiplied with the corresponding parameter of the second target image, and the dot-multiplication result is the parameter of each pixel in the adjusted second target image.
[0028] Through the above steps, the second image can be adjusted according to the first threshold value to avoid the parameters in the second image being too large or too small. In addition, in the above steps, the first small image and the second small image are first obtained by downsampling technology, and then the first small image and the second small image are processed, so the amount of processed data can be reduced and the processing efficiency can be improved.
[0029] In an optional design, adjusting the first parameter ratio corresponding to each coordinate according to the first threshold respectively includes:
[0030] The first parameter ratio corresponding to the first coordinate is greater than the first upper threshold value, and the first parameter ratio corresponding to the first coordinate is adjusted to the first upper threshold value;
[0031] Alternatively, the first parameter ratio corresponding to the second coordinate is smaller than the first threshold lower limit, and the first parameter ratio corresponding to the second coordinate is adjusted to the first threshold lower limit.
[0032] In an optional design, adjusting the parameters of each pixel in the n frames of the second image respectively according to the reference parameter and the parameter threshold includes:
[0033] Determine a second parameter ratio between the parameters of the reference image and the parameters of a third target image respectively corresponding to the same coordinates, wherein the third target image is any one frame of the n frames of second images;
[0034] According to the second threshold, respectively adjust the second parameter ratio corresponding to each coordinate, the adjusted second parameter ratio is not greater than the second threshold upper limit in the second threshold, or the adjusted second parameter ratio is not less than the second threshold lower limit in the second threshold, the parameter threshold includes the second threshold;
[0035] The second parameter ratios corresponding to the adjusted coordinates are respectively multiplied by the corresponding parameters of the third target image, and the dot product result is the parameter of each pixel in the adjusted third target image.
[0036] Through the solution provided in the embodiment of the present application, the second image can be adjusted according to the second threshold to avoid the parameters in the second image being too large or too small. In addition, in this embodiment, the second image is not downsampled, which can improve the accuracy of the second image processing.
[0037] In an optional design, adjusting the parameters of each pixel in the first image to be processed respectively according to the smoothing coefficients respectively corresponding to the n frames of second images includes:
[0038] Determine a smoothing coefficient corresponding to a fourth target image according to the number of frames of the first image and the number of frames of the second image, wherein the fourth target image is any frame image of the first image to be processed;
[0039] Determining, according to the smoothing coefficient corresponding to the fourth target image, the standard deviation of the parameters of each pixel in the fourth target image after adjustment;
[0040] Determining an adjusted average value of the parameters of each pixel in the fourth target image according to the parameter smoothing coefficient of the fourth target image and the adjusted standard deviation;
[0041] Determine, according to the smoothing coefficient of the parameter of the fourth target image and the average value after adjustment, a ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image;
[0042] According to the ratio, the adjusted parameters of each pixel in the fourth target image are determined.
[0043] In an optional design, the smoothing coefficient corresponding to the fourth target image is determined by the following formula:
[0044]
[0045] Wherein, dampRatio is the smoothing coefficient corresponding to the fourth target image, stableFrameCnt is the frame number of the first image, transitFrameCnt is the frame number of the second image, curFrameCnt is the sequence number of the fourth target image, the sequence numbers of the first to-be-processed images are incremented in sequence according to the shooting time, and the sequence numbers of the first to-be-processed images are all positive integers;
[0046] Alternatively, the adjusted standard deviation of the parameters of each pixel in the fourth target image is determined by the following formula:
[0047] dampStdev ref =stdev ref *dampRatio+stdev cur *(1-dampRatio);
[0048] Among them, dampStdev ref is the standard deviation of the parameters of each pixel in the fourth target image after adjustment, stdev ref is the standard deviation of the parameters of each pixel in the reference image, stdev cur is the standard deviation of the parameters of each pixel in the fourth target image before adjustment;
[0049] Alternatively, the adjusted average value of the parameters of each pixel in the fourth target image is determined by the following formula:
[0050] dampAverage ref =average ref *dampRatio+average cur *(1-dampRatio);
[0051] Among them, dampAverage ref is the average value of the parameters of each pixel in the fourth target image after adjustment, average ref is the average value of the parameters of each pixel in the reference image, average cur is the average value of the parameters of each pixel in the fourth target image before adjustment;
[0052] Alternatively, the ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image is determined according to the following formula:
[0053] dampRatio(x2,y2)=Ratio(x2,y2)*dampRatio+1*(1-dampRatio);
[0054] Wherein, dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment, and Ratio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the reference image to the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment;
[0055] Alternatively, the adjusted parameters of each pixel in the fourth target image are determined according to the following formula:
[0056] Param(x2,y2)=dampRatio(x2,y2)*param(x2,y2);
[0057] Among them, Param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image after adjustment, and param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image before adjustment.
[0058] In a second aspect, an embodiment of the present application discloses an image processing device, including:
[0059] Processor and transceiver interface;
[0060] The transceiver interface is used to obtain a reference image, m frames of first images, and n frames of second images, wherein the reference image is an image taken by the first camera before switching, the first image is an image taken by the second camera after switching, and the second image is an image taken by the second camera after switching, and the shooting time of the second image is later than the shooting time of the first image;
[0061] The processor is used to adjust the parameters of each pixel in the m frames of the first image respectively according to the parameters of each pixel in the reference image, the parameters at least including the brightness of the pixel, the parameters of each pixel in the reference image are reference parameters, the parameters of each pixel in the first image after adjustment are first parameters, and the mean value of the first parameters is the same as the mean value of the reference parameters;
[0062] The processor is further configured to adjust the parameters of each pixel in the n frames of second images respectively according to the parameters of each pixel in the reference image to obtain the adjusted second images, wherein the adjustment amplitude of the n frames of second images decreases frame by frame.
[0063] In an optional design, the processor is specifically used to standardize the parameters of each pixel in the first target image by using the following formula:
[0064]
[0065] The first target image is any one of the m frames of first images, original val is the original value of the parameter of the first pixel in the first target image, average cur is the average value of the parameters of each pixel in the first target image before the standardization process, stdev cur is the standard deviation of the parameters of each pixel in the first target image before the normalization process, new val is a parameter of the first pixel in the first target image after normalization;
[0066] The processor is specifically configured to perform de-standardization on the first target image after the standardization process by using the following formula, wherein the first target image after the de-standardization process is the adjusted first image:
[0067] processed val =new val *stdev ref +average ref ;
[0068] Among them, new val is the parameter of the first pixel in the first target image after normalization, stdev ref is the standard deviation of the parameters of each pixel in the reference image, average ref is the average value of the parameters of each pixel in the reference image, processed val is a parameter of the first pixel in the adjusted first target image.
[0069] In an optional design, the processor is specifically used to adjust the parameters of each pixel in the n frames of second images according to the reference parameters and the parameter threshold, and the second image adjusted according to the reference parameters and the parameter threshold is the first image to be processed; and, according to the smoothing coefficients corresponding to the n frames of second images, the parameters of each pixel in the first image to be processed are adjusted, and the first image to be processed after adjustment according to the smoothing coefficient is the adjusted second image.
[0070] In an optional design, the processor is specifically used for:
[0071] Downsampling the reference image and the second target image respectively, obtaining a first small image corresponding to the reference image, and obtaining a second small image corresponding to the second target image, wherein the second target image is any one frame of the n frames of second images, and the first small image and the second small image have the same size;
[0072] Determine a first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the same coordinates;
[0073] According to a first threshold, respectively adjusting the first parameter ratio corresponding to each coordinate, wherein the adjusted first parameter ratio is not greater than a first threshold upper limit in the first threshold, or the adjusted first parameter ratio is not less than a first threshold lower limit in the first threshold, wherein the parameter threshold includes the first threshold;
[0074] According to the coordinates respectively corresponding to the adjusted first parameter ratios, bilinear interpolation processing is performed on the adjusted first parameter ratios, and each first parameter ratio after the processing corresponds to a pixel in the second target image;
[0075] Each of the first parameter ratios after the processing is dot-multiplied with the corresponding parameter of the second target image, and the dot-multiplication result is the parameter of each pixel in the adjusted second target image.
[0076] In an optional design, the processor is specifically used for:
[0077] The first parameter ratio corresponding to the first coordinate is greater than the first upper threshold value, and the first parameter ratio corresponding to the first coordinate is adjusted to the first upper threshold value;
[0078] Alternatively, the first parameter ratio corresponding to the second coordinate is smaller than the first threshold lower limit, and the first parameter ratio corresponding to the second coordinate is adjusted to the first threshold lower limit.
[0079] In an optional design, the processor is specifically used for:
[0080] Determine a second parameter ratio between the parameters of the reference image and the parameters of a third target image respectively corresponding to the same coordinates, wherein the third target image is any one frame of the n frames of second images;
[0081] According to the second threshold, respectively adjust the second parameter ratio corresponding to each coordinate, the adjusted second parameter ratio is not greater than the second threshold upper limit in the second threshold, or the adjusted second parameter ratio is not less than the second threshold lower limit in the second threshold, the parameter threshold includes the second threshold;
[0082] The second parameter ratios corresponding to the adjusted coordinates are respectively multiplied by the corresponding parameters of the third target image, and the dot product result is the parameter of each pixel in the adjusted third target image.
[0083] In an optional design, the processor is specifically used for:
[0084] Determine a smoothing coefficient corresponding to a fourth target image according to the number of frames of the first image and the number of frames of the second image, wherein the fourth target image is any frame image of the first image to be processed;
[0085] Determining, according to the smoothing coefficient corresponding to the fourth target image, the standard deviation of the parameters of each pixel in the fourth target image after adjustment;
[0086] Determining an adjusted average value of the parameters of each pixel in the fourth target image according to the parameter smoothing coefficient of the fourth target image and the adjusted standard deviation;
[0087] Determine, according to the smoothing coefficient of the parameter of the fourth target image and the average value after adjustment, a ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image;
[0088] According to the ratio, the adjusted parameters of each pixel in the fourth target image are determined.
[0089] In an optional design, the processor is specifically configured to determine a smoothing coefficient corresponding to the fourth target image by using the following formula:
[0090]
[0091] Wherein, dampRatio is the smoothing coefficient corresponding to the fourth target image, stableFrameCnt is the frame number of the first image, transitFrameCnt is the frame number of the second image, curFrameCnt is the sequence number of the fourth target image, the sequence numbers of the first to-be-processed images are incremented in sequence according to the shooting time, and the sequence numbers of the first to-be-processed images are all positive integers;
[0092] Alternatively, the processor is specifically configured to determine the adjusted standard deviation of the parameters of each pixel in the fourth target image by using the following formula:
[0093] dampStdev ref =stdev ref *dampRatio+stdev cur *(1-dampRatio);
[0094] Among them, dampStdev ref is the standard deviation of the parameters of each pixel in the fourth target image after adjustment, stdev ref is the standard deviation of the parameters of each pixel in the reference image, stdev cur is the standard deviation of the parameters of each pixel in the fourth target image before adjustment;
[0095] Alternatively, the processor is specifically configured to determine the adjusted average value of the parameters of each pixel in the fourth target image by using the following formula:
[0096] dampAverage ref =average ref *dampRatio+average cur *(1-dampRatio);
[0097] Among them, dampAverage ref is the average value of the parameters of each pixel in the fourth target image after adjustment, average ref is the average value of the parameters of each pixel in the reference image, average cur is the average value of the parameters of each pixel in the fourth target image before adjustment;
[0098] Alternatively, the processor is specifically configured to determine the ratio of the parameter of the same pixel in the fourth target image after adjustment to the parameter of the same pixel before adjustment according to the following formula:
[0099] dampRatio(x2,y2)=Ratio(x2,y2)*dampRatio+1*(1-dampRatio);
[0100] Wherein, dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment, and Ratio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the reference image to the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment;
[0101] Alternatively, the processor is specifically configured to determine the adjusted parameters of each pixel in the fourth target image according to the following formula:
[0102] Param(x2,y2)=dampRatio(x2,y2)*param(x2,y2);
[0103] Among them, Param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image after adjustment, and param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image before adjustment.
[0104] In a third aspect, an embodiment of the present application discloses a terminal device, including:
[0105] at least one processor and memory,
[0106] The memory is used to store program instructions;
[0107] The processor is used to call and execute the program instructions stored in the memory, so that the terminal device executes the image processing method described in the first aspect.
[0108] In a fourth aspect, an embodiment of the present application discloses a computer-readable storage medium, characterized in that:
[0109] The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the image processing method as described in the first aspect.
[0110] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the image processing method as described in the first aspect.
[0111] The embodiment of the present application provides an image processing method and device. In the method, firstly, according to the parameters of each pixel in the reference image, the parameters of each pixel in the m-frame first image are adjusted respectively, the reference image is the image taken by the first camera before switching, the first image is the image taken by the second camera after switching, the parameters at least include the brightness of the pixel, the parameters of each pixel in the reference image are the reference parameters, the parameters of each pixel in the first image after adjustment are the first parameters, and the mean of the first parameters is the same as the mean of the reference parameters; then, according to the parameters of each pixel in the reference image, the parameters of each pixel in the n-frame second image are adjusted respectively to obtain the adjusted second image, the second image is the image taken by the second camera after switching, the shooting time of the second image is later than the shooting time of the first image, and the adjustment amplitude of the n-frame second image decreases frame by frame.
[0112] In this case, through the solution provided in the embodiment of the present application, the mean value of the first parameter of the first image captured by the second camera can be made the same as the mean value of the reference parameter, thereby avoiding excessive brightness difference between the first image and the reference image. In addition, the parameters of each pixel in the n-frame second image can be adjusted to obtain the adjusted second image, wherein the adjustment amplitude of the n-frame second image decreases frame by frame.
[0113] Since in the embodiment of the present application, the adjustment range of the n-frame second images decreases frame by frame, the adjusted parameters of the second image with a later shooting time are closer to the original parameters of the second image. Accordingly, the parameters of the image with an earlier shooting time among the n-frame second images are closer to the parameters of the reference image, and as the shooting time goes by, the parameters of the image with a later shooting time among the n-frame second images are closer to the parameters of the original second image shot by the second camera.
[0114] In this case, the second camera also captures other images after capturing the second image, and the adjusted second image can serve as a transition between the first image and the other images, thereby facilitating the user to adapt to the parameters of the other images and avoiding excessive differences between the parameters of the other images and the parameters of the adjusted second image.
[0115] In addition, in the embodiment of the present application, the parameters include brightness. Therefore, through the solution provided in the embodiment of the present application, the brightness differences between the first image, the second image and the reference image can be reduced, and the brightness of the adjusted second image plays a transitional role between the brightness of the first image and the other images, so that the user can adapt to the brightness of the other images taken by the second camera.
[0116] Therefore, the solution provided in the embodiment of the present application can reduce and solve the problem in the prior art that after switching cameras with different zoom magnifications, there is a large difference in brightness between the image taken before the switch and the image taken after the switch. BRIEF DESCRIPTION OF THE DRAWINGS
[0117] Figure 1 A schematic diagram of the structure of a terminal device disclosed in an embodiment of the present application;
[0118] Figure 2 A schematic diagram of a workflow of an image processing method disclosed in an embodiment of the present application;
[0119] Figure 3 A schematic diagram of a workflow of another image processing method disclosed in an embodiment of the present application;
[0120] FIG4( a ) is an example diagram of an image that has not been downsampled in an image processing method disclosed in an embodiment of the present application;
[0121] FIG4( b ) is an example diagram of an image that has been downsampled in an image processing method disclosed in an embodiment of the present application;
[0122] Figure 5 A schematic diagram of a workflow of another image processing method disclosed in an embodiment of the present application;
[0123] Figure 6A schematic diagram of a workflow of another image processing method disclosed in an embodiment of the present application;
[0124] Figure 7 A schematic diagram of the structure of an image processing device disclosed in an embodiment of the present application;
[0125] Figure 8 A schematic diagram of the structure of a terminal device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0126] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0127] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0128] In the following, the terms "first" and "second" are used for description purposes only. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0129] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:
[0130] At present, some terminal devices (such as some smart phones) are equipped with multi-camera modules, which usually include multiple cameras, and different cameras have different zoom ratios. When shooting through the terminal device, the user can apply the zoom function of the camera to select the zoom ratio required for shooting. When the zoom ratio reaches a certain threshold and the currently enabled camera cannot meet the shooting requirements, the terminal device can enable other cameras in the multi-camera module, thereby enabling cameras with different zoom ratios by switching the cameras in the multi-camera module, so as to make full use of the advantages of the multi-camera module and meet the shooting requirements of users in different scenes.
[0131] That is to say, during the shooting process, the terminal device can enable cameras with different zoom ratios for shooting.
[0132] However, cameras with different zoom ratios have different photosensitivity. Therefore, even in an environment with the same exposure, the brightness of pictures taken by cameras with different zoom ratios will be different.
[0133] In this case, if the terminal device switches cameras with different zoom ratios during continuous shooting, there will often be obvious brightness changes in the images taken before and after the switch, that is, there will often be a large difference in brightness between the images taken before and after the switch, which will affect the user experience.
[0134] In order to solve this problem, the prior art generally adopts the following two solutions: the first solution adopts a calibration method. In this solution, in a specified environment (such as a specified light source type and a specified ambient light intensity, etc.), each camera in the multi-camera module is allowed to shoot a uniform plane with a fixed exposure amount, and the exposure amount generally includes exposure time and sensitivity; then the average brightness of the center of each frame image shot is counted, and the average brightness of the center of each frame image is compared with the target brightness. Through the comparison, the coefficient between the average brightness of the center of each frame image and the target brightness is determined, and the coefficient is used as the coefficient corresponding to each camera shooting each frame image; after switching the camera, the exposure amount of the switched camera can be determined by the coefficients corresponding to the cameras before and after the switching, and in the initial stage of the shooting process, the switched camera is used to shoot at the determined exposure amount to avoid excessive brightness differences between the images before and after the camera is switched.
[0135] However, the shooting environment of the camera is often diverse. Since the first solution can only be calibrated in a specified environment, it can usually only solve the problem of large image brightness differences caused by camera switching when shooting in a specified environment. If in other environments, the camera switching will still cause large brightness differences in the images before and after the camera switching, so the scope of application is relatively small.
[0136] In the second solution, the camera to be switched is started in advance, that is, the camera before switching and the camera after switching are run at the same time, and then the brightness of the two cameras in the same field of view (FOV) area is calculated, and then the exposure corresponding to the camera after switching is calculated according to the difference in brightness. When shooting at the exposure corresponding to the camera after switching, the brightness of the image obtained by the camera after switching is smaller than that of the image obtained by the camera before switching, so as to avoid excessive difference in brightness between the images before and after the camera switching.
[0137] However, in this solution, the terminal device needs to run the camera before switching and the camera after switching at the same time, which will increase the overall power consumption of the terminal device, make the terminal device prone to heating, and easily lead to reduced performance of the terminal device.
[0138] In order to solve the problem in the prior art that after switching cameras with different zoom magnifications, there is a large difference in brightness between the image taken before the switch and the image taken after the switch, the embodiments of the present application provide an image processing method and device.
[0139] The image processing method provided in the embodiment of the present application can be applied to a terminal device, which can be a variety of types of devices. The terminal device can be connected to an imaging device, or the imaging device can be provided in the terminal device.
[0140] The imaging device generally includes a multi-camera module, and the imaging device is shot by using the multiple cameras included in the multi-camera module. For example, the imaging device may include a first camera and a second camera, and the zoom ratios of the first camera and the second camera are different. In addition, the terminal device may process the image shot by the imaging device according to the image processing method provided in the embodiment of the present application to solve the problem that there is a large difference in brightness between the image shot before switching and the image shot after switching.
[0141] In some embodiments, the terminal device may be a mobile phone, tablet computer, desktop, laptop, notebook computer, ultra-mobile personal computer (UMPC), handheld computer, netbook or other device. The present application does not impose any special restrictions on the specific form of the terminal device.
[0142] Taking a smart phone as an example, the structural diagram of the terminal device can be as follows: Figure 1 See Figure 1 The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a radio frequency module 150, a communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a screen 301, and a subscriber identification module (SIM) card interface 195, etc.
[0143] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the mobile phone. In other embodiments of the present application, the mobile phone may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0144] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0145] The controller can be the nerve center and command center of the mobile phone. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.
[0146] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0147] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0148] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface to realize the touch function of the mobile phone.
[0149] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.
[0150] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the communication module 160 via the PCM interface to implement the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0151] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the communication module 160. For example, the processor 110 communicates with the Bluetooth module in the communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
[0152] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the screen 301 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the mobile phone. The processor 110 and the screen 301 communicate via the DSI interface to realize the display function of the mobile phone.
[0153] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the screen 301, the communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0154] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the mobile phone, and can also be used to transfer data between the mobile phone and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.
[0155] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the mobile phone. In other embodiments of the present application, the mobile phone may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0156] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the mobile phone. While the charging management module 140 is charging the battery 142, it can also power the terminal device through the power management module 141.
[0157] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the screen 301, the camera 193, and the communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0158] The wireless communication function of the mobile phone can be realized through antenna 1, antenna 2, radio frequency module 150, communication module 160, modem processor and baseband processor.
[0159] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0160] The RF module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to mobile phones. The RF module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The RF module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The RF module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the RF module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the RF module 150 can be set in the same device as at least some of the modules of the processor 110.
[0161] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a screen 301. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the RF module 150 or other functional modules.
[0162] The communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to mobile phones. The communication module 160 can be one or more devices integrating at least one communication processing module. The communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0163] In some embodiments, the antenna 1 of the mobile phone is coupled to the radio frequency module 150, and the antenna 2 is coupled to the communication module 160, so that the mobile phone can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).
[0164] The mobile phone realizes the display function through GPU, screen 301, and application processor. GPU is a microprocessor for image processing, which connects screen 301 and application processor. GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information. In the embodiment of the present application, screen 301 may include a display and a touch device. The display is used to output display content to the user, and the touch device is used to receive touch events input by the user on screen 301.
[0165] In a mobile phone, the sensor module 180 may include one or more of a gyroscope, an acceleration sensor, a pressure sensor, an air pressure sensor, a magnetic sensor (such as a Hall sensor), a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, a pyroelectric infrared sensor, an ambient light sensor or a bone conduction sensor, and the embodiments of the present application do not impose any restrictions on this.
[0166] The mobile phone can realize the shooting function through ISP, camera 193, video codec, GPU, flexible screen 301 and application processor.
[0167] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0168] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile phone may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0169] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when a mobile phone selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0170] Video codecs are used to compress or decompress digital videos. A mobile phone can support one or more video codecs. In this way, the mobile phone can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0171] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of mobile phones, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0172] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.
[0173] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0174] The mobile phone can implement audio functions such as network standard determination, recording, etc. through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.
[0175] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0176] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The mobile phone can listen to music or listen to hands-free calls through the speaker 170A.
[0177] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the mobile phone receives a call or voice message, the voice can be heard by placing the receiver 170B close to the ear.
[0178] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to the microphone 170C to input the sound signal into the microphone 170C. The mobile phone can be provided with at least one microphone 170C. In other embodiments, the mobile phone can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the mobile phone can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording function, etc.
[0179] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0180] The key 190 includes a power button, a volume button, etc. The key 190 can be a mechanical key or a touch key. The mobile phone can receive key input and generate key signal input related to the user settings and function control of the mobile phone.
[0181] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the flexible screen 301, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0182] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0183] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the mobile phone by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The mobile phone can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The mobile phone interacts with the network through the SIM card to realize functions such as calls and data communications. In some embodiments, the mobile phone uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone and cannot be separated from the mobile phone.
[0184] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.
[0185] In order to clarify the solution provided by the present application, the solution provided by the present application is introduced and explained through various embodiments in conjunction with the accompanying drawings.
[0186] The embodiment of the present application provides an image processing method to solve the problem in the prior art that after switching cameras with different zoom magnifications, there is a large difference in brightness between the image taken before the switch and the image taken after the switch. Among them, a first camera and a second camera can be set in a terminal device to which the image processing method is applied, and the images taken by the first camera and the second camera are obtained. Alternatively, the terminal device can be connected to the first camera and the second camera, and the images of the first camera and the second camera are obtained. Among them, the zoom magnifications of the first camera and the second camera are different. During the shooting process, the first camera and the second camera can be switched to operate so as to shoot through different cameras.
[0187] In addition, after acquiring the images taken by the first camera and the second camera, the terminal device may perform image processing on the images taken by the first camera and the second camera using the method provided in the embodiment of the present application.
[0188] In the embodiment of the present application, the first camera may be the camera before switching, and the second camera may be the camera after switching. That is, when shooting with the first camera, if the zoom ratio of the first camera cannot meet the user's needs, the user may switch to the second camera and shoot with the second camera.
[0189] See also Figure 2As shown in the workflow diagram, the image processing method provided in the embodiment of the present application includes the following steps:
[0190] Step S11, adjusting the parameters of each pixel in the m-frame first image respectively according to the parameters of each pixel in the reference image.
[0191] The reference image is the image captured by the first camera before switching. In a feasible implementation, in order to reduce the brightness difference between images captured by different cameras before and after switching, the reference image may be the last frame of image captured by the first camera before switching.
[0192] In addition, the first image is an image captured by the second camera after switching. In order to reduce the difference in brightness of images captured by different cameras before and after switching, the first image may include the first frame image captured by the second camera after switching from the first camera to the second camera and images captured thereafter. In addition, m is a positive integer.
[0193] In the embodiment of the present application, the parameters of the pixel at least include the brightness of the pixel. The brightness of the pixel may be the brightness in the YUV color space (i.e., parameter Y). Further, in order to reduce the difference between images taken by different cameras before and after switching, the parameters may include other parameters in addition to the brightness of the pixel.
[0194] Exemplarily, the parameters may also include chromaticity in the YUV space (ie, parameter U and parameter V); in addition, the parameters may also include at least one of a red color channel (ie, parameter R), a green color channel (ie, parameter G), and a blue color channel (ie, parameter B) in the RGB space.
[0195] Of course, the parameters may also include at least one parameter in other color spaces, which is not limited in the embodiments of the present application.
[0196] In addition, in an embodiment of the present application, the parameters of each pixel in the reference image are reference parameters, the adjusted parameters of each pixel in the first image are first parameters, and the mean of the first parameters is the same as the mean of the reference parameters.
[0197] Through the operation of step S11, the average value of the adjusted parameters of each pixel in the first image is adjusted to be the same as the average value of the parameters of each pixel in the reference image. In this case, the contrast of the m-frame first image is close to that of the reference image, thereby reducing the brightness difference between the reference image and the first image.
[0198] Furthermore, if the parameters include other parameters besides the brightness of the pixels, the difference of other parameters between the reference image and the first image can also be reduced through step S11.
[0199] Step S12: adjusting the parameters of each pixel in the n frames of the second image according to the parameters of each pixel in the reference image, and obtaining the adjusted second image.
[0200] The second image is an image captured by the second camera after switching, and the shooting time of the second image is later than the shooting time of the first image. That is, the second camera captures the n frames of the second image after capturing the m frames of the first image, where n is a positive integer.
[0201] In addition, the adjusted parameters of each pixel in the second image may be referred to as second parameters.
[0202] In the embodiment of the present application, the adjustment range of the n frames of second images decreases frame by frame. Further, usually, the adjustment range of the earliest image captured among the n frames of second images is not greater than the adjustment range of any frame of first image.
[0203] The adjustment range can be implemented in a variety of forms. In a feasible form of the embodiment of the present application, the adjustment range of a frame of image is the absolute value of the difference between the mean of the parameters of the image after adjustment and the mean of the parameters before adjustment; or, in another feasible form, the adjustment range of a frame of image can be the absolute value of the difference between the standard deviation of the parameters of the image after adjustment and the standard deviation of the parameters before adjustment. Therefore, the later the shooting time of the second image, the smaller the adjustment range, and the later the shooting time of the second image, the closer the parameters after adjustment are to the parameters of the second image before adjustment.
[0204] That is to say, it is assumed that the n frames of second images include image 1 and image 2, and the shooting time of image 1 is earlier than the shooting time of image 2, then the adjustment range of image 1 is greater than the adjustment range of image 2.
[0205] Among them, in a feasible example, the adjustment amplitude of the image one is the absolute value of the difference between the mean value of the parameters of each pixel of the image one after adjustment and the mean value of the parameters of each pixel of the image one before adjustment; the adjustment amplitude of the image two is the absolute value of the difference between the mean value of the parameters of each pixel of the image two after adjustment and the mean value of the parameters of each pixel of the image two before adjustment.
[0206] Alternatively, in another feasible example, the adjustment amplitude of the image one is the absolute value of the difference between the standard deviation of the parameters of each pixel of the image one after adjustment and the standard deviation of the parameters of each pixel of the image one before adjustment; the adjustment amplitude of the image two is the absolute value of the difference between the standard deviation of the parameters of each pixel of the image two after adjustment and the standard deviation of the parameters of each pixel of the image two before adjustment.
[0207] The image with the earliest shooting time among the n frames of second images is set as image 1. In a feasible implementation method, the adjustment amplitude of image 1 may be the same as the adjustment amplitude of any frame of the first image, or the adjustment amplitude of image 1 may be smaller than the adjustment amplitude of any frame of the first image.
[0208] In addition, the image with the latest shooting time among the n frames of second images is set as image 2. In a feasible implementation, the adjustment amplitude of image 2 can be 0. In this case, the parameters of image 2 after adjustment are the same as those before adjustment.
[0209] In the solution provided in the embodiment of the present application, through step S11, the m frames of first images taken by the second camera after switching are processed, and through step S12, the n frames of second images taken by the second camera after switching are processed, wherein the shooting time of the second image is later than the shooting time of the first image.
[0210] Through the operation of step S11, the mean value of the first parameter of the first image taken by the second camera can be made the same as the mean value of the reference parameter, thereby avoiding excessive brightness difference between the first image and the reference image.
[0211] In addition, through the operation of step S12, the parameters of each pixel in the n frames of the second image can be adjusted to obtain the adjusted second image, wherein the adjustment amplitude of the n frames of the second image decreases frame by frame.
[0212] The adjustment range of an image can reflect the difference between the image before and after adjustment. Generally, the smaller the adjustment range of an image, the smaller the difference between the parameters of the image after adjustment and before adjustment, that is, the image after adjustment is closer to the image before adjustment.
[0213] Since the adjustment amplitude of the n-frame second images decreases frame by frame through the operation of step S12, the parameters of the second images with later shooting time are closer to the original parameters of the second images after adjustment. Correspondingly, the parameters of the images with earlier shooting time among the n-frame second images are closer to the parameters of the reference image, and as the shooting time goes by, the parameters of the images with later shooting time among the n-frame second images are closer to the parameters of the original second image shot by the second camera. Among them, if the adjustment amplitude of the latest image shot among the n-frame second images is 0, the parameters of the latest image shot among the n-frame second images after adjustment are the same as the parameters before adjustment, and the parameters of the image before and after adjustment remain unchanged.
[0214] In this case, the second camera also captures other images after capturing the second image, and the adjusted second image can serve as a transition between the first image and the other images, thereby facilitating the user to adapt to the parameters of the other images and avoiding excessive differences between the parameters of the other images and the parameters of the adjusted second image.
[0215] In the embodiment of the present application, the parameters include brightness. Therefore, through the solution provided in the embodiment of the present application, the brightness differences between the first image, the second image and the reference image can be reduced, and the brightness of the adjusted second image plays a transitional role between the brightness of the first image and the other images, so that the user can adapt to the brightness of the other images taken by the second camera.
[0216] That is to say, the solution provided in the embodiment of the present application can reduce and solve the problem in the prior art that after switching cameras with different zoom ratios, there is a large difference in brightness between the image taken before the switch and the image taken after the switch.
[0217] Furthermore, the solution provided in the embodiment of the present application can process images taken in various environments, and can be applied to various environments. Therefore, compared with the first solution of the prior art (i.e., the solution for calibration in a specified environment), the solution provided in the embodiment of the present application is applicable to a wider range of environments.
[0218] In addition, the solution provided by the embodiment of the present application does not need to run the first camera and the second camera at the same time. It only needs to start the second camera after switching to the second camera and process the image captured by the second camera. Therefore, compared with the second solution of the prior art (i.e., the solution of running the camera before switching and the camera after switching at the same time), the solution provided by the embodiment of the present application can reduce the overall power consumption of the terminal device, reduce the heating phenomenon of the terminal device, and improve the performance of the terminal device.
[0219] Furthermore, in the embodiment of the present application, the pixel parameters may include other parameters in addition to brightness, for example, the parameters may include at least one of chromaticity parameters, red color channel parameters, green color channel parameters, and blue color channel parameters. In this case, through the solution of the embodiment of the present application, the differences in other parameters between the first image, the second image, and the reference image may be reduced, and the adjusted other parameters of the second image may also play a transitional role between other parameters of the first image and other images, the other images being images captured by the second camera after the second image is captured.
[0220] In addition, in the embodiment of the present application, the mean of the parameters of the adjusted m frames of the first image is the same as the mean of the parameters of the reference image, so as to avoid the parameters of the image captured by the second camera after the switch being too different from the parameters of the reference image, so the adjustment range of the first image is usually large. The second image plays a transitional role for other images after the first image and the second image, so that the user can adapt to the parameters of other images captured by the second camera. In this case, m is usually less than n. Exemplarily, m can be a positive integer between 2 and 9, and n can be a positive integer between 10 and 20.
[0221] Of course, m and n may also have other values, which are not limited in the embodiments of the present application.
[0222] In the embodiment of the present application, through step S11, an operation of adjusting the parameters of each pixel in the m-frame first image respectively according to the parameters of each pixel in the reference image is provided, and the operation can be implemented by the following steps:
[0223] First, the parameters of each pixel in the first target image are standardized by the following formula:
[0224]
[0225] The first target image is any one of the m frames of first images, original val is the original value of the parameter of the first pixel in the first target image, average cur is the average value of the parameters of each pixel in the first target image before the standardization process, stdev cur is the standard deviation of the parameters of each pixel in the first target image before the normalization process, new valis a parameter of the first pixel in the first target image after the normalization process. The first pixel may be any pixel in the first target image. In this case, each pixel in the first target image may be used as the first pixel in sequence, and the parameter of each pixel in the first target image after the normalization process may be determined according to the above formula.
[0226] In the solution provided in the embodiment of the present application, the m frames of first images may be processed frame by frame. In the above formula, the standardization process of the first target image in the first image is taken as an example.
[0227] By using formula (1), the parameters of each pixel in the first target image after the standardization process can be determined. Accordingly, each frame image in the first image is selected as the first target image. By using formula (1), the parameters of each pixel in each frame of the first image after the standardization process can be determined.
[0228] Then, the first target image after the standardization process is de-standardized by the following formula, and the first target image after the de-standardization process is the adjusted first image:
[0229] processed val =new val *stdev ref +average ref Formula (2);
[0230] Among them, new val is the parameter of the first pixel in the first target image after normalization, stdev ref is the standard deviation of the parameters of each pixel in the reference image, average ref is the average value of the parameters of each pixel in the reference image, processed val is a parameter of the first pixel in the adjusted first target image.
[0231] Through the above formula, each pixel in the first target image can be de-standardized to obtain the parameters of each de-standardized pixel. In addition, each frame image in the first image is selected as the first target image, and the parameters of each pixel in each frame of the first image after de-standardization can be determined through formula (2). Among them, each frame of the first image that has been de-standardized is the adjusted first image.
[0232] That is, through formulas (1) and (2), the parameters of each pixel in the adjusted first image can be determined, and then the adjusted first image can be determined.
[0233] The mean of the parameters of the adjusted first image determined by formulas (1) and (2) is the same as the mean of the parameters of the reference image, and, through this scheme, the variance of the parameters of the adjusted first image is the same as the variance of the parameters of the reference image. In this case, the contrast of the adjusted first image is closer to the contrast of the reference image, and therefore, the brightness difference between the adjusted first image and the reference image is small.
[0234] Furthermore, if the parameters also include other parameters, the above steps can also reduce the differences in other parameters between the adjusted first image and the reference image.
[0235] Exemplarily, the parameters include, in addition to brightness, a chromaticity parameter. In this case, through the above steps, not only the differences in other parameters between the adjusted first image and the reference image can be reduced, but also the differences in chromaticity between the adjusted first image and the reference image can be reduced.
[0236] In step S12, the operation of adjusting the parameters of each pixel in the n-frame second image according to the parameters of each pixel in the reference image is disclosed. The operation can be implemented by the following steps:
[0237] In the first step, the parameters of each pixel in the n frames of the second image are adjusted respectively according to the reference parameters and the parameter threshold, and the second image adjusted according to the reference parameters and the parameter threshold is the first image to be processed.
[0238] Through this step, it is possible to avoid the parameters of each pixel in the first image to be processed being too large or too small.
[0239] In the second step, according to the smoothing coefficients respectively corresponding to the n frames of second images, the parameters of each pixel in the first image to be processed are adjusted respectively, and the first image to be processed adjusted according to the smoothing coefficients is the adjusted second image.
[0240] Through this step, the parameters of each pixel in the first image to be processed can be adjusted according to the smoothing coefficient corresponding to the second image, so that the adjustment range of the n frames of the second image decreases frame by frame, so that the second image plays a transition role.
[0241] In one possible implementation, see Figure 3 The workflow diagram shown in the figure, wherein the parameters of each pixel in the n frames of the second image are adjusted according to the reference parameters and the parameter threshold, respectively, comprises the following steps:
[0242] Step S21, obtaining a first small image corresponding to the reference image and a second small image corresponding to the second target image by downsampling the reference image and the second target image respectively.
[0243] The second target image is any one of the n frames of second images, and the first small image and the second small image have the same size. Exemplarily, if the size of the first small image is H*W, the size of the second small image is also H*W.
[0244] Downsampling (ie, subsampled) can also be called downsampling, which is a technique for generating a thumbnail of a corresponding image. In the embodiment of the present application, the downsampling method used can be a mean downsampling method.
[0245] An embodiment of the present application provides an example of mean downsampling. In this example, FIG. 4( a) is an image that has not been mean downsampling, and the resolution of the image is 8*8. FIG. 4( b) is an image obtained after mean downsampling the image of FIG. 4( a ), and the resolution of the image is 2*2. Among them, the parameters of the pixels in the first row and first column in Figure 4(b) are the average parameters of the pixels contained in the area from the first row to the fourth row, and the first column to the fourth column in the image of Figure 4(a); the parameters of the pixels in the first row and second column in Figure 4(b) are the average parameters of the pixels contained in the area from the first row to the fourth row, and the fifth column to the eighth column in the image of Figure 4(a); the parameters of the pixels in the second row and first column in Figure 4(b) are the average parameters of the pixels contained in the area from the fifth row to the eighth row, and the first column to the fourth column in the image of Figure 4(a); the parameters of the pixels in the second row and second column in Figure 4(b) are the average parameters of the pixels contained in the area from the fifth row to the eighth row, and the fifth column to the eighth column in the image of Figure 4(a).
[0246] Of course, in the solution provided in the embodiment of the present application, other forms of downsampling methods may also be used, and the embodiment of the present application is not limited to this.
[0247] Step S22: determine a first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the same coordinates.
[0248] Wherein, the first parameter ratio can be expressed by the following formula:
[0249]
[0250] In formula (3), ref ds (x0, y0) is the parameter of the pixel with coordinates (x0, y0) in the first small picture; cur ds(x0, y0) is the parameter of the pixel with coordinates (x0, y0) in the first small image; ratio(x0, y0) is the first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the coordinates (x0, y0).
[0251] Step S23: According to the first threshold, respectively adjust the first parameter ratio corresponding to each coordinate, wherein the adjusted first parameter ratio is not greater than the first threshold upper limit in the first threshold, or the adjusted first parameter ratio is not less than the first threshold lower limit in the first threshold, and the parameter threshold includes the first threshold.
[0252] In the embodiment of the present application, the parameter threshold includes the first threshold, and the first threshold may include a first threshold upper limit or a first threshold lower limit, or include both the first threshold upper limit and the first threshold lower limit.
[0253] Through the first threshold, the adjusted first parameter ratio can be not greater than the first threshold upper limit in the first threshold; or, the adjusted first parameter ratio is not less than the first threshold lower limit in the first threshold; or, the conditions that the adjusted first parameter ratio is not greater than the first threshold upper limit in the first threshold, and the adjusted first parameter ratio is not less than the first threshold lower limit in the first threshold are simultaneously satisfied.
[0254] Step S24, performing bilinear interpolation processing on the adjusted first parameter ratios according to the coordinates corresponding to the adjusted first parameter ratios, so that each first parameter ratio after processing corresponds to a pixel in the second target image.
[0255] Among them, each first parameter ratio after processing corresponds to a pixel in the second target image, which means that each first parameter ratio after processing corresponds to a pixel in the second target image one-to-one, that is, each pixel in the second target image corresponds to a first parameter ratio after processing.
[0256] Step S25, performing dot multiplication on each of the first parameter ratios after the processing and the corresponding parameters of the second target image, and the dot multiplication result is the parameter of each pixel in the adjusted second target image.
[0257] In the embodiment of the present application, the dot product result may be set as the parameter of each pixel in the first target adjusted image, and the first target adjusted image is the second target image adjusted according to the reference parameter and the parameter threshold.
[0258] Exemplarily, if a first parameter ratio after processing corresponds to a pixel (x1, y1) in the second target image, the second target image after adjustment according to the reference parameter and the parameter threshold is called the first target adjusted image, then the product of the first parameter ratio and the parameter of the pixel (x1, y1) in the second target image is the parameter of the pixel (x1, y1) in the first target adjusted image.
[0259] Through the operations of step S21 to step S25, the image of the second target image after adjustment according to the reference parameter and the parameter threshold can be obtained. In this case, the images in the n frames of the second image can be selected in sequence as the second target image, and the first image to be processed of each frame is determined according to the above steps.
[0260] Through the above embodiment, the second image can be adjusted according to the first threshold value to avoid the parameters in the second image being too large or too small. In addition, in this embodiment, the first small image and the second small image are first obtained by downsampling technology, and then the first small image and the second small image are processed, so the amount of processed data can be reduced and the processing efficiency can be improved.
[0261] In the above steps, the operation of adjusting the first parameter ratio corresponding to each coordinate according to the first threshold is disclosed, and the operation may include the following steps:
[0262] The first parameter ratio corresponding to the first coordinate is greater than the first upper threshold value, and the first parameter ratio corresponding to the first coordinate is adjusted to the first upper threshold value;
[0263] Alternatively, the first parameter ratio corresponding to the second coordinate is smaller than the first threshold lower limit, and the first parameter ratio corresponding to the second coordinate is adjusted to the first threshold lower limit.
[0264] That is to say, if the first parameter ratio corresponding to a certain coordinate is greater than the first threshold upper limit in the first threshold, the first parameter ratio corresponding to the coordinate is adjusted to the first threshold upper limit. In addition, if the first parameter ratio corresponding to a certain coordinate is less than the first threshold lower limit in the first threshold, the first parameter ratio corresponding to the coordinate is adjusted to the first threshold lower limit.
[0265] In addition, in this embodiment, if the first parameter ratio corresponding to a certain coordinate is between the first upper threshold value and the first lower threshold value, the first parameter ratio corresponding to the coordinate may be kept unchanged.
[0266] In another possible implementation, see Figure 5 The workflow diagram shown in the figure, wherein the parameters of each pixel in the n frames of the second image are adjusted according to the reference parameters and the parameter threshold, respectively, comprises the following steps:
[0267] Step S31, determining a second parameter ratio between the parameters of the reference image and the parameters of the third target image corresponding to the same coordinates.
[0268] The third target image is any one of the n frames of second images.
[0269] Step S32: according to the second threshold, respectively adjust the second parameter ratio corresponding to each coordinate, wherein the adjusted second parameter ratio is not greater than the second threshold upper limit in the second threshold, or the adjusted second parameter ratio is not less than the second threshold lower limit in the second threshold, and the parameter threshold includes the second threshold.
[0270] In an embodiment of the present application, the parameter threshold includes the second threshold, the second threshold includes a second threshold upper limit or a second threshold lower limit, or the second threshold includes both the second threshold upper limit and the second threshold lower limit.
[0271] Among them, if a second parameter ratio is greater than the second threshold upper limit, the second parameter ratio can be adjusted to the second threshold upper limit, or if a second parameter ratio is less than the second threshold upper limit, the second parameter ratio can be adjusted to the second threshold lower limit.
[0272] Step S33: perform dot multiplication of the second parameter ratios corresponding to the adjusted coordinates with the corresponding parameters of the third target image, and the dot multiplication result is the parameter of each pixel in the adjusted third target image.
[0273] Through the solution provided in the embodiment of the present application, the second image can be adjusted according to the second threshold to avoid the parameters in the second image being too large or too small. In addition, in this embodiment, the second image is not downsampled, which can improve the accuracy of the second image processing.
[0274] In the present application, an operation of adjusting the parameters of each pixel in the first to-be-processed image according to the smoothing coefficient corresponding to the second image is disclosed. Figure 6 The workflow diagram shown in the figure can be achieved by following the steps below:
[0275] Step S41: Determine a smoothing coefficient corresponding to a fourth target image according to the frame number of the first image and the frame number of the second image, wherein the fourth target image is any frame image in the first image to be processed.
[0276] Specifically, in the embodiment of the present application, the smoothing coefficient corresponding to the fourth target image is determined according to the frame number of the first image and the frame number of the second image, and the sequence number of the fourth target image.
[0277] In a feasible implementation scheme provided in an embodiment of the present application, the smoothing coefficient corresponding to the fourth target image is determined by the following formula:
[0278]
[0279] Among them, dampRatio is the smoothing coefficient corresponding to the fourth target image, stableFrameCnt is the frame number of the first image, transitFrameCnt is the frame number of the second image, curFrameCnt is the serial number of the fourth target image, the serial number of the first image to be processed increases in sequence according to the shooting time, and the serial numbers of the first images to be processed are all positive integers.
[0280] Generally, the sequence number of each frame of the first image to be processed can be determined according to the shooting time of each frame of the first image to be processed. Among them, the sequence number of the first image to be processed with the earliest shooting time is m+1, and the sequence number of the first image to be processed with the latest shooting time is m+n. That is to say, in formula (4), according to the order from early to late shooting time, the sequence numbers of the first images to be processed of n frames are m+1, m+2, ..., m+n.
[0281] In addition, in formula (4), stableFrameCnt is the number of frames of the first image, and in the embodiment of the present application, the value may be m. transitFrameCnt is the number of frames of the second image, and in the embodiment of the present application, the value may be n.
[0282] According to the above formula (4), since the serial numbers of the first images to be processed in each frame are different, and the smoothing coefficient of each first image to be processed is related to the serial number of the first image to be processed in the frame, the smoothing coefficients corresponding to the first images to be processed in each frame can be determined respectively through formula (4).
[0283] Step S42: determining the adjusted standard deviation of the parameters of each pixel in the fourth target image according to the smoothing coefficient corresponding to the fourth target image.
[0284] In a feasible implementation scheme provided in an embodiment of the present application, the standard deviation of the parameters of each pixel in the fourth target image after adjustment is determined by the following formula:
[0285] dampStdev ref =stdev ref *dampRatio+stdev cur *(1-dampRatio) formula (5);
[0286] Among them, dampStdev refis the standard deviation of the parameters of each pixel in the fourth target image after adjustment, stdev ref is the standard deviation of the parameters of each pixel in the reference image, stdev cur is the standard deviation of the parameters of each pixel in the fourth target image before adjustment.
[0287] In this case, the standard deviation of the parameters of each pixel in the fourth target image after adjustment can be determined through formula (5) and based on the standard deviation of the parameters of each pixel in the reference image and the standard deviation of the parameters of each pixel in the fourth target image before adjustment.
[0288] Of course, the adjusted standard deviation of the parameters of each pixel in the fourth target image may also be determined in other ways, which is not limited in this embodiment of the present application.
[0289] Step S43: determining the adjusted average value of the parameters of each pixel in the fourth target image according to the parameter smoothing coefficient of the fourth target image and the adjusted standard deviation.
[0290] In a feasible implementation scheme provided in an embodiment of the present application, the adjusted average value of the parameters of each pixel in the fourth target image is determined by the following formula:
[0291] dampAverage ref =average ref *dampRatio+average cur *(1-dampRatio) formula (6);
[0292] Among them, dampAverage ref is the average value of the parameters of each pixel in the fourth target image after adjustment, average ref is the average value of the parameters of each pixel in the reference image, average cur is the average value of the parameters of each pixel in the fourth target image before adjustment.
[0293] In this case, the average value of the parameters of each pixel in the fourth target image after adjustment can be determined through formula (6) and based on the average value of the parameters of each pixel in the reference image and the average value of the parameters of each pixel in the fourth target image before adjustment.
[0294] Of course, the adjusted average values of the parameters of each pixel in the fourth target image may also be determined by other methods, which is not limited in the embodiment of the present application.
[0295] Step S44: Determine the ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image according to the smoothing coefficient of the parameter of the fourth target image and the average value after adjustment.
[0296] In a feasible implementation scheme provided in an embodiment of the present application, the ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image is determined according to the following formula:
[0297] dampRatio(x2,y2)=Ratio(x2,y2)*dampRatio+1*(1-dampRatio) formula (7);
[0298] Among them, dampRatio(x2,y2) is the ratio of the parameters of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameters before adjustment, and Ratio(x2,y2) is the ratio of the parameters of the pixel with coordinates (x2,y2) in the reference image to the parameters of the pixel with coordinates (x2,y2) in the fourth target image before adjustment.
[0299] In this case, through formula (7), and based on the parameters of each pixel in the reference image and the parameters of each pixel in the fourth target image before adjustment, the ratio of the parameters of the same pixel in the fourth target image after adjustment to the parameters before adjustment can be determined.
[0300] Of course, the ratio may also be determined by other methods, which are not limited in the embodiments of the present application.
[0301] Step S45: Determine the adjusted parameters of each pixel in the fourth target image according to the ratio.
[0302] In a feasible implementation scheme provided in an embodiment of the present application, the adjusted parameters of each pixel in the fourth target image are determined by the following formula:
[0303] Param(x2,y2)=dampRatio(x2,y2)*param(x2,y2) Formula (8);
[0304] Among them, Param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image after adjustment, and param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image before adjustment.
[0305] In this case, the parameters of each pixel in the fourth target image after adjustment can be determined by formula (8) and according to the parameters of each pixel in the fourth target image before adjustment.
[0306] Of course, the adjusted parameters of each pixel in the fourth target image may also be determined by other methods, which is not limited in the embodiment of the present application.
[0307] Through the above steps, the adjusted n frames of second images can be obtained, and since the n frames of second images are adjusted according to the smoothing coefficient, and the smoothing coefficient corresponding to each frame of the second image is related to the sequence number of the second image, the adjustment amplitude of the adjusted n frames of second images decreases frame by frame, and the adjusted n frames of second images play a transitional role.
[0308] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.
[0309] As an implementation of the above embodiments, the present application discloses an image processing device. Figure 7 As shown in the structural schematic diagram, the image processing device includes: a processor 1110 and a transceiver interface 1120.
[0310] Among them, the transceiver interface 1110 is used to obtain a reference image, m frames of first images and n frames of second images, the reference image is the image taken by the first camera before switching, the first image is the image taken by the second camera after switching, the second image is the image taken by the second camera after switching, and the shooting time of the second image is later than the shooting time of the first image.
[0311] In addition, the processor 1120 is used to adjust the parameters of each pixel in the m-frame first image according to the parameters of each pixel in the reference image, the parameters at least including the brightness of the pixel, the parameters of each pixel in the reference image are reference parameters, the parameters of each pixel in the first image after adjustment are first parameters, and the mean value of the first parameters is the same as the mean value of the reference parameters;
[0312] The processor 1120 is further configured to adjust the parameters of each pixel in the n frames of the second image respectively according to the parameters of each pixel in the reference image to obtain the adjusted second image, and the adjustment amplitude of the n frames of the second image decreases frame by frame.
[0313] Through the solution provided in the embodiment of the present application, the mean value of the first parameter of the first image taken by the second camera can be made the same as the mean value of the reference parameter, thereby avoiding excessive brightness difference between the first image and the reference image.
[0314] In addition, through the solution provided in the embodiment of the present application, the parameters of each pixel in the n frames of the second image can be adjusted to obtain the adjusted second image, wherein the adjustment amplitude of the n frames of the second image decreases frame by frame.
[0315] The adjustment range of an image can reflect the difference between the image before and after adjustment. Generally, the smaller the adjustment range of an image, the smaller the difference between the parameters of the image after adjustment and before adjustment, that is, the image after adjustment is closer to the image before adjustment.
[0316] Since in the solution provided in the embodiment of the present application, the adjustment amplitude of the n frames of second images decreases frame by frame, the parameters of the second image with a later shooting time are closer to the original parameters of the second image after adjustment. Correspondingly, the parameters of the image with an earlier shooting time among the n frames of second images are closer to the parameters of the reference image, and as the shooting time goes by, the parameters of the image with a later shooting time among the n frames of second images are closer to the parameters of the original second image shot by the second camera. Among them, if the adjustment amplitude of the latest image shot among the n frames of second images is 0, the parameters of the latest image shot among the n frames of second images after adjustment are the same as the parameters before adjustment, and the parameters of the image before and after adjustment remain unchanged.
[0317] In this case, the second camera also captures other images after capturing the second image, and the adjusted second image can serve as a transition between the first image and the other images, thereby facilitating the user to adapt to the parameters of the other images and avoiding excessive differences between the parameters of the other images and the parameters of the adjusted second image.
[0318] In the embodiment of the present application, the parameters include brightness. Therefore, through the solution provided in the embodiment of the present application, the brightness differences between the first image, the second image and the reference image can be reduced, and the brightness of the adjusted second image plays a transitional role between the brightness of the first image and the other images, so that the user can adapt to the brightness of the other images taken by the second camera.
[0319] That is to say, the solution provided in the embodiment of the present application can reduce and solve the problem in the prior art that after switching cameras with different zoom ratios, there is a large difference in brightness between the image taken before the switch and the image taken after the switch.
[0320] Furthermore, the solution provided in the embodiment of the present application can process images taken in various environments, and can be applied to various environments. Therefore, compared with the first solution of the prior art (i.e., the solution for calibration in a specified environment), the solution provided in the embodiment of the present application is applicable to a wider range of environments.
[0321] In addition, the solution provided by the embodiment of the present application does not need to run the first camera and the second camera at the same time. It only needs to start the second camera after switching to the second camera and process the image captured by the second camera. Therefore, compared with the second solution of the prior art (i.e., the solution of running the camera before switching and the camera after switching at the same time), the solution provided by the embodiment of the present application can reduce the overall power consumption of the terminal device, reduce the heating phenomenon of the terminal device, and improve the performance of the terminal device.
[0322] Furthermore, in the embodiment of the present application, the pixel parameters may include other parameters in addition to brightness, for example, the parameters may include at least one of chromaticity parameters, red color channel parameters, green color channel parameters, and blue color channel parameters. In this case, through the solution of the embodiment of the present application, the differences in other parameters between the first image, the second image, and the reference image may be reduced, and the adjusted other parameters of the second image may also play a transitional role between other parameters of the first image and other images, the other images being images captured by the second camera after the second image is captured.
[0323] Furthermore, in the device provided in the embodiment of the present application, the processor is specifically configured to perform normalization processing on the parameters of each pixel in the first target image by using the following formula:
[0324]
[0325] The first target image is any one of the m frames of first images, original val is the original value of the parameter of the first pixel in the first target image, average cur is the average value of the parameters of each pixel in the first target image before the standardization process, stdev cur is the standard deviation of the parameters of each pixel in the first target image before the normalization process, new val is a parameter of the first pixel in the first target image after normalization;
[0326] The processor is specifically configured to perform de-standardization on the first target image after the standardization process by using the following formula, wherein the first target image after the de-standardization process is the adjusted first image:
[0327] processed val =new val *stdev ref +average ref ;
[0328] Among them, new valis the parameter of the first pixel in the first target image after normalization, stdev ref is the standard deviation of the parameters of each pixel in the reference image, average ref is the average value of the parameters of each pixel in the reference image, processed val is a parameter of the first pixel in the adjusted first target image.
[0329] Furthermore, in the device provided in the embodiment of the present application, the processor is specifically used to adjust the parameters of each pixel in the n frames of second images according to the reference parameters and the parameter threshold, and the second image adjusted according to the reference parameters and the parameter threshold is the first image to be processed; and, according to the smoothing coefficients corresponding to the n frames of second images, the parameters of each pixel in the first image to be processed are adjusted, and the first image to be processed after adjustment according to the smoothing coefficient is the adjusted second image.
[0330] In a feasible design, the processor is specifically used for:
[0331] Downsampling the reference image and the second target image respectively, obtaining a first small image corresponding to the reference image, and obtaining a second small image corresponding to the second target image, wherein the second target image is any one frame of the n frames of second images, and the first small image and the second small image have the same size;
[0332] Determine a first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the same coordinates;
[0333] According to a first threshold, respectively adjusting the first parameter ratio corresponding to each coordinate, wherein the adjusted first parameter ratio is not greater than a first threshold upper limit in the first threshold, or the adjusted first parameter ratio is not less than a first threshold lower limit in the first threshold, wherein the parameter threshold includes the first threshold;
[0334] According to the coordinates respectively corresponding to the adjusted first parameter ratios, bilinear interpolation processing is performed on the adjusted first parameter ratios, and each first parameter ratio after the processing corresponds to a pixel in the second target image;
[0335] Each of the first parameter ratios after the processing is dot-multiplied with the corresponding parameter of the second target image, and the dot-multiplication result is the parameter of each pixel in the adjusted second target image.
[0336] The processor is specifically used for:
[0337] The first parameter ratio corresponding to the first coordinate is greater than the first upper threshold value, and the first parameter ratio corresponding to the first coordinate is adjusted to the first upper threshold value;
[0338] Alternatively, the first parameter ratio corresponding to the second coordinate is smaller than the first threshold lower limit, and the first parameter ratio corresponding to the second coordinate is adjusted to the first threshold lower limit.
[0339] In a feasible design, the processor is specifically used for:
[0340] Determine a second parameter ratio between the parameters of the reference image and the parameters of a third target image respectively corresponding to the same coordinates, wherein the third target image is any one frame of the n frames of second images;
[0341] According to the second threshold, respectively adjust the second parameter ratio corresponding to each coordinate, the adjusted second parameter ratio is not greater than the second threshold upper limit in the second threshold, or the adjusted second parameter ratio is not less than the second threshold lower limit in the second threshold, the parameter threshold includes the second threshold;
[0342] The second parameter ratios corresponding to the adjusted coordinates are respectively multiplied by the corresponding parameters of the third target image, and the dot product result is the parameter of each pixel in the adjusted third target image.
[0343] In a feasible design, the processor is specifically used for:
[0344] Determine a smoothing coefficient corresponding to a fourth target image according to the number of frames of the first image and the number of frames of the second image, wherein the fourth target image is any frame image of the first image to be processed;
[0345] Determining, according to the smoothing coefficient corresponding to the fourth target image, the standard deviation of the parameters of each pixel in the fourth target image after adjustment;
[0346] Determining an adjusted average value of the parameters of each pixel in the fourth target image according to the parameter smoothing coefficient of the fourth target image and the adjusted standard deviation;
[0347] Determine, according to the smoothing coefficient of the parameter of the fourth target image and the average value after adjustment, a ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image;
[0348] According to the ratio, the adjusted parameters of each pixel in the fourth target image are determined.
[0349] In a feasible design, the processor is specifically configured to determine a smoothing coefficient corresponding to the fourth target image by using the following formula:
[0350]
[0351] Wherein, dampRatio is the smoothing coefficient corresponding to the fourth target image, stableFrameCnt is the frame number of the first image, transitFrameCnt is the frame number of the second image, curFrameCnt is the sequence number of the fourth target image, the sequence numbers of the first to-be-processed images are incremented in sequence according to the shooting time, and the sequence numbers of the first to-be-processed images are all positive integers;
[0352] Alternatively, the processor is specifically configured to determine the adjusted standard deviation of the parameters of each pixel in the fourth target image by using the following formula:
[0353] dampStdev ref =stdev ref *dampRatio+stdev cur *(1-dampRatio);
[0354] Among them, dampStdev ref is the standard deviation of the parameters of each pixel in the fourth target image after adjustment, stdev ref is the standard deviation of the parameters of each pixel in the reference image, stdev cur is the standard deviation of the parameters of each pixel in the fourth target image before adjustment;
[0355] Alternatively, the processor is specifically configured to determine the adjusted average value of the parameters of each pixel in the fourth target image by using the following formula:
[0356] dampAverage ref =average ref *dampRatio+average cur *(1-dampRatio);
[0357] Among them, dampAverage ref is the average value of the parameters of each pixel in the fourth target image after adjustment, average ref is the average value of the parameters of each pixel in the reference image, average cur is the average value of the parameters of each pixel in the fourth target image before adjustment;
[0358] Alternatively, the processor is specifically configured to determine the ratio of the parameter of the same pixel in the fourth target image after adjustment to the parameter of the same pixel before adjustment according to the following formula:
[0359] dampRatio(x2,y2)=Ratio(x2,y2)*dampRatio+1*(1-dampRatio);
[0360] Wherein, dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment, and Ratio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the reference image to the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment;
[0361] Alternatively, the processor is specifically configured to determine the adjusted parameters of each pixel in the fourth target image according to the following formula:
[0362] Param(x2,y2)=dampRatio(x2,y2)*param(x2,y2);
[0363] Among them, Param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image after adjustment, and param(x2, y2) is the parameter of the pixel with coordinates (x2, y2) in the fourth target image before adjustment.
[0364] Correspondingly, corresponding to the above method, the present application embodiment also discloses a terminal device. Figure 8 As shown in the structural schematic diagram, the terminal device includes:
[0365] at least one processor 1101 and a memory,
[0366] Wherein, the memory is used to store program instructions;
[0367] The processor is used to call and execute the program instructions stored in the memory so that the terminal device executes Figure 2 , Figure 3 , Figure 5 and Figure 6 All or part of the steps in the corresponding embodiments.
[0368] Furthermore, the terminal device may also include: a transceiver 1102 and a bus 1103 , and the memory includes a random access memory 1104 and a read-only memory 1105 .
[0369] The processor is coupled to the transceiver, random access memory and read-only memory through the bus. When the terminal device needs to be operated, the basic input and output system solidified in the read-only memory or the bootloader boot system in the embedded system is used to start the terminal device and guide the terminal device into a normal operating state. After the terminal device enters the normal operating state, the application program and the operating system are run in the random access memory, so that the terminal device executes Figure 2 , Figure 3 , Figure 5 and Figure 6 All or part of the steps in the corresponding embodiments.
[0370] The device of the embodiment of the present invention may correspond to the above Figure 2 , Figure 3 , Figure 5 and Figure 6 The image processing device in the corresponding embodiment, and the processor in the device can implement Figure 2 , Figure 3 , Figure 5 and Figure 6 For the sake of brevity, the functions of the image processing apparatus in the corresponding embodiment or the various steps and methods implemented are not described in detail here.
[0371] In the scheme of the embodiment of the present application, the terminal device may be in various forms. Exemplarily, the terminal device may include a mobile phone. In this case, the mobile phone includes multiple cameras, which together constitute a multi-camera module, and different cameras have different zoom ratios. When a user uses a mobile phone to shoot, the mobile phone can switch different cameras according to the zoom ratio required by the user, so that shooting can be performed through different cameras to meet the user's shooting needs. In addition, the mobile phone can perform image processing on the image captured by the switched camera according to the scheme provided in the embodiment of the present application to reduce the difference in brightness of the image captured before and after the camera is switched.
[0372] In addition, the terminal device may also be in other forms, which is not limited in the embodiments of the present application.
[0373] In a specific implementation, the present application also provides a computer-readable storage medium, which includes instructions. The computer-readable storage medium set in any device can implement the following when it is executed on a computer: Figure 2 , Figure 3 , Figure 5 and Figure 6The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0374] In addition, another embodiment of the present application further discloses a computer program product comprising instructions, when the computer program product is run on an electronic device, the electronic device can implement the following steps: Figure 2 , Figure 3 , Figure 5 and Figure 6 All or part of the steps in the corresponding embodiments.
[0375] The various illustrative logic units and circuits described in the embodiments of the present application can be implemented or operated by a general-purpose processor, a digital information processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or the design of any combination of the above functions. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital information processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital information processor core, or any other similar configuration to implement.
[0376] The steps of the method or algorithm described in the embodiments of the present application can be directly embedded in hardware, a software unit executed by a processor, or a combination of the two. The software unit can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a UE. Optionally, the processor and the storage medium can also be arranged in different components in the UE.
[0377] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0378] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0379] The same or similar parts between the various embodiments of this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.
[0380] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.
[0381] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the embodiment of the road constraint determination device disclosed in this application, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.
[0382] The above-described embodiments of the present invention do not limit the protection scope of the present invention.
Claims
1. An image processing method, characterized in that: include: According to the parameters of each pixel in the reference image, respectively adjust the parameters of each pixel in the m-frame first image, wherein the reference image is an image taken by the first camera before switching, and the first image is an image taken by the second camera after switching, the parameters at least include the brightness of the pixels, the parameters of each pixel in the reference image are the reference parameters, and the parameters of each pixel in the first image after adjustment are the first parameters, the mean of the first parameters is the same as the mean of the reference parameters, and m is a positive integer; According to the parameters of each pixel in the reference image, the parameters of each pixel in the n frames of the second image are adjusted respectively to obtain the adjusted second image, where the second image is an image taken by the second camera after switching, and the shooting time of the second image is later than the shooting time of the first image. The adjustment amplitude of the n frames of the second image decreases frame by frame, and n is a positive integer.
2. The method according to claim 1, characterized in that The adjusting the parameters of each pixel in the m-frame first image respectively according to the parameters of each pixel in the reference image comprises: The parameters of each pixel in the first target image are standardized by the following formula: The first target image is any one of the m frames of first images, original val is the original value of the parameter of the first pixel in the first target image, average cur is the average value of the parameters of each pixel in the first target image before the standardization process, stdev cur is the standard deviation of the parameters of each pixel in the first target image before the normalization process, new val is a parameter of the first pixel in the first target image after normalization; The first target image after the standardization process is subjected to a de-standardization process by the following formula, and the first target image after the de-standardization process is the adjusted first image: processed val =new val *stdev ref +average ref ; Among them, new val is the parameter of the first pixel in the first target image after normalization, stdev ref is the standard deviation of the parameters of each pixel in the reference image, average ref is the average value of the parameters of each pixel in the reference image, processed val is a parameter of the first pixel in the adjusted first target image.
3. The method according to claim 1, characterized in that The adjusting the parameters of each pixel in the n frames of the second image respectively according to the parameters of each pixel in the reference image comprises: According to the reference parameters and the parameter threshold, respectively adjust the parameters of each pixel in the n frames of the second image, and the second image adjusted according to the reference parameters and the parameter threshold is the first image to be processed; According to the smoothing coefficients respectively corresponding to the n frames of second images, the parameters of each pixel in the first image to be processed are adjusted respectively, and the first image to be processed adjusted according to the smoothing coefficients is the adjusted second image.
4. The method according to claim 3, characterized in that: The adjusting the parameters of each pixel in the n frames of the second image respectively according to the reference parameter and the parameter threshold comprises: Downsampling the reference image and the second target image respectively, obtaining a first small image corresponding to the reference image, and obtaining a second small image corresponding to the second target image, wherein the second target image is any one frame of the n frames of second images, and the first small image and the second small image have the same size; Determine a first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the same coordinates; According to a first threshold, respectively adjusting the first parameter ratio corresponding to each coordinate, wherein the adjusted first parameter ratio is not greater than a first threshold upper limit in the first threshold, or the adjusted first parameter ratio is not less than a first threshold lower limit in the first threshold, wherein the parameter threshold includes the first threshold; According to the coordinates respectively corresponding to the adjusted first parameter ratios, bilinear interpolation processing is performed on the adjusted first parameter ratios, and each first parameter ratio after the processing corresponds to a pixel in the second target image; Each of the first parameter ratios after the processing is dot-multiplied with the corresponding parameter of the second target image, and the dot-multiplication result is the parameter of each pixel in the adjusted second target image.
5. The method according to claim 4, characterized in that The adjusting the first parameter ratios corresponding to the respective coordinates according to the first threshold respectively includes: The first parameter ratio corresponding to the first coordinate is greater than the first upper threshold value, and the first parameter ratio corresponding to the first coordinate is adjusted to the first upper threshold value; Alternatively, the first parameter ratio corresponding to the second coordinate is smaller than the first threshold lower limit, and the first parameter ratio corresponding to the second coordinate is adjusted to the first threshold lower limit.
6. The method according to claim 3, characterized in that The adjusting the parameters of each pixel in the n frames of the second image respectively according to the reference parameter and the parameter threshold comprises: Determine a second parameter ratio between the parameters of the reference image and the parameters of a third target image respectively corresponding to the same coordinates, wherein the third target image is any one frame of the n frames of second images; According to the second threshold, respectively adjust the second parameter ratio corresponding to each coordinate, the adjusted second parameter ratio is not greater than the second threshold upper limit in the second threshold, or the adjusted second parameter ratio is not less than the second threshold lower limit in the second threshold, the parameter threshold includes the second threshold; The second parameter ratios corresponding to the adjusted coordinates are respectively multiplied by the corresponding parameters of the third target image, and the dot product result is the parameter of each pixel in the adjusted third target image.
7. The method according to claim 3, characterized in that The adjusting the parameters of each pixel in the first image to be processed respectively according to the smoothing coefficients respectively corresponding to the n frames of second images includes: Determine a smoothing coefficient corresponding to a fourth target image according to the number of frames of the first image and the number of frames of the second image, wherein the fourth target image is any frame image of the first image to be processed; Determining, according to the smoothing coefficient corresponding to the fourth target image, the standard deviation of the parameters of each pixel in the fourth target image after adjustment; Determining an adjusted average value of the parameters of each pixel in the fourth target image according to the parameter smoothing coefficient of the fourth target image and the adjusted standard deviation; Determine, according to the smoothing coefficient of the parameter of the fourth target image and the average value after adjustment, a ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image; According to the ratio, the adjusted parameters of each pixel in the fourth target image are determined.
8. The method according to claim 7, characterized in that The smoothing coefficient corresponding to the fourth target image is determined by the following formula: Wherein, dampRatio is the smoothing coefficient corresponding to the fourth target image, stableFrameCnt is the frame number of the first image, transitFrameCnt is the frame number of the second image, curFrameCnt is the sequence number of the fourth target image, the sequence numbers of the first to-be-processed images are incremented in sequence according to the shooting time, and the sequence numbers of the first to-be-processed images are all positive integers; Alternatively, the adjusted standard deviation of the parameters of each pixel in the fourth target image is determined by the following formula: dampStdev ref =stdev ref *dampRatio+stdev cur *(1-dampRatio); Among them, dampStdev ref is the standard deviation of the parameters of each pixel in the fourth target image after adjustment, stdev ref is the standard deviation of the parameters of each pixel in the reference image, stdev cur is the standard deviation of the parameters of each pixel in the fourth target image before adjustment, and dampRatio is the smoothing coefficient corresponding to the fourth target image; Alternatively, the adjusted average value of the parameters of each pixel in the fourth target image is determined by the following formula: dampAverage ref =average ref *dampRatio+average cur *(1-dampRatio); Among them, dampAverage ref is the average value of the parameters of each pixel in the fourth target image after adjustment, average ref is the average value of the parameters of each pixel in the reference image, average cur is the average value of the parameters of each pixel in the fourth target image before adjustment, and dampRatio is the smoothing coefficient corresponding to the fourth target image; Alternatively, the ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image is determined according to the following formula: dampRatio(x2,y2)=Ratio(x2,y2)*dampRatio+1*(1-dampRatio); Wherein, dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment, Ratio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the reference image to the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment, and dampRatio is the smoothing coefficient corresponding to the fourth target image; Alternatively, the adjusted parameters of each pixel in the fourth target image are determined according to the following formula: Param(x2,y2)=dampRatio(x2,y2)*param(x2,y2); Among them, Param(x2,y2) is the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment, param(x2,y2) is the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment, and dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment.
9. An image processing device, characterized in that: include: Processor and transceiver interface; The transceiver interface is used to obtain a reference image, m frames of first images, and n frames of second images, wherein the reference image is an image taken by the first camera before switching, the first image is an image taken by the second camera after switching, and the second image is an image taken by the second camera after switching. The shooting time of the second image is later than the shooting time of the first image, and m is a positive integer and n is a positive integer; The processor is used to adjust the parameters of each pixel in the m frames of the first image respectively according to the parameters of each pixel in the reference image, the parameters at least including the brightness of the pixel, the parameters of each pixel in the reference image are reference parameters, the parameters of each pixel in the first image after adjustment are first parameters, and the mean value of the first parameters is the same as the mean value of the reference parameters; The processor is further configured to adjust the parameters of each pixel in the n frames of second images respectively according to the parameters of each pixel in the reference image to obtain the adjusted second images, wherein the adjustment amplitude of the n frames of second images decreases frame by frame.
10. The device according to claim 9, characterized in that The processor is specifically used to perform standardization processing on the parameters of each pixel in the first target image by using the following formula: The first target image is any one of the m frames of first images, original val is the original value of the parameter of the first pixel in the first target image, average cur is the average value of the parameters of each pixel in the first target image before the standardization process, stdev cur is the standard deviation of the parameters of each pixel in the first target image before the normalization process, new val is a parameter of the first pixel in the first target image after normalization; The processor is specifically configured to perform de-standardization on the first target image after the standardization process by using the following formula, wherein the first target image after the de-standardization process is the adjusted first image: processed val =new val *stdev ref +average ref ; Among them, new val is the parameter of the first pixel in the first target image after normalization, stdev ref is the standard deviation of the parameters of each pixel in the reference image, average ref is the average value of the parameters of each pixel in the reference image, processed val is a parameter of the first pixel in the adjusted first target image.
11. The device according to claim 9, characterized in that The processor is specifically used to adjust the parameters of each pixel in the n frames of second images according to the reference parameters and the parameter threshold, and the second image adjusted according to the reference parameters and the parameter threshold is the first image to be processed; and, according to the smoothing coefficients corresponding to the n frames of second images, respectively, adjust the parameters of each pixel in the first image to be processed, and the first image to be processed adjusted according to the smoothing coefficient is the adjusted second image.
12. The device according to claim 11, characterized in that The processor is specifically used for: Downsampling the reference image and the second target image respectively, obtaining a first small image corresponding to the reference image, and obtaining a second small image corresponding to the second target image, wherein the second target image is any one frame of the n frames of second images, and the first small image and the second small image have the same size; Determine a first parameter ratio between the parameters of the first small image and the parameters of the second small image corresponding to the same coordinates; According to a first threshold, respectively adjusting the first parameter ratio corresponding to each coordinate, wherein the adjusted first parameter ratio is not greater than a first threshold upper limit in the first threshold, or the adjusted first parameter ratio is not less than a first threshold lower limit in the first threshold, wherein the parameter threshold includes the first threshold; According to the coordinates respectively corresponding to the adjusted first parameter ratios, bilinear interpolation processing is performed on the adjusted first parameter ratios, and each first parameter ratio after the processing corresponds to a pixel in the second target image; Each of the first parameter ratios after the processing is dot-multiplied with the corresponding parameter of the second target image, and the dot-multiplication result is the parameter of each pixel in the adjusted second target image.
13. The device according to claim 12, characterized in that The processor is specifically used for: The first parameter ratio corresponding to the first coordinate is greater than the first upper threshold value, and the first parameter ratio corresponding to the first coordinate is adjusted to the first upper threshold value; Alternatively, the first parameter ratio corresponding to the second coordinate is smaller than the first threshold lower limit, and the first parameter ratio corresponding to the second coordinate is adjusted to the first threshold lower limit.
14. The device according to claim 11, characterized in that The processor is specifically used for: Determine a second parameter ratio between the parameters of the reference image and the parameters of a third target image respectively corresponding to the same coordinates, wherein the third target image is any one frame of the n frames of second images; According to the second threshold, respectively adjust the second parameter ratio corresponding to each coordinate, the adjusted second parameter ratio is not greater than the second threshold upper limit in the second threshold, or the adjusted second parameter ratio is not less than the second threshold lower limit in the second threshold, the parameter threshold includes the second threshold; The second parameter ratios corresponding to the adjusted coordinates are respectively multiplied by the corresponding parameters of the third target image, and the dot product result is the parameter of each pixel in the adjusted third target image.
15. The device according to claim 11, characterized in that The processor is specifically used for: Determine a smoothing coefficient corresponding to a fourth target image according to the number of frames of the first image and the number of frames of the second image, wherein the fourth target image is any frame image of the first image to be processed; Determining, according to the smoothing coefficient corresponding to the fourth target image, the standard deviation of the parameters of each pixel in the fourth target image after adjustment; Determining an adjusted average value of the parameters of each pixel in the fourth target image according to the parameter smoothing coefficient of the fourth target image and the adjusted standard deviation; Determine, according to the smoothing coefficient of the parameter of the fourth target image and the average value after adjustment, a ratio of the parameter after adjustment to the parameter before adjustment of the same pixel in the fourth target image; According to the ratio, the adjusted parameters of each pixel in the fourth target image are determined.
16. The device according to claim 15, characterized in that The processor is specifically configured to determine a smoothing coefficient corresponding to the fourth target image by using the following formula: Wherein, dampRatio is the smoothing coefficient corresponding to the fourth target image, stableFrameCnt is the frame number of the first image, transitFrameCnt is the frame number of the second image, curFrameCnt is the sequence number of the fourth target image, the sequence numbers of the first to-be-processed images are incremented in sequence according to the shooting time, and the sequence numbers of the first to-be-processed images are all positive integers; Alternatively, the processor is specifically configured to determine the adjusted standard deviation of the parameters of each pixel in the fourth target image by using the following formula: dampStdev ref =stdev ref *dampRatio+stdev cur *(1-dampRatio); Among them, dampStdev ref is the standard deviation of the parameters of each pixel in the fourth target image after adjustment, stdev ref is the standard deviation of the parameters of each pixel in the reference image, stdev cur is the standard deviation of the parameters of each pixel in the fourth target image before adjustment, and dampRatio is the smoothing coefficient corresponding to the fourth target image; Alternatively, the processor is specifically configured to determine the adjusted average value of the parameters of each pixel in the fourth target image by using the following formula: dampAverage ref =average ref *dampRatio+average cur *(1-dampRatio); Among them, dampAverage ref is the average value of the parameters of each pixel in the fourth target image after adjustment, average ref is the average value of the parameters of each pixel in the reference image, average cur is the average value of the parameters of each pixel in the fourth target image before adjustment, and dampRatio is the smoothing coefficient corresponding to the fourth target image; Alternatively, the processor is specifically configured to determine the ratio of the parameter of the same pixel in the fourth target image after adjustment to the parameter of the same pixel before adjustment according to the following formula: dampRatio(x2,y2)=Ratio(x2,y2)*dampRatio+1*(1-dampRatio); Wherein, dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment, Ratio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the reference image to the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment, and dampRatio is the smoothing coefficient corresponding to the fourth target image; Alternatively, the processor is specifically configured to determine the adjusted parameters of each pixel in the fourth target image according to the following formula: Param(x2,y2)=dampRatio(x2,y2)*param(x2,y2); Among them, Param(x2,y2) is the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment, param(x2,y2) is the parameter of the pixel with coordinates (x2,y2) in the fourth target image before adjustment, and dampRatio(x2,y2) is the ratio of the parameter of the pixel with coordinates (x2,y2) in the fourth target image after adjustment to the parameter before adjustment.
17. A terminal device, characterized in that: include: at least one processor and memory, The memory is used to store program instructions; The processor is used to call and execute the program instructions stored in the memory so that the terminal device executes the image processing method according to any one of claims 1 to 8.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the image processing method according to any one of claims 1 to 8.
Citation Information
Patent Citations
An exposure time adjusting method, device and a camera
CN101267505A
Image shooting brightness control method and device
CN104301624A
Camera automatic shutter and automatic gain adjustment method
CN104917976A
High-dynamic-range picture switching method and device
CN105872311A
Image fusion method and apparatus, and terminal device
CN108541374A
