Image processing method and device, electronic equipment, chip and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a storage medium, and relates to the field of image processing.The method comprises the steps that a first camera in the electronic equipment is responded to be switched to a second camera, or the first camera and the second camera shoot at the same time; acquiring a first down-sampling image associated with the first camera and a second down-sampling image associated with the second camera; acquiring a local mapping graph for mapping the color parameter of the second down-sampling image to the color parameter of the first down-sampling image; and adjusting color parameters of the source video stream collected by the second camera based on the local mapping graph. Therefore, smooth transition of image quality effects such as image brightness, color, contrast and the like on the electronic equipment before and after multi-camera switching or when multi-camera switching is carried out at the same time can be realized, the problem of sudden change of the image quality is avoided, and compared with global consistent transformation, the local mapping graph can be finely adjusted for different areas of the image, and a more natural and real visual effect is provided.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, chip and storage medium. Background Art

[0002] Electronic devices (including but not limited to smartphones, XR (Extended Reality) devices, in-vehicle systems, etc.) are often equipped with multiple cameras to meet users' imaging needs for objects in different locations. However, due to factors such as manufacturing process, cost, and environment, the imaging quality between different cameras often varies greatly, and this difference will also change with changes in the actual shooting environment. As users' demand for high-definition image quality and smooth switching continues to increase, image quality smoothness processing is often required in application scenarios such as multi-camera switching and multi-camera simultaneous operation. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the present application proposes an image processing method, device, electronic device, chip and storage medium to achieve a smooth transition of image quality effects such as image brightness, color, contrast, etc. before and after multi-camera switching or when multiple cameras are turned on at the same time on an electronic device, avoid the problem of sudden change in image quality, and provide a more natural and realistic visual effect.

[0005] In one aspect, an embodiment of the present application provides an image processing method, including:

[0006] In response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, acquiring a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera;

[0007] Obtaining a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image;

[0008] Based on the first local mapping image, the color parameters of the source video stream captured by the second camera are adjusted.

[0009] Another aspect of the present application provides an image processing device, including:

[0010] A first acquisition module, configured to acquire a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera in response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously;

[0011] A second acquisition module, configured to acquire a first local mapping map that maps the color parameters of the second down-sampled image to the color parameters of the first down-sampled image;

[0012] An adjustment module is used to adjust the color parameters of the source video stream collected by the second camera based on the first local mapping map.

[0013] Another aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method as described in the aforementioned aspect is implemented.

[0014] Another aspect of the present application provides a chip, which includes an interface circuit and a processing circuit coupled to each other, wherein the interface circuit is used to input or output a signal, and the processing circuit is configured to execute the image processing method as described in the aforementioned aspect.

[0015] Another aspect of the present application provides a non-temporary computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the image processing method as described in the aforementioned aspect.

[0016] Another aspect of the present application provides a computer program product on which a computer program is stored. When the program is executed by a processor, the image processing method as described in the above aspect is implemented.

[0017] The image processing method, device, electronic device, chip and storage medium proposed in the present application, when the first camera switches to the second camera, or the two cameras are shooting at the same time, first, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are respectively obtained, and then, by using these two down-sampled images with relatively low resolution, a first local mapping map that maps the color parameters (including brightness, color, and contrast) of the second down-sampled image to the color parameters of the first down-sampled image is calculated, which can greatly reduce the amount of calculation and significantly improve the speed and efficiency of subsequent image processing. Subsequently, the first local mapping map is used to adjust the color parameters of the source video stream captured by the second camera, which can not only achieve a smooth transition of image quality effects such as image brightness, color, and contrast before and after multi-camera switching or when multiple cameras are turned on at the same time on the electronic device, avoiding the problem of sudden change in image quality, but also compared with a globally consistent transformation, the local mapping map can be finely adjusted for different areas of the image to provide a more natural and realistic visual effect. In addition, due to its simple and lightweight characteristics, the local mapping map can be used as an image post-processing algorithm on the acquisition end. It can run in parallel with other post-processing algorithms (such as beauty, distortion correction, etc.) to ensure real-time processing during preview and recording. This efficient and flexible processing method not only improves the user experience, but also enhances the overall performance and reliability of the system.

[0018] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of a flow chart of a first image processing method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a second image processing method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a flow chart of a third image processing method provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a fourth image processing method provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a fifth image processing method provided in an embodiment of the present application;

[0025] Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the implementation principle of any embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0029] Fig.10 It is a schematic diagram of the structure of a chip proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0030] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] In the related technology, the following three solutions are mainly used to achieve smooth processing of image quality in multi-camera switching scenarios:

[0032] The first method: for switching the scene from a camera with a large field of view (marked as the first camera) to a camera with a small field of view (marked as the second camera), first, the overlapping area of ​​the captured images of the first camera and the second camera is detected, and then the global tone mapping (Global Tone Mapping, abbreviated as GTM) / local tone mapping (Local Tone Mapping, abbreviated as LTM) parameters of the captured image of the first camera are applied to the corresponding overlapping area of ​​the captured image of the second camera through image signal processing (Image Signal Processing, abbreviated as ISP).

[0033] When calculating the LTM parameters of the overlapping area, the acquired image needs to be divided into blocks to obtain multiple image blocks, and the LTM parameters of the multiple image blocks are calculated and associated and then applied in blocks.

[0034] The second method is to calibrate different cameras offline to obtain the color mapping parameters of different cameras under different ambient light. When online, the corresponding color mapping parameters are selected based on the common area or color temperature between different cameras and applied in the ISP processing process.

[0035] The third method: For panoramic image stitching scenes, after acquiring the captured images of all cameras, first extract the overlapping areas, then calculate the color mapping function for the overlapping areas, and map the color brightness of each captured image to the target image one by one, and then fuse them into a panoramic image. Among them, the target image can be selected as the image with the highest estimated frequency of white points in all captured images. The color mapping function can be obtained by histogram matching calculation, or the mean brightness of pixels that do not pass through the bin (interval used to represent brightness, color or other characteristic values) can be mapped to the target image through a piecewise gamma curve. The two different color mapping function calculation methods can be performed separately or sequentially.

[0036] However, the first solution mentioned above has at least the following problems: it is necessary to explicitly calculate the overlapping areas of the captured images, the algorithm is complex, inefficient, and has high overhead; it can only handle the situation of switching from a large field of view angle to a small field of view angle; it can only handle the consistency of overlapping areas, and there may be spatial inconsistencies for non-overlapping areas with consistent textures; it can only handle contrast differences and cannot achieve color alignment of multiple cameras.

[0037] The second solution mentioned above has at least the following problems: its accuracy depends on offline calibration, and may not be very effective in some scenes with complex light sources or not covered by offline calibration; it can only achieve globally consistent color correction, and may not perform well in multi-light source scenarios; it is processed through the ISP color correction module, which can only process differences in color performance, and cannot achieve brightness alignment and contrast alignment of multiple cameras.

[0038] The third solution mentioned above has at least the following problems: it is necessary to explicitly calculate the overlapping areas of the acquired images, the algorithm is complex, inefficient, and has high overhead; the color mapping function is calculated directly on the corresponding single color channel of the given color gamut, and the coupling information between different color channels cannot be considered, so the effect is limited.

[0039] Therefore, in response to at least one of the problems existing in the above-mentioned related technologies, the present application proposes an image processing method, device, electronic device, chip and storage medium.

[0040] The image processing method, device, electronic device, chip and storage medium of the embodiments of the present application are described below with reference to the accompanying drawings.

[0041] Figure 1 A schematic flowchart of the first image processing method provided in an embodiment of the present application.

[0042] It should be noted that the image processing method of the embodiment of the present application can be applied to an image processing device. In some possible embodiments, the image processing device can be configured in an electronic device or a chip so that the electronic device or the chip can perform an image processing function. In addition, in some possible embodiments, the image processing device can also be software in an electronic device.

[0043] In any embodiment of the present application, the chip can be integrated into an electronic device. Among them, the chip includes a central processing unit (CPU), an image signal processing (ISP), an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a reduced instruction set computer (RISC), etc., which are not listed one by one this time.

[0044] like Figure 1 As shown, the image processing method may include the following steps S101 to S103:

[0045] Step S101, in response to a first camera in an electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are acquired.

[0046] The electronic device includes a plurality of cameras, and the plurality of cameras include at least a first camera and a second camera.

[0047] It should be noted that the present application does not limit the switching method between multiple cameras. For example, the switching between multiple cameras can be triggered by a zoom operation (such as the user zooming in or out of the shooting screen), or the camera to be switched can be selected through a button or menu on the shooting interface, thereby triggering the switching between multiple cameras.

[0048] The first down-sampled image is obtained by down-sampling the image captured by the first camera; and the second down-sampled image is obtained by down-sampling the image captured by the second camera.

[0049] In an embodiment of the present application, when a first camera in an electronic device is switched to a second camera, or the first camera and the second camera are turned on or shoot at the same time, a first down-sampled image associated with the first camera can be acquired, and a second down-sampled image associated with the second camera can be acquired.

[0050] It should be noted that the image processing method provided in any embodiment of the present application can be applied to a real-time preview scenario, or can also be applied to a video recording scenario, and the embodiments of the present application do not limit this.

[0051] Step S102: Obtain a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image.

[0052] Among them, the first local mapping map is dynamically generated according to different local pixels or local areas in the image, and is used to indicate the multidimensional mapping coefficients of mapping the color parameters of multiple color channels of each pixel in the second downsampled image to the color parameters of multiple color channels of the corresponding pixel in the first downsampled image.

[0053] The multiple color channels include but are not limited to: three color channels: red (Red, referred to as R), green (Green, referred to as G), and blue (Blue, referred to as B).

[0054] The color parameters include but are not limited to: brightness, color, contrast, etc.

[0055] Exemplarily, the first local mapping image may include a mapping matrix for each pixel point (hereinafter referred to as the first pixel point) in the second downsampled image; wherein the mapping matrix of each first pixel point is used to indicate the multidimensional mapping coefficients for mapping the color parameters of multiple color channels of the first pixel point to the color parameters of multiple color channels of the corresponding second pixel point in the first downsampled image.

[0056] In any embodiment of the present application, a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image may be predicted or calculated based on deep learning technology or machine learning technology.

[0057] Step S103: adjusting the color parameters of the source video stream captured by the second camera based on the first local mapping image.

[0058] In an embodiment of the present application, the color parameters of the source video stream (referred to as the second source video stream in the present application) captured by the second camera can be adjusted based on the first local mapping map.

[0059] As an example, when adjusting the color parameters of the kth (k is a positive integer) frame image in the second source video stream based on the first local mapping map, it is assumed that the color parameters of each pixel are adjusted based on a linear mapping method. For example, if the mapping matrix is ​​a 3*3 color correction matrix (Color Correction Matrix, CCM for short), the original RGB color parameters of each pixel in the kth frame image are expressed as follows: The CCM of the pixel in the first local map is expressed as: The following formula can be used to calculate the final RGB color parameter representation of the pixel after adjustment: out :

[0060]

[0061] That is: R out =a*R in +b*G in +c*B in ;

[0062] G out =d*R in +e*G in +f*B in ;

[0063] B out =g*R in +h*G in +i*B in ;

[0064] As another example, when adjusting the color parameters of the k-th frame image in the second source video stream based on the first local mapping map, assuming a nonlinear mapping method based on a polynomial (excluding cross terms), the color parameters of each pixel are adjusted. For example, the mapping matrix M can be a 3*9 matrix, and the original RGB color parameters of each pixel in the k-th frame image are expressed as follows: The mapping matrix M of the pixel point in the first local mapping image can be expressed as: Then according to Constructing 9-dimensional feature vector The following formula is used to calculate the final RGB color parameter representation of the pixel after adjustment based on M and X: out :

[0065]

[0066] Right now:

[0067]

[0068] As another example, when adjusting the color parameters of the k-th frame image in the second source video stream based on the first local mapping map, it is assumed that a nonlinear mapping method based on a polynomial (including cross terms, such as including R in G in and R in B in , but ignore G in B in To simplify the calculation), the color parameters of each pixel are adjusted. For example, the mapping matrix M can be a 3*9 matrix, and the original RGB color parameters of each pixel in the k-th frame image are expressed as: The mapping matrix M of the pixel point in the first local mapping image can be expressed as: Then according to Constructing 9-dimensional feature vector And use the above formula (2) to calculate the final RGB color parameter representation of the pixel after adjustment according to M and X: out ,Right now:

[0069]

[0070] According to the image processing method of the embodiment of the present application, when the first camera switches to the second camera, or the two cameras are shooting at the same time, first, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are respectively obtained, and then, by using these two down-sampled images with relatively low resolutions, a first local mapping map that maps the color parameters (including brightness, color, and contrast) of the second down-sampled image to the color parameters of the first down-sampled image is calculated, which can greatly reduce the amount of calculation and significantly improve the speed and efficiency of subsequent image processing. Subsequently, the first local mapping map is used to adjust the color parameters of the source video stream captured by the second camera, which can not only achieve a smooth transition of image quality effects such as image brightness, color, and contrast before and after multi-camera switching or when multiple cameras are turned on at the same time on an electronic device, and avoid the problem of sudden image quality changes, but also, compared with a globally consistent transformation, the local mapping map can be finely adjusted for different areas of the image to provide a more natural and realistic visual effect. In addition, due to its simple and lightweight characteristics, the local mapping map can be used as an image post-processing algorithm on the acquisition end. It can run in parallel with other post-processing algorithms (such as beauty, distortion correction, etc.) to ensure real-time processing during preview and recording. This efficient and flexible processing method not only improves the user experience, but also enhances the overall performance and reliability of the system.

[0071] The present application embodiment provides another image processing method. Figure 2 A schematic flowchart of the second image processing method provided in an embodiment of the present application.

[0072] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0073] like Figure 2 As shown, the image processing method may include the following steps S201 to S204:

[0074] Step S201, in response to the first camera in the electronic device switching to the second camera, or the first camera and the second camera shooting simultaneously, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are acquired.

[0075] It should be noted that the explanation of step S201 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0076] Step S202, predicting a mapping matrix diagram based on the color parameters of the first down-sampled image and the color parameters of the second down-sampled image; wherein the mapping matrix diagram includes mapping matrices of multiple first image blocks in the second down-sampled image, and the mapping matrix is ​​used to indicate multidimensional mapping coefficients of color parameters of the first image blocks mapped to color parameters of corresponding second image blocks in the first down-sampled image.

[0077] In an embodiment of the present application, a deep learning technology or a machine learning technology may be used to predict a mapping matrix diagram according to the color parameters of the first down-sampled image and the color parameters of the second down-sampled image. The mapping matrix diagram includes mapping matrices of multiple first image blocks in the second down-sampled image, and the mapping matrix of each first image block is used to indicate a multi-dimensional mapping coefficient for mapping the color parameters of the first image block to the color parameters of the corresponding second image block in the first down-sampled image.

[0078] Taking the mapping matrix as a CCM matrix as an example, the mapping matrix diagram can also be called a local CCM diagram, wherein the local CCM diagram includes CCMs of multiple first image blocks, and the CCM of each first image block is used to indicate multiple mapping coefficients for mapping the color parameters of the first image block to the color parameters of the second image block corresponding to the first image block in the first downsampled image.

[0079] Exemplarily, assuming that the resolution of the second down-sampled image is 1080p, that is, the second down-sampled image includes 1920x1080 pixels, the first down-sampled image and the second down-sampled image can be input into a deep learning model (or machine learning model) for processing to obtain a local CCM image with a spatial resolution of 24x24 output by the deep learning model, wherein the local CCM image includes a CCM matrix of 24x24 image blocks.

[0080] The deep learning model can be obtained by offline training. For example, the following steps 1 to 4 can be used to train the deep learning model:

[0081] Step 1: Acquire an image pair, wherein the image pair includes a first sample image and a second sample image.

[0082] There is no restriction on the method of acquiring the image pair. For example, the first sample image and the second sample image in the image pair can be acquired from a training set or a test set. Alternatively, the first sample image and the second sample image in the image pair can be acquired by different cameras in an electronic device equipped with multiple cameras. For example, in the case of multiple camera switching, the image acquired before the camera switching can be used as the first sample image, and the image acquired after the camera switching can be used as the second sample image; for another example, in the case of multiple cameras shooting at the same time, the image acquired by one of the cameras can be used as the first sample image, and the image acquired by another camera can be used as the second sample image, and so on. The embodiments of the present application do not limit this.

[0083] Step 2: Process the image pair using a deep learning model to obtain a prediction mapping matrix diagram, wherein the prediction mapping matrix diagram includes a mapping matrix of multiple image blocks in the second sample image, and the mapping matrix is ​​used to indicate the multidimensional mapping coefficients of the color parameters of the image blocks in the second sample image mapped to the color parameters of the corresponding image blocks in the first down-sampled image.

[0084] Step 3: Based on the prediction mapping matrix diagram, the second sample image is mapped to obtain a target image.

[0085] Step 4: Based on the color difference between the target image and the first sample image, the deep learning model is trained.

[0086] Step S203: interpolate the mapping matrix to obtain a first local mapping map.

[0087] As an example, for each pixel in the second down-sampled image, four first image blocks adjacent to the first image block where the pixel is located are determined from multiple first image blocks, and bilinear or bicubic interpolation is performed based on the mapping matrices of these adjacent first image blocks to calculate the mapping matrix that should be used for the pixel, so that a first local mapping map can be generated based on the mapping matrices of each pixel in the second down-sampled image.

[0088] Step S204: adjusting the color parameters of the source video stream captured by the second camera based on the first local mapping image.

[0089] It should be noted that the explanation of step S204 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0090] The image processing method of the embodiment of the present application can be implemented based on deep learning technology to quickly and accurately calculate the first local mapping map that maps the color parameters of the second down-sampled image to the color parameters of the first down-sampled image.

[0091] The present application embodiment provides another image processing method. Figure 3 A schematic flowchart of the third image processing method provided in an embodiment of the present application.

[0092] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0093] like Figure 3 As shown, the image processing method may include the following steps S301 to S306:

[0094] Step S301, in response to the first camera in the electronic device switching to the second camera, or the first camera and the second camera shooting simultaneously, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are acquired.

[0095] It should be noted that the explanation of step S301 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0096] Step S302: predicting a plurality of mapping matrices corresponding to each third image block in the second down-sampled image based on the color parameters of the first down-sampled image and the color parameters of the second down-sampled image.

[0097] Each third image block corresponds to a plurality of mapping matrices, and each mapping matrix is ​​used to indicate a multi-dimensional mapping coefficient for mapping the color parameters of the third image block to the color parameters of the corresponding image block in the first down-sampled image.

[0098] In an embodiment of the present application, deep learning technology or machine learning technology can be used to predict multiple mapping matrices corresponding to each third image block in the second down-sampled image based on the color parameters of the first down-sampled image and the color parameters of the second down-sampled image.

[0099] Taking the mapping matrix as CCM as an example, in this application, the form of local segmented CCM can be used to predict multiple CCMs corresponding to each third image block. For example, a high dynamic range network (High Dynamic Range Network, HDRNet) can be used to predict multiple CCMs for each third image block in the second downsampled image. Among them, for each third image block, HDRNet can divide the grayscale dimension into N parts to obtain N grayscale value intervals, so that each grayscale value interval corresponds to a specific CCM. HDRNet can predict the N CCMs corresponding to each third image block based on the characteristics of each third image block (such as grayscale value distribution), wherein each CCM is used to adjust the pixel points belonging to its corresponding grayscale value interval, that is, in the same area of ​​the image (that is, the same third image block), by inputting the grayscale value function index of the third image block, the CCM to be used for different grayscale values ​​is obtained.

[0100] Step S303 , according to the color parameter of any pixel in the same third image block, determine two adjacent mapping matrices matching the color parameter of any pixel from a plurality of mapping matrices corresponding to the same third image block.

[0101] Still taking the above example as an example, for each pixel in the second down-sampled image, two adjacent CCMs matching the grayscale value of the pixel can be determined from the N CCMs corresponding to the third image block where the pixel is located according to the grayscale value of the pixel. For example, the grayscale value of the pixel is between the grayscale value intervals corresponding to the two CCMs.

[0102] Step S304: interpolate two adjacent mapping matrices to obtain a mapping matrix of any pixel point.

[0103] The mapping matrix of each pixel is used to indicate the multi-dimensional mapping coefficients for mapping the color parameters of multiple color channels of the pixel in the second down-sampled image to the color parameters of multiple color channels of the corresponding pixel in the first down-sampled image.

[0104] For example, taking the mapping matrix as CCM, assuming that the grayscale value of a pixel is 0.65 (normalized grayscale value), and multiple grayscale value intervals are: [0, 0.25], [0.25, 0.5], [0.5, 0.75] and [0.75, 1], then the grayscale value of the pixel is between [0.5, 0.75] and [0.75, 1]. In this application, the CCM corresponding to the two grayscale value intervals [0.5, 0.75] and [0.75, 1] ​​can be selected, and linear interpolation can be performed to obtain the CCM suitable for the pixel.

[0105] Step S305: Generate a first local mapping image according to the mapping matrix of each pixel in the second down-sampled image.

[0106] That is, the first local mapping image includes a mapping matrix of each pixel in the second down-sampled image.

[0107] Step S306: Adjust the color parameters of the source video stream captured by the second camera based on the first local mapping image.

[0108] It should be noted that the explanation of step S306 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0109] The image processing method of the embodiment of the present application can use multiple methods to calculate the first local mapping map that maps the color parameters of the second down-sampled image to the color parameters of the first down-sampled image, which can improve the flexibility and applicability of the method.

[0110] The present application embodiment provides another image processing method. Figure 4 A schematic flowchart of the fourth image processing method provided in an embodiment of the present application.

[0111] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0112] like Figure 4 As shown, the image processing method may include the following steps S401 to S403:

[0113] Step S401, in response to a first camera and a second camera in an electronic device shooting simultaneously, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are acquired.

[0114] The first down-sampled image is the i-th down-sampled image in the first down-sampled video stream, and the first down-sampled video stream is obtained by down-sampling the first source video stream captured by the first camera.

[0115] The first source video stream is shot or collected by the first camera according to a default resolution or a resolution set by the user.

[0116] The second down-sampled image is the j-th down-sampled image in the second down-sampled video stream, and the second down-sampled video stream is obtained by down-sampling the second source video stream captured by the second camera.

[0117] The second source video stream is shot or collected by the second camera according to a default resolution or a resolution set by the user.

[0118] Among them, the i-th frame image in the first source video stream and the j-th frame image in the second source video stream are captured at the same time, or the i-th frame image in the first source video stream and the j-1-th frame image in the second source video stream are captured at the same time.

[0119] Among them, i and j are both positive integers.

[0120] Exemplarily, the first downsampled image may be obtained by downsampling the current frame image captured by the first camera, or by downsampling the previous frame image captured by the first camera; the second downsampled image may be obtained by downsampling the current frame image captured by the second camera.

[0121] Step S402: Obtain a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image.

[0122] It should be noted that the explanation of step S402 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0123] Step S403: adjusting the color parameters of the j-th frame image in the second source video stream based on the first local mapping map.

[0124] In the embodiment of the present application, the color parameters of the j-th frame image in the second source video stream can be adjusted based on the first local mapping map. That is, in a scene where the first camera and the second camera are turned on at the same time, the color parameters of the current frame image captured by the second camera can be adjusted based on the color parameters of the previous frame image or the current frame image captured by the first camera to maintain the continuity and consistency of the image quality.

[0125] The image processing method of the embodiment of the present application adjusts the color parameters (such as brightness, color, and contrast) of the image captured by the second camera to be consistent with the image captured by the first camera, so that in a scene where multiple cameras are turned on at the same time, the continuity and consistency of image quality between different cameras can be maintained, thereby improving the user experience in the shooting scene.

[0126] The present application embodiment provides another image processing method. Figure 5 A schematic flowchart of the fifth image processing method provided in an embodiment of the present application.

[0127] It should be noted that the image processing method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the related technology, and the embodiments of the present application are not limited to this.

[0128] like Figure 5 As shown, the image processing method may include the following steps S501 to S504:

[0129] Step S501: in response to a first camera in an electronic device switching to a second camera, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are acquired.

[0130] The second down-sampled image is a first frame down-sampled image in a second down-sampled video stream, and the second down-sampled video stream is obtained by down-sampling a second source video stream captured by a second camera.

[0131] The second source video stream is shot or collected by the second camera according to a default resolution or a resolution set by the user.

[0132] The first down-sampled image is the last down-sampled image frame in the first down-sampled video stream, and the first down-sampled video stream is obtained by down-sampling the first source video stream captured by the first camera.

[0133] The first source video stream is shot or collected by the first camera according to a default resolution or a resolution set by the user.

[0134] In an embodiment of the present application, when switching between multiple cameras of an electronic device, such as switching from a first camera to a second camera, a first down-sampled image associated with the first camera can be obtained, and a second down-sampled image associated with the second camera can be obtained.

[0135] Step S502: Obtain a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image.

[0136] It should be noted that the explanation of step S502 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0137] Step S503: mapping the first frame image and the second down-sampled image in the second source video stream according to the first local mapping map.

[0138] In an embodiment of the present application, the color parameters of the first frame image in the second source video stream can be adjusted (i.e., mapping processing) based on the first local mapping map, and the color parameters of the second down-sampled image can also be adjusted based on the first local mapping map. The implementation principle is similar to that of step S103 and will not be repeated here.

[0139] Step S504, performing at least one cycle according to the unit local mapping image and the processed second down-sampled image to adjust the color parameters of the non-first frame images in the second source video stream.

[0140] The unit local mapping map is used to keep the color parameters of the image unchanged during the image mapping process, that is, when the unit local mapping map is used to map the image, no content in the image is changed. For example, the mapping matrix of the multiple pixels included in the first local mapping map is taken as CCM. The CCMs of the multiple pixels included in the unit local mapping map are all unit matrices.

[0141] In any embodiment of the present application, the first cycle process in at least one cycle process includes the following steps A to E:

[0142] Step A: obtaining a second local mapping map that maps the color parameters of the second frame downsampled image in the second downsampled video stream to the color parameters of the processed second downsampled image. The implementation principle is similar to step S102 and will not be repeated here.

[0143] Step B: Smoothing the second local mapping map according to the unit local mapping map to obtain a smoothed second local mapping map.

[0144] As an example, the second local mapping map and the unit local mapping map can be weighted added according to a set rule to obtain a smoothed second local mapping map. The set rule of weighted addition can adopt different strategies such as linear attenuation and exponential attenuation, which can be selected according to actual application requirements. For example, taking the mapping matrix as CCM as an example, when different strategies are weighted added, the following formula can be used to weight the second local mapping map and the unit local mapping map:

[0145] ccm_map' n =(1-wn )×ccm_map n +w n ×ccm_map identity ; (3)

[0146] Among them, the weights w corresponding to different strategies n The calculation method is different; ccm_map n Represents the nth local map, where n is a positive integer greater than or equal to 2; ccm_map' n Represents the nth local map after smoothing; ccm_map identity Represents a unit local map.

[0147] Step C: Calculate the difference between the smoothed second local map and the unit local map, and determine whether the difference is less than or equal to a set difference threshold. If so, execute step D; if not, execute step E.

[0148] It should be noted that step D and step E are two parallel implementation methods. In actual application, only one needs to be executed.

[0149] Step D: When the difference between the smoothed second local mapping map and the unit local mapping map is less than or equal to the difference threshold, the second frame image in the second source video stream is mapped only based on the smoothed second local mapping map, and the loop process is terminated to stop mapping the frame images after the second frame image in the second source video stream, that is, the color parameters of the third frame image and subsequent images in the second source video stream are kept unchanged.

[0150] Step E: When the difference between the smoothed second local mapping map and the unit local mapping map is greater than a set difference threshold, mapping processing is performed on the second frame downsampled image and the second frame image in the second source video stream based on the smoothed second local mapping map.

[0151] In any embodiment of the present application, at least one non-first cycle process in a cycle process (such as the nth cycle process, where n is a positive integer greater than or equal to 2) includes the following steps A' to E':

[0152] Step A': obtaining the n+1th local mapping map that maps the color parameters of the n+1th downsampled image in the second downsampled video stream to the color parameters of the processed nth downsampled image. The implementation principle is similar to step S102 and will not be repeated here.

[0153] Step B': Smoothing the n+1th local mapping map according to the unit local mapping map to obtain a smoothed n+1th local mapping map. The implementation principle is similar to step B and will not be described in detail here.

[0154] Step C': Calculate the difference between the smoothed n+1th local mapping image and the unit local mapping image, and determine whether the difference is less than or equal to a set difference threshold. If yes, execute step D'; if no, execute step E'.

[0155] It should be noted that step D' and step E' are two parallel implementation methods. In actual application, only one needs to be executed.

[0156] Step D': When the difference between the smoothed n+1th local mapping map and the unit local mapping map is less than or equal to the difference threshold, the n+1th frame image in the second source video stream is mapped only based on the smoothed n+1th local mapping map, and the loop process is terminated to stop mapping the frame images after the n+1th frame image in the second source video stream, that is, the color parameters of the n+2th frame image and subsequent images in the second source video stream are kept unchanged.

[0157] Step E': When the difference between the smoothed n+1th local mapping map and the unit local mapping map is greater than the set difference threshold, based on the smoothed n+1th local mapping map, the n+1th frame downsampled image and the n+1th frame image in the second source video stream are mapped respectively.

[0158] The image processing method of the embodiment of the present application can align the color parameters (including brightness, color, and contrast) of the second source video stream to the color parameters of the first source video stream at the moment of camera switching, and then smoothly transition to the picture quality of the second source video stream itself, which can not only improve visual continuity but also ensure the consistency of the pictures between different cameras.

[0159] In any embodiment of the present application, the application scenario is an exemplary description of the switching scenario between multiple cameras. When a user uses an electronic device including multiple cameras to shoot, the user can trigger a zoom operation, first waking up the first camera to work, and then triggering the mutual switching between different cameras when zooming to a corresponding preset magnification, waking up the second camera to work. The electronic device at least includes the following: Figure 6 The modules shown include: multiple cameras (such as a first camera, a second camera, ..., an S camera, where S is a positive integer greater than or equal to 2), an image processing unit, a data transmission interface (i.e., a communication interface), an internal storage unit, an independent computing unit, etc. The independent computing unit supports neural network reasoning calculations.

[0160] The internal storage unit includes, but is not limited to, a random access memory (RAM), a multi-level cache, a register, and the like.

[0161] For example, the application scenario is a switching scenario between multiple cameras. The implementation principle of each embodiment of the present application can be as follows: Figure 7 As shown, it mainly includes the following steps:

[0162] 1. The first camera collects and processes to obtain a first down-sampled video stream and a first source video stream.

[0163] Among them, the resolution of the source video stream collected by each camera is relatively large, which is not conducive to real-time calculation. The source video stream collected by each camera can be downsampled to obtain a downsampled video stream, wherein the downsampled video stream and the source video stream have the same video screen, basically the same processing parameters, each frame corresponds to each other, and only differs in resolution, and there is basically no difference in image quality effect.

[0164] Optionally, in order to reduce the data transmission bandwidth, the size of the first down-sampled video stream and the second down-sampled video stream can be reduced to 256x256, which can also greatly improve the computing efficiency.

[0165] 2. When the user triggers camera switching from the first camera to the second camera, the second camera collects and processes to obtain a second down-sampled video stream and a second source video stream.

[0166] 3. At the moment of camera switching, the last frame of the first down-sampled video stream before switching is sent to display (corresponding to the real-time preview scene) or the last frame of the image sent to editor (corresponding to the video recording scene) (referred to as the first down-sampled image in this application) and the first frame of the second down-sampled video stream after switching is sent to display (corresponding to the real-time preview scene) or the image sent to editor (corresponding to the video recording scene) (referred to as the second down-sampled image in this application), through Figure 6 The data transmission interface and the internal storage unit are transmitted to the independent computing unit;

[0167] 4. The independent computing unit calculates and obtains a first local mapping map that maps the color parameters (or image quality effect) of the second down-sampled image to the color parameters (or image quality effect) of the first down-sampled image.

[0168] Exemplarily, the independent computing unit may input the second down-sampled image and the first down-sampled image into a neural network model, and the model may predict the first local map.

[0169] Optionally, the neural network model can be learned by offline training of the neural network.

[0170] Among them, the local mapping map can be calculated using the downsampled video stream, and the calculation can be completed before the corresponding frame of the source video stream completes the image processing. After the source video stream completes the image processing, the local mapping effect is directly performed, and real-time processing can be achieved.

[0171] Optionally, the calculation process of the local map can be performed in parallel with other post-processing algorithms after the downsampled video stream completes image processing.

[0172] 5. Apply the first local mapping image to the first frame image in the second source video stream, and transmit the processed first frame image to subsequent modules, such as display (corresponding to real-time preview scenario) / editor (corresponding to video recording scenario).

[0173] 6. At the same time, the first local mapping image is applied to the second down-sampled image to obtain a processed second down-sampled image for subsequent video frame processing.

[0174] 7. The nth frame of the downsampled image in the second downsampled video stream is passed through Figure 6 The data transmission interface and internal storage unit are transmitted to the independent computing unit.

[0175] 8. The independent computing unit calculates the color parameters (or image quality effect) of the nth frame down-sampled image and maps them into the nth local mapping map of the color parameters (or image quality effect) of the processed n-1th frame down-sampled image.

[0176] 9. Weighted addition of the nth local mapping image and the unit local mapping image is performed according to a certain rule to obtain a smoothed nth local mapping image.

[0177] 10. Apply the smoothed nth local mapping image to the nth frame image in the second source video stream, and transmit the processed nth frame image to subsequent modules, such as display (corresponding to the real-time preview scene) / editor (corresponding to the video recording scene).

[0178] 11. Apply the smoothed nth local mapping image to the nth down-sampled image to obtain a processed nth down-sampled image.

[0179] 12. Repeat steps 7 to 11 until the smoothed Mth local mapping image gradually approaches the unit local mapping image.

[0180] Therefore, after M frames, the picture quality of the second source video stream gradually changes from the picture quality of the last frame sent for display or editing with the first source video stream to the picture quality of the second source video stream itself, thereby achieving smooth changes in the camera switching process and improving the user's shooting experience.

[0181] In summary, at the moment of camera switching, the brightness, color, and contrast of the second source video stream can be aligned to the picture quality of the first source video stream at the same time, and smoothly transition to the picture quality of the second source video stream itself within the next 1 to 2 seconds; and, there is no need for explicit image matching, and the complexity is low; it can act on the switching between cameras with field of view angle changes in any relative relationship, thereby improving the flexibility and applicability of the method; by taking effect of the local mapping map, local transformation of the image can be achieved rather than just a globally consistent transformation; by taking effect of the local mapping map, the relationship between each color channel is fully considered to improve the image quality; the local mapping map is simple and lightweight, and as an image post-processing algorithm, it can be paralleled with other post-processing algorithms (such as beauty, distortion correction, etc.), and real-time processing of the preview and recording process can be achieved.

[0182] In order to implement the above embodiment, the embodiment of the present application also proposes an image processing device.

[0183] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0184] like Figure 8 As shown, the image processing device 800 may include: a first acquisition module 810 , a second acquisition module 820 and an adjustment module 830 .

[0185] The first acquisition module 810 is used to acquire a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera in response to the first camera in the electronic device switching to the second camera, or the first camera and the second camera shooting simultaneously;

[0186] A second acquisition module 820, configured to acquire a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image;

[0187] The adjustment module 830 is used to adjust the color parameters of the source video stream collected by the second camera based on the first local mapping map.

[0188] Furthermore, in an implementation method of an embodiment of the present application, the second acquisition module 820 is used to: predict a mapping matrix diagram based on color parameters of the first down-sampled image and color parameters of the second down-sampled image; wherein the mapping matrix diagram includes mapping matrices of multiple first image blocks in the second down-sampled image, and the mapping matrix is ​​used to indicate multidimensional mapping coefficients of color parameters of the first image blocks mapped to corresponding color parameters of the second image blocks in the first down-sampled image; and interpolate the mapping matrix diagram to obtain a first local mapping diagram.

[0189] In one implementation of the embodiment of the present application, the second acquisition module 820 is used to: predict multiple mapping matrices corresponding to each third image block in the second down-sampled image based on the color parameters of the first down-sampled image and the color parameters of the second down-sampled image; determine two adjacent mapping matrices that match the color parameters of any pixel point from the multiple mapping matrices corresponding to the same third image block according to the color parameters of any pixel point in the same third image block; interpolate the two adjacent mapping matrices to obtain the mapping matrix of any pixel point; and generate a first local mapping map according to the mapping matrix of each pixel point in the second down-sampled image.

[0190] In one implementation of the embodiment of the present application, in response to simultaneous shooting by a first camera and a second camera, the first down-sampled image is the i-th down-sampled image in the first down-sampled video stream, and the first down-sampled video stream is obtained by downsampling the first source video stream captured by the first camera; the second down-sampled image is the j-th down-sampled image in the second down-sampled video stream, and the second down-sampled video stream is obtained by downsampling the second source video stream captured by the second camera; wherein the i-th frame image in the first source video stream and the j-th frame image in the second source video stream are captured simultaneously, or the i-th frame image in the first source video stream and the j-1-th frame image in the second source video stream are captured simultaneously.

[0191] In an implementation of the embodiment of the present application, the adjustment module 830 is used to adjust the color parameters of the j-th frame image in the second source video stream based on the first local mapping map.

[0192] In one implementation of the embodiment of the present application, in response to the first camera switching to the second camera, the first down-sampled image is the last frame of the down-sampled image in the first down-sampled video stream, and the first down-sampled video stream is obtained by downsampling the first source video stream captured by the first camera; the second down-sampled image is the first frame of the down-sampled image in the second down-sampled video stream, and the second down-sampled video stream is obtained by downsampling the second source video stream captured by the second camera.

[0193] In an implementation of the embodiment of the present application, the adjustment module 830 is used to: perform mapping processing on the first frame image and the second down-sampled image in the second source video stream respectively according to the first local mapping map; perform at least one cycle process according to the unit local mapping map and the processed second down-sampled image to adjust the color parameters of the non-first frame images in the second source video stream;

[0194] The unit local mapping map is used to keep the color parameters of the image unchanged during image mapping processing.

[0195] In one implementation of the embodiment of the present application, the adjustment module 830 performs a first loop process in at least one loop process, specifically: obtaining a second local mapping map that maps the color parameters of the second frame downsampled image in the second downsampled video stream to the color parameters of the processed second downsampled image; smoothing the second local mapping map according to the unit local mapping map to obtain a smoothed second local mapping map; when the difference between the smoothed second local mapping map and the unit local mapping map is less than or equal to the difference threshold, mapping the second frame image in the second source video stream based on the smoothed second local mapping map, and ending the loop process to stop mapping each frame image after the second frame image in the second source video stream; or, when the difference between the smoothed second local mapping map and the unit local mapping map is greater than the difference threshold, mapping the second frame downsampled image and the second frame image in the second source video stream based on the smoothed second local mapping map.

[0196] In one implementation of the embodiment of the present application, the adjustment module 830 performs an nth loop process in at least one loop process, specifically: obtaining an n+1th local mapping map that maps the color parameters of the n+1th frame downsampled image in the second downsampled video stream to the color parameters of the processed nth frame downsampled image; wherein n is a positive integer greater than 1; smoothing the n+1th local mapping map according to the unit local mapping map to obtain a smoothed n+1th local mapping map; and performing a phase shifting operation between the smoothed n+1th local mapping map and the unit local mapping map. When the difference between the smoothed n+1th local mapping map and the unit local mapping map is less than or equal to the difference threshold, the n+1th frame image in the second source video stream is mapped based on the smoothed n+1th local mapping map, and the loop process is ended to stop mapping the frame images after the n+1th frame image in the second source video stream; or, when the difference between the smoothed n+1th local mapping map and the unit local mapping map is greater than the difference threshold, the n+1th frame down-sampled image and the n+1th frame image in the second source video stream are mapped based on the smoothed n+1th local mapping map.

[0197] It should be noted that the above explanation of the embodiment of the image processing method is also applicable to the image processing device of this embodiment, and will not be repeated here.

[0198] In the image processing device of the embodiment of the present application, when the first camera switches to the second camera, or the two cameras are shooting at the same time, first, a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera are respectively obtained, and then, by using these two down-sampled images with relatively low resolution, a first local mapping map that maps the color parameters (including brightness, color, and contrast) of the second down-sampled image to the color parameters of the first down-sampled image is calculated, which can greatly reduce the amount of calculation and significantly improve the speed and efficiency of subsequent image processing. Subsequently, the first local mapping map is used to adjust the color parameters of the source video stream captured by the second camera, which can not only achieve a smooth transition of image quality effects such as image brightness, color, and contrast before and after multi-camera switching or when multiple cameras are turned on at the same time on an electronic device, and avoid the problem of sudden change in image quality, but also, compared with a globally consistent transformation, the local mapping map can be finely adjusted for different areas of the image to provide a more natural and realistic visual effect. In addition, due to its simple and lightweight characteristics, the local mapping map can be used as an image post-processing algorithm on the acquisition end. It can run in parallel with other post-processing algorithms (such as beauty, distortion correction, etc.) to ensure real-time processing during preview and recording. This efficient and flexible processing method not only improves the user experience, but also enhances the overall performance and reliability of the system.

[0199] In order to implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method as described in any of the above embodiments is implemented.

[0200] Fig. 9 This is a schematic diagram of another electronic device provided in an embodiment of the present application. For example, the electronic device 900 may be a vehicle, a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0201] Reference Fig. 9 The electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0202] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0203] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0204] The power component 906 provides power to the various components of the electronic device 900. The power component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0205] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0206] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0207] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0208] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of the components, such as the display and keypad of the electronic device 900, and the sensor assembly 914 can also detect the position change of the electronic device 900 or a component of the electronic device 900, the presence or absence of contact between the user and the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0209] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 900 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (Ultra-Wideband, UWB) technology, Bluetooth (BT) technology and other technologies.

[0210] In an exemplary embodiment, the electronic device 900 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above method.

[0211] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by a processor 920 of an electronic device 900 to complete the above method. For example, the non-transitory computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0212] In order to implement the above embodiments, the present application also proposes a chip, wherein the chip includes an interface circuit and a processing circuit coupled to each other. The interface circuit is used to input or output signals, and the processing circuit is configured to execute the image processing method provided in any of the above embodiments.

[0213] Fig.10 Schematic diagram of a chip structure proposed in the embodiment of the present application. Fig.10 The structure of the chip 1000 is shown, but is not limited to this.

[0214] The chip 1000 includes a processing circuit 1001 , and the processing circuit 1001 is configured to execute any of the above image processing methods.

[0215] In some embodiments, the chip 1000 further includes one or more interface circuits 1002. Optionally, the interface circuit 1002 is connected to the memory 1003. The interface circuit 1002 can be used to receive signals from the memory 1003 or other devices, and the interface circuit 1002 can be used to send signals to the memory 1003 or other devices. For example, the interface circuit 1002 can read instructions stored in the memory 1003 and send the instructions to the processing circuit 1001.

[0216] In some embodiments, the interface circuit 1002 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 1001 performs other steps.

[0217] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0218] In some embodiments, the chip 1000 further includes one or more memories 1003 for storing instructions. Optionally, all or part of the memory 1003 may be outside the chip 1000.

[0219] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method as described in any of the above method embodiments is implemented.

[0220] In order to implement the above embodiments, the present application also proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in any of the above method embodiments is implemented.

[0221] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0222] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0223] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0224] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic devices), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0225] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA for short), a field programmable gate array (FPGA for short), etc.

[0226] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0227] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0228] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: In response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously, acquiring a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera; Obtaining a first local mapping map that maps color parameters of the second down-sampled image to color parameters of the first down-sampled image; Based on the first local mapping image, the color parameters of the source video stream captured by the second camera are adjusted.

2. The method according to claim 1, characterized in that: The obtaining of a first local mapping map that maps the color parameters of the second down-sampled image to the color parameters of the first down-sampled image includes: Based on the color parameters of the first down-sampled image and the color parameters of the second down-sampled image, predict a mapping matrix diagram; wherein the mapping matrix diagram includes mapping matrices of a plurality of first image blocks in the second down-sampled image, and the mapping matrix is ​​used to indicate multidimensional mapping coefficients of color parameters of the first image blocks mapped to color parameters of corresponding second image blocks in the first down-sampled image; The mapping matrix is ​​interpolated to obtain the first local mapping map.

3. The method according to claim 1, characterized in that: The obtaining of a first local mapping map that maps the color parameters of the second down-sampled image to the color parameters of the first down-sampled image includes: Based on the color parameters of the first down-sampled image and the color parameters of the second down-sampled image, predict a plurality of mapping matrices corresponding to each third image block in the second down-sampled image; According to a color parameter of any pixel point in the same third image block, determining two adjacent mapping matrices matching the color parameter of any pixel point from a plurality of mapping matrices corresponding to the same third image block; Interpolating the two adjacent mapping matrices to obtain a mapping matrix of any pixel point; The first local mapping image is generated according to the mapping matrix of each pixel in the second down-sampled image.

4. The method according to claim 1, characterized in that In response to the first camera and the second camera shooting simultaneously, the first down-sampled image is the i-th frame down-sampled image in a first down-sampled video stream, and the first down-sampled video stream is obtained by downsampling the first source video stream captured by the first camera; The second down-sampled image is a j-th down-sampled image in a second down-sampled video stream, where the second down-sampled video stream is obtained by down-sampling a second source video stream captured by the second camera; The i-th frame image in the first source video stream and the j-th frame image in the second source video stream are captured at the same time, or the i-th frame image in the first source video stream and the j-1-th frame image in the second source video stream are captured at the same time.

5. The method according to claim 4, characterized in that The adjusting, based on the first local mapping image, the color parameter of the source video stream collected by the second camera includes: Based on the first local mapping map, the color parameters of the j-th frame image in the second source video stream are adjusted.

6. The method according to claim 1, characterized in that In response to the first camera switching to the second camera, the first down-sampled image is the last down-sampled image frame in a first down-sampled video stream, and the first down-sampled video stream is obtained by downsampling a first source video stream captured by the first camera; The second down-sampled image is a first frame down-sampled image in a second down-sampled video stream, and the second down-sampled video stream is obtained by down-sampling a second source video stream captured by the second camera.

7. The method according to claim 6, characterized in that The adjusting, based on the first local mapping image, the color parameter of the source video stream collected by the second camera includes: According to the first local mapping map, mapping processing is performed on the first frame image and the second down-sampled image in the second source video stream respectively; Perform at least one cycle according to the unit local mapping image and the processed second down-sampled image to adjust the color parameters of the non-first frame images in the second source video stream; The unit local mapping map is used to keep the color parameters of the image unchanged during image mapping processing.

8. The method according to claim 7, characterized in that The first cycle process in the at least one cycle process includes: Obtaining a second local mapping map that maps color parameters of a second frame of a down-sampled image in the second down-sampled video stream to color parameters of a processed second down-sampled image; Smoothing the second local mapping map according to the unit local mapping map to obtain a smoothed second local mapping map; When the difference between the smoothed second local mapping map and the unit local mapping map is less than or equal to the difference threshold, mapping processing is performed on the second frame image in the second source video stream based on the smoothed second local mapping map, and the loop process is terminated to stop mapping processing on each frame image after the second frame image in the second source video stream; or, When the difference between the smoothed second local mapping map and the unit local mapping map is greater than the difference threshold, mapping processing is performed on the second frame downsampled image and the second frame image in the second source video stream based on the smoothed second local mapping map.

9. The method according to claim 7, characterized in that: The nth cycle process in the at least one cycle process comprises: Obtaining an n+1th local mapping map that maps the color parameters of the n+1th down-sampled image frame in the second down-sampled video stream to the color parameters of the processed nth down-sampled image frame; wherein n is a positive integer greater than 1; Smoothing the n+1th local mapping map according to the unit local mapping map to obtain a smoothed n+1th local mapping map; When the difference between the smoothed n+1th local mapping map and the unit local mapping map is less than or equal to the difference threshold, mapping processing is performed on the n+1th frame image in the second source video stream based on the smoothed n+1th local mapping map, and the loop process is terminated to stop mapping processing on each frame image after the n+1th frame image in the second source video stream; or, When the difference between the smoothed n+1th local mapping map and the unit local mapping map is greater than the difference threshold, based on the smoothed n+1th local mapping map, mapping processing is performed on the n+1th frame downsampled image and the n+1th frame image in the second source video stream respectively.

10. An image processing device, characterized in that: include: A first acquisition module, configured to acquire a first down-sampled image associated with the first camera and a second down-sampled image associated with the second camera in response to a first camera in the electronic device switching to a second camera, or the first camera and the second camera shooting simultaneously; A second acquisition module, configured to acquire a first local mapping map that maps the color parameters of the second down-sampled image to the color parameters of the first down-sampled image; An adjustment module is used to adjust the color parameters of the source video stream collected by the second camera based on the first local mapping map.

11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A chip, characterized in that: The chip comprises an interface circuit and a processing circuit coupled to each other, the interface circuit is used to input or output a signal, and the processing circuit is used to implement the method according to any one of claims 1 to 9.

13. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images