Video processing method, electronic device, storage medium, chip system and computer program product

Abstract

The embodiment of the invention provides a video processing method, electronic equipment, a storage medium, a chip system and a computer program product, and relates to the technical field of terminals. The method comprises the following steps: under the condition of video recording, using a plurality of cameras to shoot a plurality of images obtained by a shot object to obtain video frames in a video; the plurality of cameras comprise a first camera which adopts a relatively small exposure time length to balance the frame rate and a second camera which adopts a relatively large exposure time length to improve the signal-to-noise ratio. Therefore, in the plurality of images, the first image shot by the first camera is used as the reference image, the second image shot by the second camera is used as the auxiliary image, the first image has a relatively small exposure time length, the second image has a relatively large exposure time length, and the reference image is subjected to noise reduction by using the auxiliary image to obtain the reference image subjected to noise reduction. The denoised reference image is a video frame in the video. Therefore, under the condition of balancing the frame rate, the video noise reduction effect is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a video processing method, an electronic device, a storage medium, a chip system, and a computer program product. Background Art

[0002] With the continuous development of electronic devices, the functions of electronic devices are becoming more and more abundant. For example, the video recording function provided by electronic devices can be used to realize video recording in various scenes. During the video recording process, noise will be generated, and the noise will affect the video quality. Videos in different scenes are affected by noise to different degrees. For example, videos obtained in dark light scenes are greatly affected by noise.

[0003] In order to improve the video quality, video noise reduction can be performed. In the related art, the video noise reduction effect is poor. Summary of the invention

[0004] The embodiments of the present application provide a video processing method, an electronic device, a storage medium, a chip system, and a computer program product, which are applied to the field of terminal technology and can improve the video noise reduction effect while balancing the frame rate.

[0005] In a first aspect, an embodiment of the present application proposes a video processing method. The method may include: in response to a first operation, recording a video. During the video recording process, the first image is denoised according to the second image to obtain a video frame. The video frame may be a frame in the recorded video. The first image and the second image may be taken using multiple cameras. The shooting time of the first image and the shooting time of the second image may overlap, and the lens direction of the first camera and the lens direction of the second camera are the same, indicating that the contents of the first image and the second image are similar. The first camera may be a camera that shoots the first image, and the second camera may be a camera that shoots the second image. The exposure time of the first image may be less than the exposure time of the second image. The first image may be used as a reference image. The second image may be used as an auxiliary image.

[0006] The video processing method provided in the embodiment of the present application is that, in the case of video recording, the first image is taken by a camera that uses a shorter exposure time to balance the frame rate, and the second image is taken by a camera that uses a longer exposure time to improve the signal-to-noise ratio, and the shooting time of the first image and the shooting time of the second image overlap, which indicates that the contents of the first image and the second image are similar. Therefore, the second image is used to reduce the noise of the first image to obtain the video frame in the video, thereby achieving the improvement of the video noise reduction effect while balancing the frame rate.

[0007] In a possible implementation, in response to the first operation, performing video recording may include: in response to the second operation, displaying a camera application interface. The camera application interface may include a low-light recording control. In response to a third operation on the low-light recording control, displaying a low-light recording interface. The low-light recording interface may include a start control. In response to the first operation on the start control, performing video recording.

[0008] Since the camera application can provide a low-light recording mode, low-light video recording can be performed using the camera application.

[0009] In another possible implementation, in response to the first operation, performing video recording may include: in response to detecting that the scene is in a dark light scene, displaying a dark light recording interface. The dark light recording interface may include a start control. In response to the first operation on the start control, performing video recording.

[0010] When a low-light scene is detected, a low-light recording interface is displayed to automatically start the low-light recording mode, thereby simplifying the steps for starting low-light recording and improving the convenience of low-light recording.

[0011] In another possible implementation, in response to the first operation, performing video recording may include: in response to the second operation, displaying a dark-light recording interface. The dark-light recording interface may include a start control. In response to the first operation on the start control, performing video recording.

[0012] When the second operation is detected, the dark light recording interface is displayed, so that when the camera application is started, the dark light recording mode is started by default. The dark light recording mode can be the default startup mode of the camera application or the last time the dark light recording mode was started, thereby simplifying the steps of starting the dark light recording and improving the convenience of the dark light recording.

[0013] In a possible implementation, the definition of the first image may be higher than that of the second image. And / or, the exposure time of the second image may be greater than the frame interval of the first image. The exposure time of the first image may be less than or equal to the frame interval.

[0014] By setting the exposure time of the reference image to be less than or equal to the frame interval of the reference image, a balanced frame rate is achieved, and by setting the exposure time of the auxiliary image to be greater than the frame interval of the reference image, an improved signal-to-noise ratio is achieved. Thus, the video noise reduction effect is improved while balancing the frame rate.

[0015] By making the clarity of the first image higher than that of the second image, the video noise reduction effect is further improved.

[0016] In a possible implementation manner, the first image may be a short frame image, and the second image may be a long frame image.

[0017] Since the first image is a short-frame image, the short-frame image can balance the frame rate, maintain the texture edge and highlight area information of the scene, and has high definition and low signal-to-noise ratio. The second image is a long-frame image, which has a high signal-to-noise ratio, low definition, richer dark light area information, and richer color information. Therefore, it can be shown that the long-frame image and the short-frame image can complement each other to a certain extent. For example, the high-noise sharp texture of the short-frame image and the low-noise fuzzy texture of the long-frame image can constrain each other, which is conducive to detail recovery. The highlight area information of the short-frame image can repair the overexposed cutoff area of ​​the long-frame image. In addition, the color information of the long-frame image can correctly restore the color of the scene. Therefore, the short-frame image has the advantage of balancing the frame rate and / or high definition, and the long-frame image has the advantage of improving the signal-to-noise ratio. By using the long-frame image as a supplement to the short-frame image, the quality of the short-frame image is improved, thereby achieving the improvement of the video noise reduction effect under the condition of balancing the frame rate.

[0018] In a possible implementation, the exposure time of the second image may be greater than or equal to 100 ms.

[0019] Since the exposure time of the second image is greater than or equal to 100 ms, the exposure time of the second image can improve the signal-to-noise ratio, thereby improving the video noise reduction effect.

[0020] In a possible implementation, the first image and the second image may have respective timestamps. The timestamp may be used as one of the bases for determining whether the first image and the second image have similar contents, and further may be used as a basis for whether the second image can be used to reduce noise in the first image. In a possible implementation, when it is determined that the timestamp of the first image and the timestamp of the second image are within a predetermined time range, if the lens orientation of the first camera is the same as the lens orientation of the second camera, it can be said that the contents of the first image and the second image are similar, and thus, the second image can be used to reduce noise in the first image.

[0021] In a possible implementation, reducing the noise of the first image according to the second image to obtain the video frame may include: fusing the first image and the second image to obtain the video frame.

[0022] Since the video frame is obtained by fusing the first image and the second image, the first image is taken by a camera that uses a shorter exposure time to balance the frame rate, and the second image is taken by a camera that uses a longer exposure time to improve the signal-to-noise ratio. Therefore, the video noise reduction effect is improved while balancing the frame rate.

[0023] In a possible implementation, fusing the first image and the second image to obtain a video frame may include: fusing the first image and a registered second image to obtain a video frame. The registered second image is obtained by registering the second image and the first image.

[0024] Since the video frame is obtained by fusing the first image and the registered second image, and the registered second image is obtained by registering the second image and the first image, image-level denoising is achieved. Since image-level denoising can make full use of pixel information, retain image detail information as much as possible, and has good interpretability, the video denoising effect is improved.

[0025] In a possible implementation, the registered second image is obtained by registering the second image and the first image, which may include: the registered second image may be obtained by registering the first image and the second image using an optical flow method. For example, the registered second image may be obtained by registering the second image and the first image using optical flow information. The optical flow information may be pixel-by-pixel relative displacement information between the first image and the second image.

[0026] The image registration between the second image and the first image is realized by using the optical flow method, thereby improving the efficiency of image registration.

[0027] In a possible implementation, fusing the first image and the second image to obtain a video frame may include: fusing the first image and an image converted by histogram to obtain a video frame. The image converted by histogram may be obtained by transferring the color distribution of the second image to the first image based on a histogram conversion function.

[0028] Since there is no spatial misalignment problem between the image obtained by histogram conversion and the first image, the probability of ghosting effect caused by the movement of the camera or the object is reduced. In addition, since the image registration step is not performed, the video noise reduction process is simplified.

[0029] In another possible implementation, fusing the first image and the second image to obtain a video frame may include: fusing the tenth feature data and the eleventh feature data to obtain the video frame. The tenth feature data and the eleventh feature data may be obtained based on the first feature data of the first image and the second feature data of the second image.

[0030] In a possible implementation, the tenth feature data and the eleventh feature data may be obtained based on the first feature data of the first image and the second feature data of the second image, and may include: the tenth feature data may be obtained by processing the first feature data of the first image and the second feature data of the second image using a cross-attention mechanism. The first feature data may be used for query. The second feature data may be used for keys and values. The eleventh feature data may be obtained by processing the first feature data of the first image and the second feature data of the second image using a cross-attention mechanism. The second feature data may be used for query. The first feature data may be used for keys and values.

[0031] In a possible implementation, the tenth feature data may be obtained by processing the first feature data of the first image and the second feature data of the second image using a cross-attention mechanism, and may include: the tenth feature data may be obtained based on the seventh attention weight matrix and the sixth value matrix. The seventh attention weight matrix may be obtained based on the seventh query matrix and the seventh key matrix. The seventh query matrix may be obtained by performing the fourteenth linear transformation on the first feature data of the first image. The seventh key matrix may be obtained by performing the fifteenth linear transformation on the second feature data of the second image. The seventh attention weight matrix may be used to indicate the similarity between the second feature and the second feature data.

[0032] In a possible implementation, the eleventh feature data may be obtained by processing the first feature data of the first image and the second feature data of the second image using a cross-attention mechanism, and may include: the eleventh feature data may be obtained based on the eighth attention weight matrix and the seventh value matrix. The eighth attention weight matrix may be obtained based on the eighth query matrix and the eighth key matrix. The eighth query matrix may be obtained by performing the sixteenth linear transformation on the second feature data of the second image. The eighth key matrix may be obtained by performing the seventeenth linear transformation on the first feature data of the first image. The eighth attention weight matrix may be used to indicate the similarity between the first feature and the second feature data.

[0033] Since the tenth feature data is obtained by processing the first feature data of the first image and the second feature data of the second image using the cross attention mechanism, the first feature data is used for query, and the second feature data is used for key and value, therefore, the tenth feature in the tenth feature data is obtained by fusing the second feature data with the first feature data. The eleventh feature data is obtained by processing the first feature data of the first image and the second feature data of the second image using the cross attention mechanism, the second feature data is used for query, and the first feature data can be used for key and value, therefore, the eleventh feature in the eleventh feature data is obtained by fusing the first feature data with the second feature data, thereby achieving more effective use of the first feature data and the second feature data. On this basis, the video frame is obtained by fusing the tenth feature data and the eleventh feature data, thereby improving the video noise reduction effect.

[0034] In another possible implementation, fusing the first image and the second image to obtain a video frame may include: fusing the first feature data of the first image and the registered second feature data to obtain the video frame. The registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image.

[0035] Since the first feature data abstracts the first image while retaining the detail information of the first image, and the second feature data abstracts the second image while retaining the detail information of the second image, the first feature data and the second feature data have higher robustness and discriminability. On this basis, the video frame is obtained by fusing the first feature data of the first image and the registered second feature data, and the registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image, thereby improving the reliability and robustness of video denoising and improving the video denoising effect.

[0036] In a possible implementation, the registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image, which may include: the registered second feature data may be obtained by registering the second feature data of the second image and the first feature data of the first image using a cross-attention mechanism. The first feature data may be used for query. The second feature data may be used for keys and values. The first feature data may include multiple first features. The second feature data may include multiple second features.

[0037] Since the cross-attention mechanism can mine the association between the second feature data and the first feature data, realize the interaction and matching of the first image and the second image at the feature level, thereby capturing the similarities and differences between the features, focusing on the similarities with high weights, suppressing the differences with low weights, and adaptively adjusting the weights, it has higher robustness and scalability. Therefore, by using the cross-attention mechanism to align the second feature data of the second image with the first feature data of the first image, the feature weight of the second feature in the second feature data can be adjusted. For example, a larger weight is assigned to the second feature in the second feature data that has a large similarity with the first feature data, and a smaller weight is assigned to the second feature in the second feature data that has a small similarity with the first feature data, thereby improving the accuracy of feature alignment and thus improving the video denoising effect.

[0038] In one possible implementation, the cross-attention mechanism may include at least one of the following: a dot product attention mechanism, a scaled dot product attention mechanism, an additive attention mechanism, a linear attention mechanism, or a sparse attention mechanism.

[0039] Since the video processing method described in the embodiment of the present application can be applied to resource-constrained electronic devices, such as terminal devices, the computational complexity of the cross-attention mechanism is proportional to the square of the number of feature data included in the feature data. Therefore, by adopting an optimization strategy based on optimizing the cross-attention mechanism, such as a linear attention mechanism and a sparse attention mechanism, the desired ability to reduce the computational complexity while maintaining or improving the accuracy of feature registration is achieved.

[0040] In a possible implementation, the registered second feature data may be obtained by registering the second feature data of the second image and the first feature data of the first image using a cross-attention mechanism, and may include: the registered second feature data may be obtained according to at least one fourth intermediate matrix. The fourth intermediate matrix may be obtained according to the third attention weight matrix and the fifth value matrix. The third attention weight matrix may be obtained according to the fourth query matrix and the fourth key matrix. The third attention weight matrix may be used to indicate the similarity between the second feature and the first feature data. The fourth query matrix may be obtained by performing the seventh linear transformation on the first feature data. For example, the fourth query matrix may be obtained according to the first feature data and the seventh transformation matrix. The fourth key matrix may be obtained by performing the eighth linear transformation on the second feature data. For example, the fourth key matrix may be obtained according to the second feature data and the eighth transformation matrix. The fifth value matrix may be obtained by performing the ninth linear transformation on the second feature data. For example, the fifth value matrix may be obtained according to the second feature data and the ninth transformation matrix.

[0041] It should be noted that, when there is one fourth intermediate matrix, the above method can be understood as a single-head cross attention mechanism. When there are multiple fourth intermediate matrices, the above method can be understood as a multi-head cross attention mechanism. The transformation matrix groups corresponding to at least one fourth intermediate matrix can be different. The transformation matrix group can include a seventh transformation matrix, an eighth transformation matrix, and a ninth transformation matrix. The element values ​​in the transformation matrix group can be model parameters of the deep learning model.

[0042] In a possible implementation, the third attention weight matrix may be obtained based on the fourth query matrix and the fourth key matrix, and may include: the third attention weight matrix may be obtained by normalizing the fifth intermediate matrix. For example, the third attention weight matrix may be obtained by processing the fifth intermediate matrix using a Softmax function, a Sigmoid function, or a tanh function. The fifth intermediate matrix may be obtained based on the dot product between the fourth query matrix and the fourth key matrix.

[0043] In another possible implementation, the third attention weight matrix may be obtained based on the fourth query matrix and the fourth key matrix, and may include: the third attention mechanism may be obtained by normalizing the sixth intermediate matrix. The sixth intermediate matrix may be obtained by performing the tenth linear transformation on the seventh intermediate matrix. For example, the sixth intermediate matrix may be obtained based on the seventh intermediate matrix and the tenth transformation matrix. The element values ​​in the tenth transformation matrix may be model parameters of the deep learning model. The seventh intermediate matrix may be obtained by performing nonlinear activation on the eighth intermediate matrix. The eighth intermediate matrix may be obtained based on the fourth query matrix and the fourth key matrix.

[0044] In a possible implementation, the eighth intermediate matrix may be obtained according to the fourth query matrix and the fourth key matrix, which may include: the eighth intermediate matrix may be obtained according to the fifth query matrix and the fifth key matrix. The fifth query matrix may be obtained by performing an eleventh linear transformation on the fourth query matrix. For example, the fifth query matrix may be obtained according to the fourth query matrix and the eleventh transformation matrix. The fifth key matrix may be obtained by performing a twelfth linear transformation on the fourth key matrix. For example, the fifth key matrix may be obtained according to the fourth key matrix and the twelfth transformation matrix.

[0045] In another possible implementation, the eighth intermediate matrix may be obtained according to the fourth query matrix and the fourth key matrix, which may include: the eighth intermediate matrix may be obtained by performing a thirteenth linear transformation on the ninth intermediate matrix. For example, the eighth intermediate matrix may be obtained according to the ninth intermediate matrix and the thirteenth transformation matrix. The ninth intermediate matrix may be obtained by concatenating the fourth query matrix and the fourth key matrix.

[0046] In a possible implementation, the optimization strategy may include at least one of the following: a linear strategy, a sparse strategy, a quantization strategy, a pruning strategy, or other strategies.

[0047] By adopting an optimization strategy based on optimizing the cross-attention mechanism, it is possible to reduce the computational complexity while maintaining or improving the accuracy of feature registration, so that the video processing method described in the embodiment of the present application can be better applied to resource-constrained electronic devices, such as terminal devices.

[0048] In a possible implementation, the fourth key matrix may be obtained by performing the eighth linear transformation and sparse selection on the second feature data of the second image. For example, the fourth key matrix may be obtained by performing the sparse selection on the sixth key matrix. The sixth key matrix may be obtained by performing the eighth linear transformation on the second feature data of the second image.

[0049] Since the fourth key matrix is ​​obtained by sparsely selecting the sixth key matrix, the data volume of the fourth key matrix is ​​reduced, thereby reducing the consumption of computing resources, so that the video processing method described in the embodiment of the present application can be better applied to electronic devices with limited resources.

[0050] In one possible implementation, the third attention weight matrix may be obtained based on the fourth attention weight matrix. In one possible implementation, the third attention weight matrix may be obtained by quantizing the twelfth intermediate matrix. Optionally, the third attention weight matrix may be obtained by quantizing the twelfth intermediate matrix to a 4-bit or 8-bit fixed-point integer. The twelfth intermediate matrix may be obtained by pruning the fourth attention weight matrix. Optionally, the twelfth intermediate matrix may be obtained by setting the element values ​​in the fourth attention weight matrix less than or equal to the second predetermined threshold to zero. The second predetermined threshold can be configured according to actual business needs and is not limited here. The fourth attention weight matrix may be obtained by normalizing the tenth intermediate matrix. The tenth intermediate matrix may be obtained by dequantizing the eleventh intermediate matrix. The eleventh intermediate matrix may be obtained based on the sixth query matrix and the sixth key matrix. The sixth query matrix and the sixth key matrix may be obtained by quantizing the fourth query matrix and the fourth key matrix.

[0051] Since the third attention weight matrix is ​​obtained through quantization and cropping, the consumption of computing resources is reduced, so that the video processing method described in the embodiment of the present application can be better applied to electronic devices with limited resources.

[0052] In another possible implementation, the registered second feature data may be obtained by registering the second feature data of the second image and the first feature data of the first image using a cross-attention mechanism, which may include: the registered second feature data may be obtained according to the first query matrix, the first intermediate matrix and the first value matrix. The first intermediate matrix may be used for keys. The first query matrix may be used for queries. The first value matrix may be used for values. The first value matrix may be obtained according to the first intermediate matrix, the first key matrix and the second value matrix. The first intermediate matrix may be used for queries. The first key matrix may be used for keys. The second value matrix may be used for values. For example, the first value matrix may be obtained by processing the first intermediate matrix, the first key matrix and the second value matrix using a cross-attention mechanism. Optionally, the first value matrix may be obtained according to the fifth attention weight matrix and the second value matrix. The fifth attention weight matrix may be obtained according to the first intermediate matrix and the first key matrix. The first query matrix may be obtained by performing a first linear transformation on the first feature data of the first image. For example, the first query matrix may be obtained according to the first feature data of the first image and the first transformation matrix. The element values ​​in the first transformation matrix may be model parameters of a deep learning model. The first key matrix may be obtained by performing a second linear transformation on the second feature data of the second image. For example, the first key matrix may be obtained based on the second feature data of the second image and the second transformation matrix. The element values ​​in the second transformation matrix may be model parameters of the deep learning model. The second value matrix may be obtained by performing a third linear transformation on the second feature data. For example, the second value matrix may be obtained based on the second feature data and the third transformation matrix. The first intermediate matrix may be obtained based on the first query matrix. For example, the first intermediate matrix may be obtained by pooling the first query matrix. Pooling may include maximum pooling, average pooling, or L2 norm pooling, etc. Optionally, the first intermediate matrix may be obtained based on the first query matrix and the fourteenth transformation matrix. The element values ​​in the fourteenth transformation matrix may be model parameters of the deep learning model. Optionally, the first intermediate matrix may be obtained by fusing similar elements in the first query matrix.

[0053] Since the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix, and the first intermediate matrix is ​​used as a query, the first intermediate matrix can be used to aggregate information of the first key matrix and the second value matrix. Since the registered second feature data is obtained according to the first query matrix, the first intermediate matrix and the first value matrix, and the first query matrix is ​​used as a query and the first intermediate matrix is ​​used as a key, it is explained that the first intermediate matrix can pass the aggregated information to the first query matrix.

[0054] Since the first intermediate matrix is ​​obtained according to the first query matrix, the data volume of the first intermediate matrix can be less than the data volume of the first query matrix. Therefore, in the process of determining the first value matrix, the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix, and the first intermediate matrix is ​​used as a query. Therefore, compared with the first query matrix as a query, the data processing volume is reduced, the computing resource consumption is reduced, and the first value matrix aggregates the information of the first key matrix and the second value matrix. In the process of determining the second feature data after registration, the second feature data after registration is obtained according to the first value matrix obtained by the first query matrix, the first intermediate matrix and the first value matrix. The first query matrix is ​​used as a query, the first intermediate matrix is ​​used as a key, and the first value matrix is ​​used as a value. Therefore, the first value matrix can pass the aggregated information to the first query matrix, thereby realizing global modeling. Among them, the first intermediate matrix can be used to aggregate the information of the first key matrix and the second value matrix, and pass the aggregated information to the first query matrix.

[0055] In a possible implementation, the first intermediate matrix may be obtained by fusing similar elements in the first query matrix, which may include: the first intermediate matrix may be obtained based on fused elements and non-fused elements. The fused element may be obtained by fusing a target element and other elements having an association relationship. The other elements having an association relationship with the target element may be elements that are most similar to the target element determined from other sets. The target element may be any one in the target set. When the target set is the first set, the other set is the second set. When the target set is the second set, the other set is the first set. The first set and the second set may be obtained by dividing the elements included in the first query matrix. Non-fused elements may refer to elements that have no association relationship with other elements.

[0056] Since the first intermediate matrix can be obtained by fusing similar elements in the first query matrix, the data volume of the first intermediate matrix is ​​smaller than that of the first query matrix. Therefore, in the process of determining the first value matrix, the first intermediate matrix is ​​used as a query, and in the process of determining the aligned second feature data, the first intermediate matrix is ​​used as a key, which reduces the amount of data processing and reduces the consumption of computing resources.

[0057] In a possible implementation, the registered second feature data may be obtained based on the first query matrix, the first intermediate matrix and the first value matrix, and may include: the registered second feature data may be obtained based on the second intermediate matrix and the third value matrix. For example, the registered second feature data may be obtained by adding the second intermediate matrix and the third value matrix. The second intermediate matrix may be obtained based on the first query matrix, the first intermediate matrix and the first value matrix. For example, the third intermediate matrix may be obtained based on the sixth attention weight matrix and the first value matrix. The sixth attention weight matrix may be obtained based on the first query matrix and the first intermediate matrix. The third value matrix may be obtained by performing a first depth-separable convolution on the first value matrix.

[0058] Since the third value matrix is ​​obtained by performing a first depth-wise separable convolution on the first value matrix, feature diversity is improved.

[0059] In another possible implementation, the registered second feature data may be obtained according to the first query matrix, the first intermediate matrix and the first value matrix, which may include: the registered second feature data may be obtained according to the sixth attention weight matrix and the first value matrix. The sixth attention weight matrix may be obtained according to the first query matrix and the first intermediate matrix.

[0060] In a possible implementation, the registered second feature data may be obtained by registering the second feature data of the second image and the first feature data of the first image using a cross-attention mechanism, which may include: the registered second feature data may be obtained according to the third intermediate matrix and the second query matrix. For example, the registered second feature data may be obtained by multiplying the third intermediate matrix and the second query matrix. The third intermediate matrix may be obtained according to the second key matrix and the fourth value matrix. The second query matrix and the second key matrix may be obtained by processing the third query matrix and the third key matrix respectively using a focusing function. The focusing function may be used to adjust the proximity between the third query matrix and the third key matrix, that is, the aggregation function may be used to make similar elements in the third query matrix and the third key matrix closer and dissimilar elements farther away. For example, the third query matrix may be input into the aggregation function to obtain the second query matrix. The third key matrix may be input into the aggregation function to obtain the second key matrix. The third query matrix may be obtained by performing a fourth linear transformation on the first feature data of the first image. For example, the third query matrix may be obtained according to the first feature data of the first image and the fourth transformation matrix. The third key matrix may be obtained by performing a fifth linear transformation on the second feature data of the second image. For example, the third key matrix may be obtained according to the second feature data of the second image and the fifth transformation matrix. The third intermediate matrix may be obtained based on the second key matrix and the fourth value matrix. For example, the third intermediate matrix may be obtained by multiplying the second key matrix and the fourth value matrix. The fourth value matrix may be obtained by performing a sixth linear transformation on the second feature data of the second image. For example, the fourth matrix may be obtained based on the second feature data of the second image and the sixth transformation matrix.

[0061] Since the aggregation function can be used to adjust the proximity between the third query matrix and the third key matrix, that is, the aggregation function can be used to make similar elements in the third query matrix and the third key matrix closer and dissimilar elements farther away, it is achieved that the second feature with a large similarity between the aligned second feature data and the first feature data is given a larger weight, and the second feature with a small similarity between the second feature data and the first feature data is given a smaller weight, thereby improving the accuracy of feature alignment and further improving the video noise reduction effect. In addition, since the third intermediate matrix is ​​obtained based on the second key matrix and the fourth value matrix, the aligned second feature data is obtained based on the second query matrix and the third intermediate matrix, that is, the key and value are calculated first, and the result is then calculated with the query, thereby reducing the amount of data processing and reducing the consumption of computing resources.

[0062] In one possible implementation, the first feature data may be determined based on the first visual feature data and the first semantic feature data. Optionally, the first feature data may be determined based on the first visual feature data, the first semantic feature data and a first position code. The first position code may be used to indicate a pixel position of a pixel in the first image.

[0063] The second feature data may be determined based on the second visual feature data and the second semantic feature data. Optionally, the second feature data may be determined based on the second visual feature data, the second semantic feature data, and the second position code. The second position code may be used to indicate the pixel position of a pixel in the second image.

[0064] Since position coding can facilitate better capturing of dependencies and patterns between features in feature data and can focus on the relative positions of features, the reliability of the first feature data and the second feature data is improved, thereby improving the video noise reduction effect.

[0065] In a possible implementation, the first query matrix may be obtained by performing a second depth classifiable convolution on the first feature data of the first image. And / or, the first key matrix may be obtained by performing a third depth separable convolution on the second feature data of the second image. And / or, the first value matrix may be obtained by performing a fourth depth separable convolution on the second feature data.

[0066] Since the first query matrix, the first key matrix, and the first value matrix can be obtained by using a deep classifiable convolution method, the deep separable convolution can reduce the amount of data processing, thereby reducing the amount of data processing and reducing computing resource consumption.

[0067] In a possible implementation, fusing the first feature data of the first image and the registered second feature data to obtain a video frame may include: decoding the fused feature data to obtain the video frame. The fused feature data may be obtained by fusing the first feature of the first image and the registered second feature data.

[0068] Since the fused feature data is obtained by fusing the first feature data and the registered second feature data, the fused feature data can reflect the fusion characteristics of the first feature data and the registered second feature data, thereby making the information carried by the fused feature more comprehensive and accurate. On this basis, the fused feature data is decoded to obtain a video frame, thereby improving the quality of the video frame, thereby improving the video noise reduction effect.

[0069] In a possible implementation, the fused feature data may be obtained by fusing the first feature of the first image and the registered second feature data, which may include: the fused feature data may be obtained by splicing the first feature data and the registered second feature data. Optionally, the fused feature data may be obtained by adding the first feature data and the registered second feature data. Optionally, the fused feature data may be obtained by multiplying the first feature data and the registered second feature data.

[0070] In another possible implementation, the fused feature data may be obtained by fusing the first feature of the first image and the registered second feature data, which may include: the fused feature data may be obtained based on the first feature data and the third feature data. For example, the fused feature data may be obtained by fusing the first feature data and the third feature data. The third feature data may be obtained based on the fourth feature data and the registered second feature data. For example, the third feature data may be obtained by multiplying the fourth feature data and the registered second feature data. The fourth feature data may be obtained based on the first feature data and the registered second feature data. For example, the fourth feature data may be obtained by convolving the first feature data and the registered second feature data.

[0071] Since the fused feature data is obtained based on the first feature data and the third feature data, the third feature data is obtained based on the fourth feature data and the registered second feature data, and the fourth feature data is obtained based on the first feature data and the registered second feature data, a relatively complete fusion of the first feature data and the second feature data is achieved, thereby making the information carried by the fused feature data more comprehensive and accurate.

[0072] In a possible implementation, decoding the fused feature data to obtain a video frame may include: the video frame may be obtained based on the first feature data and the fifth feature data. The fifth feature data may be obtained based on the sixth feature data. The sixth feature data may be obtained based on the seventh feature data and the fused feature data. The seventh feature data may be obtained based on the eighth feature data. The eighth feature data may be obtained based on the fused feature data and the ninth feature data. The ninth feature data may be obtained based on the fused feature data.

[0073] Since the video frame is obtained based on the first feature data and the fifth feature data, the fifth feature data is obtained based on the sixth feature data, the sixth feature data is obtained based on the seventh feature data and the fused feature data, the seventh feature data is obtained based on the eighth feature data, the eighth feature data is obtained based on the fused feature data and the ninth feature data, and the ninth feature data can be obtained based on the fused feature data, therefore, more full utilization of the feature data is achieved, thereby improving the quality of the video frame.

[0074] In a possible implementation manner, the image brightness of the first image and the second image are aligned, and / or the image brightness of the first image and the second image are aligned and the field of view angles are aligned.

[0075] By aligning the image brightness of the first image and the second image, the brightness range of the first image and the second image is made the same, thereby improving the accuracy of image registration and image fusion, thereby improving the video noise reduction effect. By aligning the field of view of the first image and the second image, the accuracy of image registration and image fusion is improved, thereby improving the video noise reduction effect.

[0076] In one possible implementation, the image brightness of the second image may be determined based on the original image brightness of the second image, the amount of light entering the second image, and the amount of light entering the first image. The amount of light entering the first image may be determined based on the exposure time and sensitivity of the first image. The amount of light entering the second image may be determined based on the exposure time and sensitivity of the second image. For example, the image brightness of the second image may be determined based on a ratio and the original image brightness of the second image. The ratio may be the ratio between the amount of light entering the second image and the amount of light entering the first image.

[0077] By adjusting the image brightness, the image brightness ranges of multiple images are made the same, that is, the image brightness of multiple images are aligned, thereby improving the accuracy of image registration and image fusion, and further improving the video noise reduction effect.

[0078] In a possible implementation, the target image may be obtained by image cropping. The target image may be an image captured by a target camera among the first image and the second image. The target camera may be a camera with a large field of view among multiple cameras. Optionally, the target camera may be a camera with a long exposure time among multiple cameras. Optionally, the target camera may be a camera with a large field of view and a long exposure time among multiple cameras.

[0079] By using image cropping to achieve the field of view alignment of multiple images, the accuracy of image registration and image fusion is improved, thereby improving the video noise reduction effect. In addition, the amount of data processing is reduced and the consumption of computing resources is reduced.

[0080] In a possible implementation manner, the first image and the second image may be RAW images.

[0081] Since the RAW image is not processed by the image signal processor, the ISP processing will make the noise complicated. Therefore, the first image and the second image are RAW images, which reduces the difficulty of video noise reduction and improves the video noise reduction effect.

[0082] In a second aspect, an embodiment of the present application provides a video processing device, which may include: a recording module, for performing video recording in response to a first operation. A processing module, for reducing the noise of a first image according to a second image to obtain a video frame during video recording. The video frame may be a frame in the recorded video. The first image and the second image may be taken using multiple cameras. The shooting time of the first image and the shooting time of the second image may overlap. The lens direction of the first camera may be the same as the lens direction of the second camera. The first camera may be a camera that shoots the first image. The second camera may be a camera that shoots the second image. The exposure time of the first image may be less than the exposure time of the second image.

[0083] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store computer-executable instructions, and the processor is used to run the computer-executable instructions stored in the memory, so that the electronic device executes the method described in any possible implementation of the first aspect.

[0084] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed on an electronic device, the electronic device executes the method described in any possible implementation of the first aspect.

[0085] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program runs on an electronic device, the electronic device executes the method described in any possible implementation manner of the first aspect.

[0086] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0087] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0088] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic diagram of an image output mode provided in an embodiment of the present application;

[0090] Figure 2 A schematic diagram of video recording according to an embodiment of the present application;

[0091] Figure 3 A schematic diagram of the principle of the video processing method provided in the embodiment of the present application;

[0092] Figure 4 A schematic diagram of the principle of implementing video noise reduction by using image registration and image fusion proposed in an embodiment of the present application;

[0093] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0094] Figure 6 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0095] Figure 7 A flowchart of a video processing method proposed in an embodiment of the present application;

[0096] Fig. 8A A schematic diagram of an interface of a dark light recording mode provided in an embodiment of the present application;

[0097] Figure 8B A schematic diagram of another interface of a dark light recording mode provided in an embodiment of the present application;

[0098] Fig. 9A A schematic diagram of starting a dark light recording mode provided in an embodiment of the present application;

[0099] Fig. 9B A schematic diagram of another method of starting a dark light recording mode provided in an embodiment of the present application;

[0100] Fig. 9C A schematic diagram of another method of starting a dark light recording mode provided in an embodiment of the present application;

[0101] Fig.9D A schematic diagram of a dark light recording mode provided in an embodiment of the present application;

[0102] Fig.10 A schematic diagram of a principle for obtaining registered second feature data according to an embodiment of the present application;

[0103] Fig.11 Another schematic diagram of the principle of obtaining the registered second feature data provided in the embodiment of the present application;

[0104] Fig.12 A schematic diagram of the principle of obtaining fusion feature data provided in an embodiment of the present application;

[0105] Fig.13 A flowchart of another video processing method provided in an embodiment of the present application;

[0106] Fig.14 This is a structural block diagram of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0107] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0108] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0109] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0110] In order to clearly describe the technical solutions of the embodiments of the present application, some terms or concepts involved in the embodiments of the present application are explained below.

[0111] Image clarity can be a measure of the quality of an image or video frame, which reflects the details and sharpness of the visual content.

[0112] Image brightness can refer to the light brightness or brightness of an image or video frame, which can affect the visual consistency and clarity of the image or video frame.

[0113] Video may refer to a data stream that encodes video frames in a time sequence. A video may include multiple video frames. Video quality may be evaluated by at least one of image clarity, image brightness, or smoothness.

[0114] Frame rate (i.e., Frame Rate) can refer to the number of frames recorded or played per second. Frame rate can be expressed in FPS (Frames PerSecond). Frame interval (or frame duration) can refer to the time interval between two consecutive frames in a video, that is, the duration of each frame. Frame interval is the reciprocal of frame rate. For example, for a frame rate of 30FPS, the frame interval is 1 / 30 second.

[0115] The frame rate may refer to the shooting frame rate or the playback frame rate, which may be determined based on the actual business. It should be noted that in the embodiment of the present application, the frame rate may refer to the shooting frame rate. More specifically, the shooting frame rate may refer to the number of frames recorded per second by the camera during shooting. The camera may support at least one shooting frame rate, for example, 30FPS, 90FPS, 120FPS, 480FPS. The shooting frame rates supported by different cameras may be the same or different.

[0116] Noise can refer to factors that affect the understanding of source information. Source information can include images or videos. Noise not only affects the quality of source information, but also may interfere with subsequent tasks based on source information. For example, subsequent tasks may include at least one of the following: image classification, object detection, image recognition, image segmentation, pedestrian re-identification, multimodal matching, image retrieval or image description, etc.

[0117] The signal-to-noise ratio (SNR) can be used to evaluate the quality of source information. The signal-to-noise ratio can be the ratio between useful information and noise in the source information. A higher signal-to-noise ratio can mean that the proportion of useful information in the source information is higher and the proportion of noise is lower, thereby providing clearer and higher quality source information.

[0118] Exposure refers to the process by which the camera's photosensitive element receives light. Exposure determines the image brightness (i.e., light and dark). During the exposure process, the exposure parameters that affect the image brightness may include exposure time (ET), sensitivity, and aperture value (AV).

[0119] The shutter can be used to control the length of time light enters the camera to determine the exposure time of the image. The longer the shutter is open, the more light enters the camera and the longer the image exposure time. The shorter the shutter is open, the less light enters the camera and the shorter the image exposure time.

[0120] The exposure duration (or exposure time, shutter speed) may refer to the duration from when the shutter is opened to when it is closed. The exposure duration may be less than or equal to the frame time interval. According to the exposure duration, the exposure may be divided into long exposure, short exposure and medium exposure, thereby obtaining a long frame image, a short frame image and a medium frame image. The long exposure may have a longer exposure duration. The short exposure may have a shorter exposure duration. The exposure duration of the medium exposure is greater than the exposure duration of the short exposure and less than the exposure duration of the long exposure. The long frame image may refer to an image obtained under long exposure. The short frame image may refer to an image obtained under short exposure. The medium frame image may refer to an image obtained under medium exposure.

[0121] Sensitivity refers to the sensitivity of the camera's photosensitive element to light, which can be represented by ISO (International Organization for Standardization).

[0122] The aperture can be an adjustable opening in the camera's lens, the size of which determines the amount of light that enters. The amount of light that enters can refer to the amount of light that enters the camera's photosensitive element. The aperture value is the ratio of the camera's lens focal length to the aperture diameter. The larger the aperture value, the more light that enters. The smaller the aperture value, the less light that enters. The amount of light that enters can be determined based on the exposure time and sensitivity of the image.

[0123] A dark light scene may refer to a scene with low ambient light intensity (or ambient illumination) and / or low ambient light brightness. As an implementation method, a dark light scene may refer to a scene with an ambient light intensity less than or equal to a predetermined ambient light intensity. As another implementation method, a dark light scene may refer to a scene with an ambient light brightness less than or equal to a predetermined ambient light brightness. The predetermined ambient light intensity and the predetermined ambient light brightness may be configured according to actual business needs and are not limited here. For example, the predetermined ambient light intensity may be less than or equal to 5 lux. For example, dark light scenes may include outdoor low-light scenes, indoor low-light scenes, and the like. Outdoor low-light scenes may include night scenes, cloudy scenes, haze scenes, tunnel scenes, and the like. Ambient light intensity may refer to the luminous flux received per unit area, which describes the distribution of light in space. Ambient light brightness may refer to the intensity of light emitted by a light source or the surface of a subject.

[0124] Field of View (FOV) refers to the range of view (or content of view) that a camera can cover. The larger the camera's FOV, the larger the range of view it can cover. The smaller the camera's FOV, the smaller the range of view it can cover.

[0125] Image registration can be the process of transforming different images of the same scene into the same coordinate system. Image registration is a critical task used to align images acquired at different times, viewing angles, or sensors for subsequent analysis and comparison. Image registration can be used to make the pixels of objects in multiple images consistent.

[0126] Image registration can refer to the process of matching and aligning multiple images of the same scene obtained under different conditions (e.g., different times, different perspectives, different cameras). For example, multiple images can be taken at different perspectives at the same time, at the same perspective at different times, at different perspectives at different times, etc.

[0127] Video noise reduction (or video denoising) is used to suppress the noise in the video while maintaining the texture details of the video as much as possible and restoring the real scene to improve the reliability and availability of the video. The demand for video noise reduction is extensive and involves many fields, such as security, autonomous driving, audio and video entertainment, etc.

[0128] The Cross-Attention mechanism enables interaction and matching of different images at the feature level, thereby capturing the similarities and differences between features, focusing on similarities with high weights and suppressing differences with low weights, and can adaptively adjust weights. Therefore, it has higher robustness and scalability.

[0129] The RAW domain (or RAW format) is an unprocessed format. RAW image can refer to an unprocessed image.

[0130] Image output modes may include a high definition mode, a low signal-to-noise ratio mode, and a low definition, high signal-to-noise ratio mode.

[0131] For example, the high-definition, low signal-to-noise ratio mode may include a pixel-in-one mode. Optionally, the pixel-in-one mode may include a Quadr mode. The Quadr mode may add the charges induced by adjacent pixels together and read them out in a one-pixel mode in low light conditions. The Quadr mode may read out in a single-pixel mode in high light conditions to improve image resolution.

[0132] For example, a low-definition, high signal-to-noise ratio image output mode may include a pixel merging mode, etc. Optionally, the pixel merging mode may include a Binning mode. The Binning mode may add together the charges induced by adjacent pixels and read them out in a one-pixel mode. The Binning mode may reduce the image resolution, increase the photosensitive area, and improve the signal-to-noise ratio while maintaining the field of view angle. The Binning mode may include a horizontal Binning mode or a vertical Binning mode. Horizontal Binning is to add together the charges induced by pixels in adjacent rows and read them out. Vertical Binning is to add together the charges induced by pixels in adjacent columns and read them out.

[0133] Combine the following Figure 1 A pixel-merging mode (eg, Quadr mode) and a pixel-merging mode (eg, Binning mode) are described.

[0134] Figure 1 A schematic diagram of an image output mode provided in an embodiment of the present application. Figure 1 In the figure, "R" indicates a red pixel, "G" indicates a green pixel, and "B" indicates a blue pixel.

[0135] like Figure 1 As shown, the pixel unit may include a red sub-pixel unit, two green sub-pixel units and a blue sub-pixel unit. The red sub-pixel unit may include four red pixels. The green sub-pixel unit may include four green pixels. The blue sub-pixel unit may include four blue pixels.

[0136] It should be noted that the Quadr mode in the embodiment of the present application can be read out in a single pixel mode under weak light conditions, for example, Figure 1 The Binning mode in the embodiment of the present application can adopt, for example, Figure 1 Pixel binning mode in . And, Figure 1 It is merely an illustrative illustration and does not constitute a limitation on the embodiments of the present application.

[0137] The inventive concept of the embodiments of the present application is described below with reference to the accompanying drawings.

[0138] For scenes where the video quality is greatly affected by noise, it is necessary to reduce the noise of the video in order to improve the video quality. For example, scenes where the video quality is greatly affected by noise may be dark scenes.

[0139] In related technologies, video noise reduction can be achieved by extending the exposure time. This is because extending the exposure time can increase the amount of light entering, and the increase in the amount of light entering can reduce noise and improve the signal-to-noise ratio. For example, during video recording, a single camera is used to shoot with a longer exposure time to obtain a video frame with noise reduction.

[0140] It is found that to achieve a better video noise reduction effect, the exposure time needs to be greater than or equal to 100ms. However, in the case of video recording, the exposure time is limited by the frame rate, that is, the exposure time needs to be less than or equal to the frame time interval. For example, the frame time interval is 30ms. Therefore, the extended exposure time is difficult to meet the requirements. Therefore, when the frame rate needs to be balanced, the above video noise reduction method is difficult to achieve a good noise reduction effect. For this reason, it is necessary to improve the video noise reduction effect.

[0141] In order to improve the video noise reduction effect, it is found that the long-frame image has a high signal-to-noise ratio, low definition, richer dark light area information, and richer color information. The short-frame image can balance the frame rate, maintain the texture edge and highlight area information of the scene, and has high definition and low signal-to-noise ratio. Therefore, it is shown that the long-frame image and the short-frame image can complement each other to a certain extent. For example, the high-noise sharp texture of the short-frame image and the low-noise fuzzy texture of the long-frame image can constrain each other, which is conducive to detail recovery. The highlight area information of the short-frame image can repair the overexposed cutoff area of ​​the long-frame image. In addition, the color information of the long-frame image can correctly restore the color of the scene. Therefore, the embodiment of the present application takes into account the advantages of short-frame images in balancing frame rate and / or high definition, and the advantages of long-frame images in improving signal-to-noise ratio, and proposes that long-frame images can be used as a supplement to short-frame images to improve the quality of short-frame images, so as to improve the video noise reduction effect while balancing the frame rate. Among them, the short-frame image can be used as a reference image. The long-frame image can be used as an auxiliary image.

[0142] Next, we need to consider how to obtain long-frame images and short-frame images, and we find that there are single-camera and multi-camera methods.

[0143] Based on the single camera method, that is, using a single camera to shoot to obtain long-frame images and short-frame images. However, as mentioned above, the exposure time of the long-frame images obtained by this method is difficult to break through the frame rate limit. Therefore, it is difficult to obtain long-frame images that meet the requirements of video noise reduction using this method.

[0144] Based on the multi-camera method, that is, using different cameras to shoot long-frame images and short-frame images. Since the long-frame images and short-frame images are shot by different cameras, when the exposure time of the short-frame images can match the frame rate requirements, the exposure time of the long-frame images can break through the frame rate limit. Therefore, this method can be used to obtain long-frame images that meet the video noise reduction requirements.

[0145] Next, we need to consider how to match the needs of multiple cameras. It is found that with the continuous development of electronic devices, the functions of electronic devices have been enriched and improved. For example, multiple cameras can provide different shooting functions. The field of view of different cameras can be different. As a result, electronic devices can be configured with multiple cameras to provide multi-camera functions. Multiple cameras can include at least two of the following: main camera, telephoto camera, wide-angle camera, ultra-wide-angle camera, periscope telephoto camera, macro camera, fisheye camera, infrared camera, depth camera, black and white camera, color camera, etc.

[0146] Based on this, the embodiment of the present application proposes that the multi-camera function provided by the electronic device can be used to obtain images with different exposure times to achieve video noise reduction. For example, in the case of video recording, multiple images of the subject are obtained by using multiple cameras to obtain video frames in the video. In addition to the cameras that use a shorter exposure time to balance the frame rate, there are also cameras that use a longer exposure time to improve the signal-to-noise ratio. Thus, an image with a shorter exposure time (for example, a short frame image) among the multiple images can be used as a reference image, and an image with a longer exposure time (for example, a long frame image) among the multiple images can be used as an auxiliary image, and the auxiliary image is used to reduce the noise of the reference image to obtain a noise-reduced reference image. The noise-reduced reference image is a video frame in the video.

[0147] In the case of video recording, the reference image is obtained by photographing the subject with a camera that uses a shorter exposure time to balance the frame rate, and the auxiliary image is obtained by photographing the subject with a camera that uses a longer exposure time to improve the signal-to-noise ratio. Therefore, the auxiliary image is used to reduce the noise of the reference image to obtain the video frame in the video, so that the video noise reduction effect can be improved while balancing the frame rate.

[0148] It should be noted that the multiple images may include at least one auxiliary image. The exposure durations of different auxiliary images may be different. Different auxiliary images may be captured by different cameras. The exposure duration of the auxiliary image is greater than the exposure duration of the reference image. For example, a mid-frame image may be used as an auxiliary image. Thus, the denoised reference image may be obtained by denoising the reference image using at least one auxiliary image.

[0149] It should also be noted that the image output modes of different cameras can be the same or different, and are not limited here. For example, the first camera can adopt an image output mode with high definition and low signal-to-noise ratio. The second camera can adopt an image output mode with low definition and high signal-to-noise ratio. Optionally, both the first camera and the second camera can adopt an image output mode with high definition and low signal-to-noise ratio. Optionally, both the first camera and the second camera can adopt an image output mode with low definition and high signal-to-noise ratio. The first camera may refer to a camera that uses a shorter exposure time to balance the frame rate. The second camera may refer to a camera that uses a longer exposure time to improve the signal-to-noise ratio.

[0150] As an implementation method, the exposure time of the reference image can be set to be less than or equal to the frame interval of the reference image to balance the frame rate. The exposure time of the auxiliary image is set to be greater than the frame interval of the reference image to improve the signal-to-noise ratio. For example, the frame interval of the reference image can be 30ms. The exposure time of the reference image can be less than or equal to 30ms. Optionally, the exposure time of the reference image can be 20ms. The exposure time of the auxiliary image can be greater than or equal to 100ms. Optionally, the exposure time of the auxiliary image can be 150ms.

[0151] As an implementation method, the shooting time of the reference image and the shooting time of the auxiliary image may overlap, and the lens direction of the first camera and the lens direction of the second camera may be the same, so that the content of the reference image and the auxiliary image are similar. It should be noted that in the embodiment of the present application, the lens direction of the first camera and the lens of the second camera are the same but not absolutely the same, and the two lens directions may be roughly the same. The two lens directions can make the content of the reference image and the auxiliary image similar. For example, the first camera and the second camera are both front cameras. Optionally, the first camera and the second camera are both rear cameras.

[0152] As an implementation, the reference image may be obtained by photographing the subject by the first camera at intervals of a first predetermined time. The auxiliary image may be obtained by photographing the subject by the second camera at intervals of a second predetermined time. The first predetermined time is greater than or equal to the exposure time of the reference image and less than the second predetermined time. The second predetermined time is greater than or equal to the exposure time of the auxiliary image. For example, the first predetermined time may be 30 ms. The second predetermined time may be 150 ms.

[0153] In order to better understand the video recording process of the embodiment of the present application, the following takes multiple cameras including a first camera and a second camera as an example. Figure 2 Provide explanation.

[0154] Figure 2 A schematic diagram of video recording provided according to an embodiment of the present application.

[0155] like Figure 2 As shown, the first camera can shoot reference image 1, reference image 2, reference image 3, reference image 4 and reference image 5 at intervals of a first predetermined time. The second camera can shoot auxiliary image 1 at intervals of a second predetermined time. Each time an auxiliary image is shot, five reference images can be obtained.

[0156] The exposure duration of each reference image is the first exposure duration. The exposure duration of the auxiliary image is the second exposure duration. The first exposure duration is less than the first predetermined duration. The second exposure duration is equal to the second predetermined duration. The first exposure duration is less than the second exposure duration.

[0157] It should be noted that Figure 2 It is merely an illustrative illustration and does not constitute a limitation on the embodiments of the present application.

[0158] In order to better understand the inventive concept of the embodiment of the present application, Figure 3 Provide further explanation.

[0159] Figure 3 The schematic diagram of the principle of the video processing method provided in the embodiment of the present application is shown in FIG. The method can be applied to electronic devices.

[0160] like Figure 3 As shown, the electronic device may include N cameras, namely a first camera, ..., an nth camera, ..., an Nth camera. N may be an integer greater than 1. n may be an integer greater than 1 and less than or equal to N. The lens directions of the N cameras may be the same. The exposure duration corresponding to the nth camera may be the nth exposure duration. The nth exposure duration is greater than the n-1th exposure duration. The first camera may be a camera that uses a smaller exposure duration to balance the frame rate. The nth to Nth cameras may be cameras that use a larger exposure duration to improve the signal-to-noise ratio. The image captured by the first camera may be used as a reference image. The images captured by the nth to Nth cameras may be used as auxiliary images.

[0161] In the case of video recording, multiple cameras can capture the subject according to their respective exposure times to obtain at least one image corresponding to each of the multiple cameras.

[0162] Specifically, the first camera may shoot the object according to the first exposure time to obtain first images 1, ..., first images i, .... i may be an integer greater than or equal to 1.

[0163] The nth camera may shoot the object according to the nth exposure time to obtain the nth image 1, ..., the nth image j, .... j may be an integer greater than or equal to 1 and less than i.

[0164] The Nth camera may shoot the object according to the Nth exposure time to obtain the Nth image 1, ..., the Nth image k, .... k may be an integer greater than or equal to 1 and less than or equal to j.

[0165] The following takes the first image 1, ..., the nth image 1, ..., the Nth image 1 as an example to illustrate how to implement video noise reduction. The first image 1 is used as a reference image. The nth image 1 to the Nth image 1 are used as auxiliary images. The shooting time of the first image 1 to the Nth image 1 overlaps.

[0166] In the case of 2≤n≤N, the first image 1 (i.e., the reference image) is denoised using the nth image 1 (i.e., the auxiliary image) to obtain intermediate data. The denoised first image 1 is obtained based on the N-1 intermediate data, thereby achieving denoising of the first image 1 using the nth image 1 to the Nth image 1 to obtain the denoised first image 1. The denoised first image 1 is a video frame in the video.

[0167] As an implementation manner, the first camera may be a telephoto camera, and the second camera may be a main camera.

[0168] In addition, in order to further improve the video noise reduction effect, the embodiment of the present application proposes that the clarity of the reference image can be made higher than the clarity of the auxiliary image.

[0169] In addition, in order to further improve the video noise reduction effect, the embodiment of the present application proposes that it can be implemented in combination with the image output mode. The first camera can adopt a high-definition, low signal-to-noise ratio image output mode. The second camera can adopt a low-definition, high signal-to-noise ratio image output mode, thereby making the reference image have high definition and low signal-to-noise ratio, and the auxiliary image has low definition and high signal-to-noise ratio, and the definition of the reference image is higher than the definition of the auxiliary image.

[0170] As an implementation, a high-definition, low signal-to-noise ratio image output mode may include a pixel-in-one mode. For example, the pixel-in-one mode may include a Quadr mode. A low-definition, high signal-to-noise ratio image output mode may include a pixel-in-one mode, etc. For example, the pixel-in-one mode may include a Binning mode.

[0171] In addition, regarding how to use the auxiliary image to reduce the noise of the reference image to obtain a video frame, the embodiment of the present application proposes that image registration and image fusion can be used to achieve this. Figure 4 Provide explanation.

[0172] Figure 4 A schematic diagram of the principle of implementing video noise reduction by using image registration and image fusion proposed in an embodiment of the present application.

[0173] As a way to achieve this, Figure 4 The implicit denoising method shown in FIG. 1 can refer to realizing feature registration based on feature data extracted from the image, fusing the registered feature data, and finally obtaining a denoised video frame based on the fused feature data. Therefore, the implicit denoising method can be understood as feature-level denoising.

[0174] For example, a video frame may be obtained based on fused feature data. The fused feature data may be obtained by fusing the reference feature data of a reference image and the registered auxiliary feature data. The registered auxiliary feature data may be obtained by registering the auxiliary feature data of an auxiliary image and the reference feature data. That is, the auxiliary feature data of the auxiliary image and the reference feature data of the reference image may be registered to obtain the registered auxiliary feature data. Next, the reference feature data and the registered auxiliary feature data are fused to obtain the fused feature data. Finally, a video frame is obtained based on the fused feature data.

[0175] It should be noted that the feature data may include at least one of the following: semantic feature data or visual feature data. Semantic feature data may be used to indicate the semantic information expressed by the image. Semantic feature data may include at least one of the following: local semantic feature data or global semantic feature data. Local semantic feature data may be used to indicate the local association between images. The receptive field of global semantic feature data is larger than that of local semantic feature data. Visual feature data may include at least one of the following: shallow visual feature data or deep visual feature data. Shallow visual feature data may be used to indicate the fine-grained visual features of the image. Fine-grained visual features may include at least one of the following: color features, texture features, edge features or angular features, etc. Deep visual feature data may be used to indicate the coarse-grained visual features of the image. Coarse-grained visual features may refer to abstract visual features. Abstract visual features may refer to visual features that can express semantic information.

[0176] Since the feature data abstracts the image information while retaining the detail information, it has higher robustness and discriminability, thereby improving the reliability and robustness of video denoising and improving the video denoising effect.

[0177] As another implementation, Figure 4 The explicit denoising method shown in FIG. Explicit denoising method may refer to registering multiple images, and obtaining a denoised video frame based on the registered images. Therefore, the explicit denoising method may be understood as image-level denoising.

[0178] For example, the video frame may be obtained by fusing the reference image and the registered auxiliary image. The registered auxiliary image may be obtained by registering the auxiliary image and the reference image. That is, the auxiliary image and the reference image may be registered to obtain the registered auxiliary image. Then, the reference image and the registered auxiliary image are fused to obtain the video frame.

[0179] Since image-level denoising can make full use of pixel information, retain image detail information as much as possible, and has good interpretability, the video denoising effect is improved.

[0180] Further, in order to improve the accuracy of feature registration based on implicit noise reduction, it is found that the association between auxiliary feature data and reference feature data can be mined so that the registered auxiliary feature data is as similar as possible to the reference feature data. Therefore, the embodiment of the present application proposes that the feature weight can be adjusted based on similarity to achieve this. The auxiliary feature data may include multiple auxiliary features. The reference feature data may include multiple reference features. The registered auxiliary feature data may include multiple registered auxiliary features.

[0181] For example, Figure 4 As shown, the registered auxiliary feature data can be obtained based on the similarity data and the auxiliary feature data. The similarity data may include the similarity corresponding to each of the multiple auxiliary features. The similarity corresponding to the auxiliary feature can be used to indicate the degree of similarity between the auxiliary feature and the reference feature data. For example, the greater the similarity corresponding to the auxiliary feature, the greater the similarity between the auxiliary feature and the reference feature data. The smaller the similarity corresponding to the auxiliary feature, the smaller the similarity between the auxiliary feature and the reference feature data. Thus, according to the similarity data, each auxiliary feature in the auxiliary feature data can be given a weight to reflect the importance of the auxiliary feature in the feature data registration. For example, a larger weight is given to the auxiliary feature with a large similarity between the auxiliary feature data and the reference feature data, and a smaller weight is given to the auxiliary feature with a small similarity between the auxiliary feature data and the reference feature data. Furthermore, according to the auxiliary feature and the weight of the auxiliary feature, the registered auxiliary feature and the registered auxiliary feature are obtained.

[0182] In addition, it is found that the cross-attention mechanism can realize the interaction and matching of different images at the feature level, thereby capturing the similarities and differences between features, focusing on similarities with high weights, suppressing differences with low weights, and adaptively adjusting weights, and therefore, having higher robustness and scalability. Therefore, an embodiment of the present application proposes that feature weights can be adjusted in a manner based on a cross-attention mechanism. For example, the registered auxiliary feature data can be obtained by registering the auxiliary feature data and reference feature data using a cross-attention mechanism. Reference feature data is used for query. Auxiliary feature data is used for keys and values.

[0183] The cross-attention mechanism may include at least one of the following: a single-head cross-attention mechanism or a multi-head cross-attention mechanism. The similarity determination method involved in the cross-attention mechanism may include at least one of the following: dot product similarity, scaled dot product similarity, additive similarity or cosine similarity, etc. The cross-attention mechanism based on dot product similarity may be called a dot product attention mechanism. The cross-attention mechanism based on scaled dot product similarity may be called a scaled dot product attention mechanism. The cross-attention mechanism based on additive similarity may be called an additive attention mechanism. The cross-attention mechanism based on cosine similarity may be called a cosine attention mechanism.

[0184] In addition, it is found that since the video processing method described in the embodiment of the present application can be applied to electronic devices with limited resources, such as terminal devices, the computational complexity of the cross-attention mechanism is proportional to the square of the number of feature data included in the feature data, and therefore, it is expected to reduce the computational complexity while maintaining or improving the accuracy of feature registration. To this end, the embodiment of the present application proposes that the cross-attention mechanism can be optimized based on an optimization strategy to achieve this.

[0185] As an implementation, the optimization strategy may include at least one of the following: a linear strategy, a sparse strategy, a quantization strategy, or a pruning strategy, etc. A cross-attention mechanism based on a linear strategy may be referred to as a linear attention mechanism. A cross-attention mechanism based on a sparse strategy may be referred to as a sparse attention mechanism.

[0186] In addition, it is found that the field of view angles, displacements, etc. of multiple images are different, which increases the difficulty of registration, reduces the accuracy of registration, and further affects the video noise reduction effect. To this end, the embodiment of the present application proposes that a method based on a deep learning model can be used to achieve video noise reduction.

[0187] For example, the auxiliary image and the reference image may be input into a video denoising model to obtain a video frame. The video denoising model may be obtained by training a deep learning model using a first sample image, a second sample image, and a third sample image. The third sample image may be a denoised image corresponding to the first sample image. Specifically, the video denoising model may be obtained by adjusting model parameters of the deep learning model according to a loss function value. The loss function value may be obtained by inputting a sample denoised image and a third sample image into a loss function. The sample denoised image may be obtained by inputting a first sample image and a second sample image into a deep learning model.

[0188] The exposure time of the first sample image may be shorter than the exposure time of the second sample image. The clarity of the first sample image may be higher than the clarity of the second sample image. The first sample image, the second sample image, and the third sample image may be taken using multiple cameras. The shooting time of the first sample image and the second sample image may overlap. The lens orientation of the camera used to shoot the first sample image, the lens orientation of the camera used to shoot the second sample image, and the lens orientation of the camera used to shoot the second sample image may be the same.

[0189] As an implementation, the exposure time of the second sample image may be greater than the frame interval of the first sample image. The exposure time of the first sample image may be less than or equal to the frame interval of the first sample image. For example, the frame interval of the first sample image may be 30 ms. The exposure time of the first sample image may be less than or equal to 30 ms. Optionally, the exposure time of the first sample image may be 20 ms. The exposure time of the second sample image may be greater than or equal to 100 ms. Optionally, the exposure time of the second sample image may be 150 ms.

[0190] It should be noted that the model structure of the deep learning model can be configured according to actual business needs as long as it can improve the video noise reduction effect, and is not limited here.

[0191] For example, the deep learning model may include a cascaded feature extraction module, a feature registration module, a feature fusion module, and a noise reduction module. The feature extraction module may be used for feature extraction. The feature registration module may be used to implement feature registration based on a cross-attention mechanism. The feature fusion module may be used for feature fusion. The noise reduction module may be used to obtain video frames. Based on this, how to obtain video frames using a video noise reduction model may be implemented in the following manner.

[0192] The auxiliary image and the reference image are input into the feature extraction module to obtain auxiliary feature data and reference feature data. As an implementation method, the feature extraction module may include feature extraction units corresponding to each of the multiple images. The model parameters of the multiple feature extraction units are different, but the model structure is the same. Therefore, feature extraction is implemented based on the non-sharing of model parameters, which can more effectively reflect the characteristics of each of the multiple images. As another implementation method, the auxiliary image and the reference image are processed using the same feature extraction module to obtain auxiliary feature data and reference feature data. Therefore, feature extraction is implemented based on the sharing of model parameters, which can make the space and semantics of the features consistent and the number of model parameters is small.

[0193] The auxiliary feature data and the reference feature data are input into the feature registration module to obtain the registered auxiliary feature data. The feature registration module can be configured as follows: according to the fourth query matrix and the fourth key matrix, a third attention weight matrix is ​​obtained. According to the third attention weight matrix and the fifth value matrix, the registered auxiliary feature data is obtained. The fourth query matrix can be obtained according to the reference feature data and the seventh transformation matrix. The fourth key matrix can be obtained according to the auxiliary feature data and the eighth transformation matrix. The fifth value matrix can be obtained according to the auxiliary feature data and the ninth transformation matrix. The element values ​​in the seventh transformation matrix, the eighth transformation matrix and the ninth transformation matrix can be model parameters of the deep learning model.

[0194] The reference feature data and the registered auxiliary feature data are input into the feature fusion module to obtain fused feature data. The fused feature data is input into the noise reduction module to obtain a video frame.

[0195] In addition, it is found that due to the different exposure times of multiple images, the image brightness ranges of the multiple images may be different. The difference in image brightness ranges will affect image registration and image fusion, and thus affect the video noise reduction effect. To this end, the embodiment of the present application proposes that the image brightness ranges of multiple images can be made the same by adjusting the image brightness, that is, the image brightness of multiple images can be aligned. For example, since the auxiliary image needs to be registered with the reference image, the image brightness alignment with the reference image can be achieved by adjusting the image brightness of the auxiliary image. As an implementation method, a first ratio between the amount of light entering the auxiliary image and the amount of light entering the reference image can be determined. A second ratio between the original image brightness of the auxiliary image and the first ratio is determined to obtain the image brightness of the auxiliary image. The amount of light entering the auxiliary image can be determined based on the exposure time and sensitivity of the auxiliary image. The amount of light entering the reference image can be determined based on the exposure time and sensitivity of the reference image.

[0196] In addition, it is found that the field of view angles of multiple cameras may be different, which will affect subsequent image fusion and increase computing resource consumption. To this end, the embodiment of the present application proposes that an image cropping method can be used to achieve the alignment of the field of view angles of multiple images. For example, the original image can be cropped to obtain a target image. The target image can be an image taken by a target camera among multiple images. The target camera can be a camera with a large field of view angle among multiple cameras and / or the target camera can be a camera with a long exposure time among multiple cameras.

[0197] As an implementation manner, image cropping may include at least one of the following: proportional cropping, center cropping, edge cropping, etc. The embodiment of the present application does not limit the image cropping manner.

[0198] In addition, it is found that the RAW images are not processed by an image signal processor (ISP), and the ISP processing will make the noise complicated. Therefore, the embodiment of the present application proposes that the multiple images can be RAW images.

[0199] The electronic device applicable to the video processing method provided in the embodiment of the present application and the specific process of the method are described below in conjunction with the embodiment of the present application.

[0200] The video processing method provided in the embodiment of the present application can be used in electronic devices. The electronic devices may include handheld devices, vehicle-mounted devices, etc. with video processing functions. For example, some electronic devices are mobile phones, tablet computers, PDAs, laptop computers, mobile Internet devices (MID), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control (Industrial Control), wireless terminals in self-driving (Self Driving), wireless terminals in remote medical surgery, wireless terminals in smart grids (Smart Grid), wireless terminals in transportation safety (Transportation Safety), wireless terminals in smart cities (Smart City), wireless terminals in smart homes (Smart Home), cellular phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDA), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, terminal devices in 5G networks or future evolved public land mobile communication networks (Public Land Mobile The embodiments of the present application do not limit this.

[0201] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0202] The electronic device in the embodiments of the present application may also be referred to as terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0203] It should be noted that the electronic device described in the embodiments of the present application may be configured with a video processing device.

[0204] As an implementation, the video processing device may include a recording module and a processing module. The recording module may be used for video recording, for example, the recording module may be used to capture a subject using multiple cameras to obtain multiple images. The multiple cameras may include a camera that uses a shorter exposure time to balance the frame rate and a camera that uses a longer exposure time to improve the signal-to-noise ratio.

[0205] The processing module can be used to reduce the noise of the reference image using the auxiliary image to obtain a reduced noise reference image. The reduced noise reference image is a video frame in the video. The reference image can be an image with a shorter exposure time among multiple images. The auxiliary image can be an image with a longer exposure time among multiple images.

[0206] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0207] The electronic device 500 may include a processor 510, an external memory interface 520, an internal memory 521, a Universal Serial Bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, a sensor module 580, a button 590, a motor 591, an indicator 592, a camera 593, a display screen 594, and a Subscriber Identification Module (SIM) card interface 595, etc. Among them, the sensor module 580 may include a pressure sensor 580A, a gyroscope sensor 580B, an air pressure sensor 580C, a magnetic sensor 580D, an acceleration sensor 580E, a distance sensor 580F, a proximity light sensor 580G, a fingerprint sensor 580H, a temperature sensor 580J, a touch sensor 580K, an ambient light sensor 580L, a bone conduction sensor 580M, etc.

[0208] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0209] The processor 510 may include one or more processing units. For example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor, a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0210] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0211] The processor 510 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0212] In some instances, the memory of the embodiment of the present application may store instructions and data for implementing the video processing method described in the embodiment of the present application, and when the processor is running, the video processing method described in the embodiment of the present application may be implemented.

[0213] The internal memory 521 can be used to store computer executable program codes, which include instructions. The internal memory 521 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (for example, a video recording function, etc.), etc. The data storage area may store data created during the use of the electronic device 500, etc. In addition, the internal memory 521 may include a high-speed random access memory, and may also include a non-volatile memory, for example, at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 510 executes various functional applications and data processing of the electronic device 500 by running instructions stored in the internal memory 521 and / or instructions stored in a memory provided in the processor 510. In an embodiment of the present application, the internal memory 521 may be used to implement instructions and data of the video processing method described in an embodiment of the present application.

[0214] The electronic device 500 can implement a video recording function through an ISP, a camera 593 , a video codec, a GPU, a display screen 594 , and an AP, etc. In some embodiments, the ISP can be set in the camera 593 .

[0215] The camera 593 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 500 may include one or N cameras 593, where N may be an integer greater than 1. For example, in an embodiment of the present application, the camera 593 may be used for shooting in the case of video recording.

[0216] The touch sensor 580K is also called a "touch control device". The touch sensor 580K can be arranged on the display screen 594, and the touch sensor 580K and the display screen 594 form a touch screen, also called a "touch control screen". The touch sensor 580K is used to detect touch operations acting on or near it. The touch sensor 580K can pass the detected touch operation to the application processor to determine the type of touch event. For example, in an embodiment of the present application, the touch sensor 580K can be used to pass the detected first operation, second operation, and third operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 594. In other embodiments, the touch sensor 580K can also be set on the surface of the electronic device 500, which is different from the position of the display screen 594.

[0217] The software system of the electronic device 500 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 500.

[0218] Figure 6 A software structure block diagram of an electronic device provided in an embodiment of the present application.

[0219] The layered architecture divides the software into multiple layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer (i.e., Framework Layer), the hardware abstraction layer (HAL), the driver layer (i.e., Driver Layer), and the hardware layer.

[0220] The application layer may include a series of application packages. For example, an application package may include a camera application. When the electronic device runs the camera application, the electronic device may start the camera and use the camera to capture images. In an embodiment of the present application, the electronic device may include multiple cameras.

[0221] The application framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface. The camera access interface may include camera management and camera devices. The camera access interface can be used to provide an API and a programming framework for camera applications.

[0222] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer. It can be an encapsulation of the hardware driver and provide a unified interface for the call of the upper layer application. In an embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer can call the algorithm in the camera algorithm library.

[0223] The camera hardware abstraction layer may provide virtual hardware for multiple camera devices, for example, camera device 1, camera device 2, or more camera devices. The camera device may correspond to a camera configured on the electronic device 500. The electronic device 500 may call the camera through the camera device. The camera algorithm library may include the video processing method provided in the embodiment of the present application, so that the video processing method in the camera algorithm library is called by the camera hardware abstraction layer, and the video processing method described in the embodiment of the present application can be executed.

[0224] The driver layer may be a layer between hardware and software, which may be used to drive the hardware. The driver layer may include drivers for multiple hardware. For example, the driver layer may include a camera device driver, a digital signal processor driver, a graphics processor driver, and a central processing unit driver. The camera device driver may be used to drive the camera to record video and drive the image signal processor to process images. The digital signal processor driver may be used to drive the digital signal processor to process images. The graphics processor driver may be used to drive the graphics processor to process images. The central processing unit driver may be used to drive the central processing unit to process images.

[0225] The hardware layer may include multiple cameras (eg, camera 1, camera 2, other cameras), an image signal processor, a digital signal processor, a graphics processor, a central processing unit, etc. The camera may be used for video recording.

[0226] It should be noted that the embodiment of the present application is only illustrated by the Android system. In other operating systems, if the functions implemented by each functional module are similar to those in the embodiment of the present application, the video processing method described in the embodiment of the present application can also be implemented. The camera application is an application installed on an electronic device that can call the camera to provide a video recording function. It is not limited to the camera application. Other applications that can call the camera to provide a video recording function can be installed on the electronic device.

[0227] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 5 and Figure 6 Taking the electronic device shown as an example, the video processing method provided by the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios. It should be noted that the embodiments of the present application can be implemented independently or in combination with each other, and the same or similar concepts or processes are not repeated in some embodiments.

[0228] It should be noted that the sequence numbers of the steps in the following method are only used as representations of the steps for the purpose of description, and should not be regarded as representing the execution order of the steps. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0229] Figure 7 This is a flow chart of a video processing method proposed in an embodiment of the present application. This method can be applied to Figure 5 and Figure 6 The electronic device shown. The electronic device may include a first camera and a second camera. The field of view of the second camera may be greater than the field of view of the first camera. The first camera may be used to balance the frame rate, whereby the first image captured by the first camera may be used as a reference image. The second camera may be used to improve the signal-to-noise ratio, whereby the second image captured by the second camera may be used as an auxiliary image. On this basis, the second image may be used to reduce noise in the first image.

[0230] As an implementation, the first camera may adopt a high-definition, low signal-to-noise ratio image output mode. The second camera may adopt a low-definition, high signal-to-noise ratio image output mode. Optionally, the high-definition, low signal-to-noise ratio image output mode may include a pixel-in-one mode. For example, the pixel-in-one mode may include a Quadr mode. The low-definition, high signal-to-noise ratio image output mode may include a pixel-in-one mode. For example, the pixel-in-one mode may include a Binning mode.

[0231] It should be noted that the first shooting parameters of the first camera may include an interval duration of a first predetermined duration and an exposure duration of a first exposure duration, thereby, the exposure duration of the first image captured by the first camera is the first exposure duration. The second shooting parameters of the second camera may include an interval duration of a second predetermined duration and an exposure duration of a second exposure duration, thereby, the exposure duration of the second image captured by the second camera is the second exposure duration. The first exposure duration may be less than the second exposure duration. The first predetermined duration may be greater than or equal to the first exposure duration and less than the second predetermined duration. The second predetermined duration may be greater than or equal to the second exposure duration. The shooting time of the first image and the shooting time of the second image may overlap, and the lens orientation of the first camera and the lens orientation of the second camera may be the same, so that the contents of the first image and the second image are similar.

[0232] As an implementation, the first exposure duration may be less than or equal to the frame interval of the first image to balance the frame rate. The second exposure duration may be greater than the frame interval of the first image to improve the signal-to-noise ratio. For example, the frame interval of the first image may be 30 ms. The first exposure duration may be less than or equal to 30 ms. Optionally, the first exposure duration may be 20 ms. The first predetermined duration may be 30 ms. The second exposure duration may be greater than or equal to 100 ms. Optionally, the second exposure duration may be 150 ms. The second predetermined duration may be 150 ms.

[0233] As an implementation manner, the first image may be a short frame image, and the second image may be a long frame image.

[0234] As an implementation, the first image and the second image may have respective timestamps. The timestamp may be used as one of the bases for determining whether the first image and the second image have similar contents, and further may be used as a basis for whether the second image can be used to reduce noise in the first image. As an implementation, when it is determined that the timestamp of the first image and the timestamp of the second image are within a predetermined time range, if the lens orientation of the first camera is the same as the lens orientation of the second camera, it can be said that the contents of the first image and the second image are similar, and thus, the second image can be used to reduce noise in the first image.

[0235] It should also be noted that, since the video processing method described in the embodiment of the present application can be applied to scenes where the video quality is greatly affected by noise, for example, dark light scenes, the video recording mode provided in the embodiment of the present application can be characterized by a dark light recording mode, that is, the dark light recording mode can refer to video recording for scenes where the video quality is greatly affected by noise. Currently, the shooting modes provided by the camera application may include at least one of the following: recording mode, photo mode, night scene mode, portrait mode, movie mode, short video mode, or time-lapse photography mode. The camera application interface may include controls corresponding to the shooting mode, whereby the camera application interface may include recording controls, photo controls, night scene controls, portrait controls, movie controls, short video controls, or time-lapse photography controls.

[0236] Regarding how to combine the dark light recording mode provided in the embodiment of the present application with the camera application, the following is combined with Fig. 8A and Figure 8B To explain. It should be noted that Fig. 8A and Figure 8B The possible interface styles of the dark light recording mode are only schematically shown and should not constitute a limitation on the embodiments of the present application.

[0237] Fig. 8A A schematic diagram of an interface of a dark light recording mode provided in an embodiment of the present application.

[0238] As an implementation method, the low-light recording mode can be used as a shooting mode in parallel with other shooting modes in the camera application. For example, the low-light recording mode is parallel to other shooting modes such as night scene mode, portrait mode, photo mode, recording mode, short video mode, etc. Fig. 8A That is, in the camera application interface, the low-light recording controls are listed alongside controls for other shooting modes, such as night scene controls, portrait controls, photo controls, recording controls, and short video controls.

[0239] Figure 8B A schematic diagram of the interface of another dark light recording mode provided in an embodiment of the present application.

[0240] As another implementation, the low-light recording mode can be used as an optional sub-mode in the recording mode in the camera application. Figure 8B , that is, in the camera application interface, the low-light recording control may be a control displayed when the recording control is triggered.

[0241] like Figure 7 As shown, the method includes S701-S710. Among them, S701-S702 are video recording start steps. S703-S705 are video recording steps. S706-S710 are video noise reduction steps during video recording. S706 is a field of view angle alignment step. S707 is a brightness alignment step. S708 is an image registration step. S709 is an image fusion step. The following will be explained in conjunction with the accompanying drawings.

[0242] In S701 , the camera application generates a recording instruction in response to a first operation.

[0243] According to an embodiment of the present application, the first operation may refer to an operation for starting recording. For example, the first operation may include at least one of the following: a touch operation, a key operation, a gesture operation, or a voice operation. The touch operation may include at least one of the following: a click operation or a slide operation. The recording instruction may be used to instruct the first camera and the second camera to record a video.

[0244] Regarding how to start video recording, the following Fig. 9A , Fig. 9B , Fig. 9C and Fig.9D To explain. It should be noted that Fig. 9A , Fig. 9B and Fig. 9C The possible startup methods are only schematically shown and should not constitute a limitation on the embodiments of the present application.

[0245] As an implementation method, that is, an active startup method. The active startup method may refer to the user actively triggering the low-light recording mode. Depending on the different ways in which the low-light recording mode is combined with the camera application, the active startup method may include the following implementation methods.

[0246] As a shooting mode in parallel with other shooting modes in the camera application, as an implementation, in response to the second operation, a first camera application interface is displayed. The first camera application interface may include a dark light recording control. In response to a third operation on the dark light recording control, the dark light recording interface is displayed.

[0247] Specifically, the camera application can call the camera access interface of the application framework layer in response to the second operation to start the camera application. When the low-light recording control included in the first camera application interface is displayed, the camera application can call the camera access interface of the application framework layer in response to the third operation on the low-light recording control to start the low-light recording mode.

[0248] As another implementation, in response to the second operation, a dark light recording interface is displayed. In this case, when the camera application is started, the dark light recording mode is started by default. Thus, the dark light recording mode can be the default startup mode of the camera application or the last time the dark light recording mode was started. In some embodiments, if the user used the dark light recording mode of the camera application last time, the camera application is started again, and the camera application can start the dark light recording mode by default. In other embodiments, the camera application can set the dark light recording mode as the default startup mode. Specifically, the camera application can call the camera access interface of the application framework layer in response to the second operation to start the camera application. When the camera application is started, the dark light recording interface is displayed.

[0249] It should be noted that the second operation may refer to an operation for starting a camera application. For other descriptions of the second operation and the third operation, please refer to the description of the first operation above, which will not be repeated here.

[0250] Combine the following Fig. 9A To explain. Fig. 9A A schematic diagram of starting a dark light recording mode provided in an embodiment of the present application.

[0251] like Fig. 9A As shown, in response to the second operation on the first camera application control 901, the first camera application interface 902 is displayed. The first camera application interface 902 may include a low-light recording control 903. In response to the third operation on the low-light recording control 903, the low-light recording interface 904 is displayed, thereby starting the low-light recording mode. The second operation and the third operation may be touch operations.

[0252] For the low-light recording mode as an optional sub-mode in the recording mode in the camera application, as an implementation, in response to the fourth operation, the second camera application interface is displayed. The second camera application interface may include a recording control. In response to the fifth operation on the recording control, the first recording interface is displayed. The first recording interface may include a low-light recording control. In response to the third operation on the low-light recording control, the low-light recording interface is displayed.

[0253] Specifically, the camera application may call the camera access interface of the application framework layer in response to the fourth operation to start the camera application. When the recording control included in the second camera application interface is displayed, the camera application may call the camera access interface of the application framework layer in response to the fifth operation for the recording control to display the first recording interface. When the low-light recording control included in the first recording interface is displayed, the camera application may call the camera access interface of the application framework layer in response to the third operation for the low-light recording control to start the low-light recording mode.

[0254] As another implementation, in response to the fourth operation, the first recording interface is displayed. In response to the third operation for the dark light recording control, the dark light recording interface is displayed. In this case, when the camera application is started, the recording mode is started by default. Thus, the recording mode can be the default startup mode of the camera application or the last time the recording mode was started. In some embodiments, if the user used the recording mode of the camera application last time, the camera application can start the recording mode by default when the camera application is started again. In other embodiments, the camera application can set the recording mode to the default startup mode. Specifically, the camera application can call the camera access interface of the application framework layer in response to the fourth operation to start the camera application. When the camera application is started, the first recording interface is displayed. When the dark light recording control included in the first recording interface is displayed, the camera application can call the camera access interface of the application framework layer in response to the third operation for the dark light recording control to start the dark light recording mode.

[0255] It should be noted that the fourth operation may refer to an operation for starting a camera application. For other descriptions of the fourth and fifth operations, please refer to the description of the first operation above, which will not be repeated here. Fig. 9B To explain. Fig. 9B A schematic diagram of another method of starting a dark light recording mode provided in an embodiment of the present application.

[0256] like Fig. 9BAs shown, in response to the fourth operation on the second camera application control 905, the second camera application interface 906 is displayed. The second camera application interface 906 may include a recording control 907. In response to the fifth operation on the recording control 907, the first recording interface 908 is displayed. The first recording interface 908 may include a dark light recording control 909. In response to the third operation on the dark light recording control 909, the dark light recording interface is displayed. The dark light recording interface may be Fig. 9A The dark light recording interface 904 is displayed, thereby starting the dark light recording mode. The fourth operation, the fifth operation and the third operation can be touch operations.

[0257] As another implementation, namely, an automatic start mode, the automatic start mode may refer to automatically starting the low-light recording mode in response to detecting a scene where the video quality is greatly affected by noise (eg, a low-light scene) when the recording mode is started.

[0258] In response to detecting that the scene is in a low-light scene, a low-light recording interface is displayed, thereby starting a low-light recording mode.

[0259] How to detect a dark scene can be achieved in the following ways.

[0260] As an implementation method, when the ambient light intensity is less than or equal to the predetermined ambient light intensity, it is determined to be in a dark light scene. The ambient light intensity may refer to the luminous flux received per unit area, which describes the distribution of light in space. The predetermined ambient light intensity may be configured according to actual business needs and is not limited here. For example, the predetermined ambient light intensity may be less than or equal to 5 lux.

[0261] As another implementation, when the ambient light brightness is less than or equal to a predetermined ambient light brightness, it is determined to be in a dark light scene. The ambient light brightness may refer to the intensity of light emitted by a light source or the surface of a subject. The predetermined ambient light intensity may be configured according to actual business needs and is not limited here. The ambient light brightness may be sensed by an ambient light sensor of an electronic device. Optionally, the ambient light brightness may be determined based on an aperture value, image brightness of a predetermined image, exposure time of a predetermined image, and sensitivity of a predetermined image. Optionally, the image brightness of a predetermined image may be used to indicate the ambient light brightness. The image brightness may be determined based on an average brightness method or a weighted mean method, etc. The average brightness method may refer to determining the average brightness of the entire image. The predetermined image may be an image obtained in a recording mode.

[0262] In response to detecting that a low-light scene is present, displaying a low-light recording interface may be implemented in the following manner.

[0263] As an implementation, in response to the fourth operation, a second camera application interface is displayed. The second camera application interface may include a recording control. In response to a fifth operation on the recording control, a second recording interface is displayed. Or in response to the fourth operation, the second recording interface is displayed. In the case where the second recording interface is displayed, in response to detecting that it is in a dark light scene, a dark light recording interface is displayed. The fourth operation may refer to an operation for starting a camera application.

[0264] As another implementation, in response to the fourth operation, the second recording interface is displayed. When the second recording interface is displayed, in response to detecting that the scene is in a dark light, the dark light recording interface is displayed. In this case, when the camera application is started, the recording mode is started by default. Thus, the recording mode can be the default startup mode of the camera application or the last time the recording mode was started.

[0265] As another implementation, in response to detecting that the scene is in a dark light, a prompt message is generated. The prompt message may be used to prompt whether to start the dark light recording mode. In response to a sixth operation on the prompt message, a dark light recording interface is displayed. The sixth operation may refer to an operation for starting the dark light recording mode. For other descriptions of the sixth operation, please refer to the description of the first operation above, which will not be repeated here.

[0266] As another implementation, in response to the fourth operation, a second camera application interface is displayed. The second camera application interface may include a recording control. In response to a fifth operation on the recording control, a second recording interface is displayed. Or in response to the fourth operation, the second recording interface is displayed. In the case of displaying the second recording interface, in response to detecting that it is in a dark light scene, a prompt message is generated. The prompt message may be used to prompt whether to start the dark light recording mode. In response to a sixth operation on the prompt message, a dark light recording interface is displayed.

[0267] As another implementation, in response to the fourth operation, the second recording interface is displayed. When the second recording interface is displayed, in response to detecting that the scene is in a dark light, a prompt message is generated. In response to the sixth operation for the prompt message, the dark light recording interface is displayed. In this case, when the camera application is started, the recording mode is started by default. Thus, the recording mode can be the default startup mode of the camera application or the last time the recording mode was started.

[0268] Combine the following Fig. 9C To explain. Fig. 9C A schematic diagram of another method of starting a dark light recording mode provided in an embodiment of the present application.

[0269] like Fig. 9CAs shown, in response to the fourth operation on the second camera application control 905, the second camera application interface 906 is displayed. The second camera application interface 906 may include a recording control 907. In response to the fifth operation on the recording control 907, the second recording interface 910 is displayed. In the case where the second recording interface 910 is displayed, in response to detecting that it is in a dark light scene, prompt information 911 is generated. Prompt information 911 can be used to prompt whether to start the dark light recording mode. In response to the sixth operation on the prompt information 911, the dark light recording interface is displayed. The dark light recording interface can be Fig. 8A The dark light recording interface 904 is displayed, thereby starting the dark light recording mode. The fourth operation, the fifth operation and the sixth operation can be touch operations.

[0270] The following describes how to record a video with low-light recording mode enabled.

[0271] The dark-light recording interface may include a start control. In response to a first operation on the start control, video recording is performed. Specifically, when the start control included in the dark-light recording interface is displayed, the camera application may send a recording instruction to the first camera and the second camera through the camera access interface, the camera hardware abstraction layer, and the camera device driver in response to the first operation on the start control. The recording instruction may be used to instruct the first camera and the second camera to perform video recording.

[0272] Combine the following Fig.9D To explain. Fig.9D A schematic diagram of a dark light recording mode provided in an embodiment of the present application.

[0273] like Fig.9D As shown, the dark light recording interface 904 may include a start control 912. In response to a first operation on the start control 912, video recording is performed. Fig.9D The right side of the dotted line schematically shows the recording screen 913 of the first camera. Fig.9D The left side of the middle dotted line schematically shows the recording screen 914 of the second camera. Fig.9D It can be seen that since the field of view of the second camera is greater than the field of view of the first camera, the recording screen 914 shows more content than the recording screen 913.

[0274] The camera application may call the camera access interface of the application framework layer in response to the second operation on the camera application to start the camera application. In the case of displaying the low-light recording control included in the camera application interface, the camera application may call the camera access interface of the application framework layer in response to the first operation on the low-light recording control to start the low-light scene recording mode. In the case of displaying the start control included in the low-light recording interface, the camera application may send a recording request to the application framework layer in response to the third operation on the start control, and the recording request may be used to request the camera to perform video recording.

[0275] In response to receiving the recording request, the application framework layer sends a recording request to multiple camera device drivers in the camera hardware abstraction layer. In response to receiving the recording request, the camera device driver drives the camera corresponding to the camera device to capture the subject to obtain an image. The multiple cameras may include a first camera and a second camera. The first camera may refer to a camera that uses a shorter exposure time to balance the frame rate. The second camera may refer to a camera that uses a longer exposure time to improve the signal-to-noise ratio. The multiple images may include a reference image captured by the first camera and an auxiliary image captured by the second camera. The exposure time of the reference image is less than the exposure time of the auxiliary image. The camera may send an image to the camera hardware abstraction layer through the camera device driver layer. The camera hardware abstraction layer may store the image in a memory buffer.

[0276] At S702 , the camera application transmits a recording instruction to the first camera and the second camera through the camera access interface, the camera hardware abstraction layer, and the camera device driver.

[0277] According to an embodiment of the present application, the camera access interface transmits the recording instruction to the camera hardware abstraction layer and the camera device driver in response to receiving the recording instruction. The camera device driver transmits the recording instruction to the first camera and the second camera in response to receiving the recording instruction, so as to drive the first camera and the second camera to capture the subject to obtain the first image and the second image.

[0278] In S703, in response to receiving the recording instruction, the first camera shoots according to the first shooting parameter to obtain a first image, and in response to receiving the recording instruction, the second camera shoots according to the second shooting parameter to obtain a second image.

[0279] According to an embodiment of the present application, the first image may include at least one. The second image may include at least one.

[0280] In S704, the first camera sends the first image to the camera hardware abstraction layer through the camera device driver, and the second camera transmits the second image to the camera hardware abstraction layer through the camera device driver.

[0281] At S705 , the camera hardware abstraction layer transmits the first image and the second image to the camera algorithm library.

[0282] According to an embodiment of the present application, the camera hardware abstraction layer can call the video processing method in the camera algorithm library to implement video noise reduction using the video processing method. For example, the camera hardware abstraction layer can use the second image to reduce the noise of the first image based on the support of the image signal processor, the digital signal processor, and the graphics processor to obtain a first image after noise reduction. The first image after noise reduction is a video frame in the video.

[0283] In S706 , the camera algorithm library crops the second image to obtain a cropped second image.

[0284] According to an embodiment of the present application, the camera hardware abstraction layer may call a video processing method in a camera algorithm library to use the video processing method to achieve alignment of the field of view angles of the first image and the second image. For example, since the field of view angle of the second camera is greater than the field of view angle of the first camera, the second image may be cropped so that the cropped second image can be aligned with the field of view angle of the first image. Image cropping may include at least one of the following: proportional cropping, center cropping, or edge cropping, etc. The embodiment of the present application does not limit the image cropping method.

[0285] In S707, the camera algorithm library adjusts the original image brightness of the cropped second image according to the first exposure time and sensitivity of the first image and the second exposure time, sensitivity and original image brightness of the cropped second image to obtain a second image with adjusted brightness.

[0286] According to an embodiment of the present application, the camera hardware abstraction layer may call a video processing method in a camera algorithm library to achieve brightness alignment of a first image and a cropped second image using the video processing method. For example, a brightness-adjusted second image is obtained based on a ratio and the original image brightness of the cropped second image. The ratio may be determined based on a second amount of light entering the second image and a first amount of light entering the first image. The first amount of light entering may be determined based on a first exposure time and sensitivity of the first image. The second amount of light entering may be determined based on a second exposure time and sensitivity of the cropped second image.

[0287] As an implementation manner, the original image brightness of the cropped second image may be adjusted according to the following formulas (1)-(2).

[0288] (1)

[0289] (2)

[0290] in, The image brightness of the brightness adjusted second image may be characterized. The original image brightness of the cropped second image may be characterized. Can characterize ratio. The first exposure duration can be represented. The sensitivity of the first image may be characterized. The second exposure duration can be represented. The sensitivity of the second image can be characterized.

[0291] In S708 , the camera algorithm library uses a cross-attention mechanism to align the second feature data of the second image whose brightness has been adjusted with the first feature data of the first image to obtain aligned second feature data.

[0292] According to an embodiment of the present application, it should be noted that the second image after brightness adjustment is referred to as the second image below. The camera hardware abstraction layer can call the video processing method in the camera algorithm library to implement image registration using the video processing method. For example, the second feature data of the second image and the first feature data of the first image are registered using a cross-attention mechanism to obtain the registered second feature data. The first feature data can be used for query (ie, Q). The second feature data can be used for key (ie, K) and value (ie, V).

[0293] According to an embodiment of the present application, the first feature data may include multiple first features. The second feature data may include multiple second features. The cross-attention mechanism may include at least one of the following: a single-head cross-attention mechanism or a multi-head cross-attention mechanism. The similarity determination method involved in the cross-attention mechanism may include at least one of the following: dot product similarity, scaled dot product similarity, additive similarity or cosine similarity, etc.

[0294] Regarding how to use the cross-attention mechanism to achieve alignment, it can be achieved in the following ways.

[0295] As an implementation, the registered second feature data may be obtained based on at least one fourth intermediate matrix. The fourth intermediate matrix may be obtained based on the third attention weight matrix and the fifth value matrix. The third attention weight matrix may be obtained based on the fourth query matrix and the fourth key matrix. The third attention weight matrix may be used to indicate the similarity between the second feature and the first feature data.

[0296] The fourth query matrix may be obtained by performing a seventh linear transformation on the first feature data. For example, the fourth query matrix may be obtained based on the first feature data and the seventh transformation matrix. The fourth key matrix may be obtained by performing an eighth linear transformation on the second feature data. For example, the fourth key matrix may be obtained based on the second feature data and the eighth transformation matrix. The fifth value matrix may be obtained by performing a ninth linear transformation on the second feature data. For example, the fifth value matrix may be obtained based on the second feature data and the ninth transformation matrix.

[0297] It should be noted that, when there is one fourth intermediate matrix, the above method can be understood as a single-head cross attention mechanism. When there are multiple fourth intermediate matrices, the above method can be understood as a multi-head cross attention mechanism. The transformation matrix groups corresponding to at least one fourth intermediate matrix can be different. The transformation matrix group can include a seventh transformation matrix, an eighth transformation matrix, and a ninth transformation matrix. The element values ​​in the transformation matrix group can be model parameters of the deep learning model.

[0298] Regarding how to obtain the third attention weight matrix, it can be achieved in the following way.

[0299] As an implementation, based on dot product similarity or scaled dot product similarity. For example, the third attention weight matrix can be obtained by normalizing the fifth intermediate matrix. For example, the third attention weight matrix can be obtained by processing the fifth intermediate matrix using a Softmax function, a Sigmoid function, or a tanh function. The fifth intermediate matrix can be obtained based on the dot product between the fourth query matrix and the fourth key matrix.

[0300] As another implementation, based on additive similarity. For example, the third attention mechanism can be obtained by normalizing the sixth intermediate matrix. The sixth intermediate matrix can be obtained by performing the tenth linear transformation on the seventh intermediate matrix. For example, the sixth intermediate matrix can be obtained based on the seventh intermediate matrix and the tenth transformation matrix. The element values ​​in the tenth transformation matrix can be model parameters of the deep learning model. The seventh intermediate matrix can be obtained by performing nonlinear activation on the eighth intermediate matrix. The eighth intermediate matrix can be obtained based on the fourth query matrix and the fourth key matrix.

[0301] Regarding how to obtain the eighth intermediate matrix, it can be achieved in the following manner.

[0302] As an implementation, the eighth intermediate matrix may be obtained according to the fifth query matrix and the fifth key matrix. The fifth query matrix may be obtained by performing the eleventh linear transformation on the fourth query matrix. For example, the fifth query matrix may be obtained according to the fourth query matrix and the eleventh transformation matrix. The fifth key matrix may be obtained by performing the twelfth linear transformation on the fourth key matrix. For example, the fifth key matrix may be obtained according to the fourth key matrix and the twelfth transformation matrix.

[0303] As another implementation, the eighth intermediate matrix may be obtained by performing a thirteenth linear transformation on the ninth intermediate matrix. For example, the eighth intermediate matrix may be obtained based on the ninth intermediate matrix and the thirteenth transformation matrix. The ninth intermediate matrix may be obtained by concatenating the fourth query matrix and the fourth key matrix.

[0304] It should be noted that the element values ​​in the eleventh transformation matrix, the twelfth transformation matrix and the thirteenth transformation matrix can be model parameters of the deep learning model.

[0305] In order to reduce the computational complexity while maintaining or improving the accuracy of feature registration, so that the video processing method described in the embodiment of the present application can be better applied to electronic devices with limited resources, such as terminal devices. To this end, a method of optimizing the cross attention mechanism based on an optimization strategy can be adopted.

[0306] As an implementation manner, the optimization strategy may include at least one of the following: a linear strategy, a sparse strategy, a quantization strategy, a pruning strategy or other strategies.

[0307] Regarding how to optimize the cross-attention mechanism based on the sparse strategy, it can be achieved in the following ways.

[0308] As an implementation method, the fourth key matrix may be obtained by performing an eighth linear transformation and sparse selection on the second feature data of the second image. For example, the fourth key matrix may be obtained by performing a sparse selection on the sixth key matrix. The sixth key matrix may be obtained by performing an eighth linear transformation on the second feature data of the second image. Sparse selection may refer to selecting element values ​​that meet predetermined conditions from the sixth key matrix so that the fourth key matrix is ​​formed by these element values, that is, the element values ​​in the fourth key matrix meet the predetermined conditions. The element values ​​that meet the predetermined conditions may refer to element values ​​in the sixth key matrix whose similarity with the elements in the fourth query matrix is ​​greater than or equal to the first predetermined threshold.

[0309] Since the fourth key matrix is ​​obtained by sparsely selecting the sixth key matrix, the data volume of the fourth key matrix is ​​reduced, thereby reducing the consumption of computing resources, so that the video processing method described in the embodiment of the present application can be better applied to electronic devices with limited resources.

[0310] Regarding how to optimize the cross-attention mechanism based on quantization strategy and pruning strategy, it can be achieved in the following ways.

[0311] As an implementation method, the third attention weight matrix can be obtained based on the fourth attention weight matrix. For example, the third attention weight matrix can be obtained by quantizing the twelfth intermediate matrix. Optionally, the third attention weight matrix can be obtained by quantizing the twelfth intermediate matrix to a 4-bit or 8-bit fixed-point integer. The twelfth intermediate matrix can be obtained by pruning the fourth attention weight matrix. Optionally, the twelfth intermediate matrix can be obtained by setting the element values ​​in the fourth attention weight matrix that are less than or equal to the second predetermined threshold to zero. The second predetermined threshold can be configured according to actual business needs and is not limited here.

[0312] The fourth attention weight matrix may be obtained by normalizing the tenth intermediate matrix. The tenth intermediate matrix may be obtained by dequantizing the eleventh intermediate matrix. The eleventh intermediate matrix may be obtained based on the sixth query matrix and the sixth key matrix. The sixth query matrix and the sixth key matrix may be obtained by quantizing the fourth query matrix and the fourth key matrix.

[0313] Regarding how to optimize the cross attention mechanism based on other strategies, it can be achieved in the following ways. For ease of understanding, the following is combined Fig.10 and Fig.11 To explain.

[0314] As a way to achieve this, Fig.10 The figure is a schematic diagram of a principle for obtaining the registered second feature data according to an embodiment of the present application.

[0315] like Fig.10 As shown, the registered second feature data can be obtained according to the first query matrix, the first intermediate matrix and the first value matrix. The first intermediate matrix can be used for K (i.e., key). The first query matrix can be used for Q (i.e., query). The first value matrix can be used for V (i.e., value).

[0316] The first value matrix may be obtained according to the first intermediate matrix, the first key matrix, and the second value matrix. The first intermediate matrix may be used for Q. The first key matrix may be used for K. The second value matrix may be used for V. For example, the first value matrix may be obtained by processing the first intermediate matrix, the first key matrix, and the second value matrix using a cross attention mechanism. Optionally, the first value matrix may be obtained according to the fifth attention weight matrix and the second value matrix. The fifth attention weight matrix may be obtained according to the first intermediate matrix and the first key matrix.

[0317] Since the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix, and the first intermediate matrix is ​​used as a query, the first intermediate matrix can be used to aggregate information of the first key matrix and the second value matrix. Since the registered second feature data is obtained according to the first query matrix, the first intermediate matrix and the first value matrix, and the first query matrix is ​​used as a query and the first intermediate matrix is ​​used as a key, it is explained that the first intermediate matrix can pass the aggregated information to the first query matrix.

[0318] The first query matrix may be obtained by performing a first linear transformation on the first feature data of the first image. For example, the first query matrix may be obtained based on the first feature data of the first image and the first transformation matrix. The element values ​​in the first transformation matrix may be model parameters of the deep learning model.

[0319] The first key matrix may be obtained by performing a second linear transformation on the second feature data of the second image. For example, the first key matrix may be obtained based on the second feature data of the second image and the second transformation matrix. The element values ​​in the second transformation matrix may be model parameters of the deep learning model.

[0320] The second value matrix may be obtained by performing a third linear transformation on the second feature data. For example, the second value matrix may be obtained based on the second feature data and the third transformation matrix.

[0321] The first intermediate matrix may be obtained according to the first query matrix. For example, the first intermediate matrix may be obtained by pooling the first query matrix. Pooling may include maximum pooling, average pooling, or L2 norm pooling, etc. Optionally, the first intermediate matrix may be obtained according to the first query matrix and the fourteenth transformation matrix. The element values ​​in the fourteenth transformation matrix may be model parameters of the deep learning model. Optionally, the first intermediate matrix may be obtained by fusing similar elements in the first query matrix.

[0322] Regarding how to fuse similar elements in the first query matrix, it can be achieved in the following manner.

[0323] As an implementation method, the first intermediate matrix can be obtained based on fused elements and non-fused elements. The fused element can be obtained by fusing the target element and other elements having an association relationship. The other elements having an association relationship with the target element can be the elements most similar to the target element determined from other sets. The target element can be any one in the target set. When the target set is the first set, the other set is the second set. When the target set is the second set, the other set is the first set. The first set and the second set can be obtained by dividing the elements included in the first query matrix. Non-fused elements can refer to elements that do not have an association relationship with other elements.

[0324] Regarding how to obtain the registered second feature data according to the first query matrix, the first intermediate matrix and the first value matrix, it can be implemented in the following manner.

[0325] As an implementation manner, the registered second feature data may be obtained according to the second intermediate matrix and the third value matrix. For example, the registered second feature data may be obtained by adding the second intermediate matrix and the third value matrix.

[0326] The second intermediate matrix can be obtained according to the first query matrix, the first intermediate matrix and the first value matrix. For example, the third intermediate matrix can be obtained according to the sixth attention weight matrix and the first value matrix. The sixth attention weight matrix can be obtained according to the first query matrix and the first intermediate matrix.

[0327] The third value matrix may be obtained by performing a first depth-wise separable convolution on the first value matrix.

[0328] Since the third value matrix is ​​obtained by performing a first depth-wise separable convolution on the first value matrix, feature diversity is improved.

[0329] As another implementation, the registered second feature data may be obtained according to the sixth attention weight matrix and the first value matrix. The sixth attention weight matrix may be obtained according to the first query matrix and the first intermediate matrix.

[0330] Since the first intermediate matrix is ​​obtained according to the first query matrix, the data volume of the first intermediate matrix can be less than the data volume of the first query matrix. Therefore, in the process of determining the first value matrix, the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix, and the first intermediate matrix is ​​used as a query. Therefore, compared with the first query matrix as a query, the data processing volume is reduced, the computing resource consumption is reduced, and the first value matrix aggregates the information of the first key matrix and the second value matrix. In the process of determining the second feature data after registration, the second feature data after registration is obtained according to the first value matrix obtained by the first query matrix, the first intermediate matrix and the first value matrix. The first query matrix is ​​used as a query, the first intermediate matrix is ​​used as a key, and the first value matrix is ​​used as a value. Therefore, the first value matrix can pass the aggregated information to the first query matrix, thereby realizing global modeling. Among them, the first intermediate matrix can be used to aggregate the information of the first key matrix and the second value matrix, and pass the aggregated information to the first query matrix.

[0331] As another implementation, Fig.11 A schematic diagram of another principle for obtaining the registered second feature data provided in an embodiment of the present application.

[0332] like Fig.11As shown, the registered second feature data may be obtained according to the third intermediate matrix and the second query matrix. For example, the registered second feature data may be obtained by multiplying the third intermediate matrix and the second query matrix.

[0333] The third intermediate matrix may be obtained according to the second key matrix and the fourth value matrix.

[0334] The second query matrix and the second key matrix can be obtained by processing the third query matrix and the third key matrix respectively using a focusing function. The focusing function can be used to adjust the proximity between the third query matrix and the third key matrix, that is, the aggregation function can be used to make similar elements in the third query matrix and the third key matrix closer and dissimilar elements farther away. For example, the third query matrix can be input into the aggregation function to obtain the second query matrix. The third key matrix is ​​input into the aggregation function to obtain the second key matrix.

[0335] As an implementation manner, the second query matrix and the second key matrix may be determined according to the following formulas (3)-(4).

[0336] (3)

[0337] (4)

[0338] in, Aggregation functions can be characterized. It can be the third query matrix or the fourth query matrix. It can be the second query matrix or the second key matrix. Can be characterized Model. Can be characterized of Power. Can be greater than 1.

[0339] The third query matrix may be obtained by performing a fourth linear transformation on the first feature data of the first image. For example, the third query matrix may be obtained based on the first feature data of the first image and the fourth transformation matrix.

[0340] The third key matrix may be obtained by performing the fifth linear transformation on the second feature data of the second image. For example, the third key matrix may be obtained according to the second feature data of the second image and the fifth transformation matrix.

[0341] The third intermediate matrix may be obtained according to the second key matrix and the fourth value matrix. For example, the third intermediate matrix may be obtained by multiplying the second key matrix and the fourth value matrix.

[0342] The fourth value matrix may be obtained by performing a sixth linear transformation on the second feature data of the second image. For example, the fourth matrix may be obtained according to the second feature data of the second image and the sixth transformation matrix.

[0343] It should be noted that the element values ​​in the fourth transformation matrix, the fifth transformation matrix and the sixth transformation matrix can be model parameters of the deep learning model.

[0344] Since the aggregation function can be used to adjust the proximity between the third query matrix and the third key matrix, that is, the aggregation function can be used to make similar elements in the third query matrix and the third key matrix closer and dissimilar elements farther away, it is achieved that the second feature with a large similarity between the aligned second feature data and the first feature data is given a larger weight, and the second feature with a small similarity between the second feature data and the first feature data is given a smaller weight, thereby improving the accuracy of feature alignment and further improving the video noise reduction effect. In addition, since the third intermediate matrix is ​​obtained based on the second key matrix and the fourth value matrix, the aligned second feature data is obtained based on the second query matrix and the third intermediate matrix, that is, the key and value are calculated first, and the result is then calculated with the query, thereby reducing the amount of data processing and reducing the consumption of computing resources.

[0345] As an implementation, the first feature data may be determined based on the first visual feature data and the first semantic feature data. Optionally, the first feature data may be determined based on the first visual feature data, the first semantic feature data and the first position code. The first position code may be used to indicate a pixel position of a pixel in the first image.

[0346] The second feature data may be determined based on the second visual feature data and the second semantic feature data. Optionally, the second feature data may be determined based on the second visual feature data, the second semantic feature data, and the second position code. The second position code may be used to indicate the pixel position of a pixel in the second image.

[0347] Since position coding can facilitate better capturing of dependencies and patterns between features in feature data and can focus on the relative positions of features, the reliability of the first feature data and the second feature data is improved, thereby improving the video noise reduction effect.

[0348] As an implementation, the first query matrix may be obtained by performing a second depth classifiable convolution on the first feature data of the first image. And / or, the first key matrix may be obtained by performing a third depth separable convolution on the second feature data of the second image. And / or, the first value matrix may be obtained by performing a fourth depth separable convolution on the second feature data.

[0349] Since the first query matrix, the first key matrix, and the first value matrix can be obtained by using a deep classifiable convolution method, the deep separable convolution can reduce the amount of data processing, thereby reducing the amount of data processing and reducing computing resource consumption.

[0350] In S709 , the camera algorithm library fuses the first feature data and the registered second feature data to obtain fused feature data.

[0351] According to the embodiment of the present application, the camera hardware abstraction layer can call the video processing method in the camera algorithm library to use the video processing method to achieve image fusion. How to achieve image fusion can be achieved in the following ways. Fig.12 To explain.

[0352] As an implementation method, the first feature data and the registered second feature data are concatenated to obtain fused feature data. Optionally, the first feature data and the registered second feature data are added to obtain fused feature data. Optionally, the first feature data and the registered second feature data are multiplied to obtain fused feature data.

[0353] As another implementation, Fig.12 A schematic diagram of the principle of obtaining fused feature data provided in an embodiment of the present application.

[0354] like Fig.12 As shown, the fused feature data may be obtained based on the first feature data and the third feature data. For example, the fused feature data may be obtained by fusing the first feature data and the third feature data. The third feature data may be obtained based on the fourth feature data and the registered second feature data. For example, the third feature data may be obtained by multiplying the fourth feature data and the registered second feature data. The fourth feature data may be obtained based on the first feature data and the registered second feature data. For example, the fourth feature data may be obtained by convolving the first feature data and the registered second feature data.

[0355] Since the fused feature data is obtained based on the first feature data and the third feature data, the third feature data is obtained based on the fourth feature data and the registered second feature data, and the fourth feature data is obtained based on the first feature data and the registered second feature data, a relatively complete fusion of the first feature data and the second feature data is achieved, thereby making the information carried by the fused feature data more comprehensive and accurate.

[0356] At S710, the camera algorithm library obtains a video frame according to the fused feature data.

[0357] According to an embodiment of the present application, the camera hardware abstraction layer can call the video processing method in the camera algorithm library to obtain the video frame by using the video processing method. For example, the fusion feature data can be decoded to obtain the video frame.

[0358] Since the fused feature data is obtained by fusing the first feature data and the registered second feature data, the fused feature data can reflect the fusion characteristics of the first feature data and the registered second feature data, thereby making the information carried by the fused feature more comprehensive and accurate. On this basis, the fused feature data is decoded to obtain a video frame, thereby improving the quality of the video frame, thereby improving the video noise reduction effect.

[0359] Regarding how to decode the fused feature data to obtain the video frame, it can be achieved in the following ways.

[0360] As an implementation, the video frame may be obtained based on the first feature data and the fifth feature data. The fifth feature data may be obtained based on the sixth feature data. The sixth feature data may be obtained based on the seventh feature data and the fused feature data. The seventh feature data may be obtained based on the eighth feature data. The eighth feature data may be obtained based on the fused feature data and the ninth feature data. The ninth feature data may be obtained based on the fused feature data.

[0361] Since the video frame is obtained based on the first feature data and the fifth feature data, the fifth feature data is obtained based on the sixth feature data, the sixth feature data is obtained based on the seventh feature data and the fused feature data, the seventh feature data is obtained based on the eighth feature data, the eighth feature data is obtained based on the fused feature data and the ninth feature data, and the ninth feature data can be obtained based on the fused feature data, therefore, more full utilization of the feature data is achieved, thereby improving the quality of the video frame.

[0362] It should be noted that the second image may be used to reduce the noise of the first image during the video recording process, and the second image may be used to reduce the noise of the first image after the video recording is finished.

[0363] It should also be noted that Figure 7 It is only schematically shown that the viewing angle alignment step S706 can be performed first, and then the brightness alignment step S707 can be performed. In addition, the brightness alignment step S707 can also be performed first, and then the viewing angle alignment step S706 can be performed. The embodiment of the present application does not limit the execution order of the viewing angle alignment step and the brightness alignment step. In addition, Figure 7 S706 and S707 may be optional steps.

[0364] Fig.13A flowchart of another video processing method provided in an embodiment of the present application.

[0365] like Fig.13 As shown, the method includes S1310-S1320.

[0366] At S1310 , in response to a first operation, video recording is performed.

[0367] As an implementation, in response to the second operation, a camera application interface is displayed. The camera application interface may include a dark light recording control. In response to a third operation on the dark light recording control, a dark light recording interface is displayed. The dark light recording interface may include a start control. In response to the first operation on the start control, video recording is performed.

[0368] Since the camera application can provide a low-light recording mode, low-light video recording can be performed using the camera application.

[0369] As another implementation, in response to detecting that the scene is in low light, a low light recording interface is displayed. In response to a first operation on the start control, video recording is performed.

[0370] When a low-light scene is detected, a low-light recording interface is displayed to automatically start the low-light recording mode, thereby simplifying the steps for starting low-light recording and improving the convenience of low-light recording.

[0371] As another implementation, in response to the second operation, the low-light recording interface is displayed. In response to the first operation on the start control, the video recording is performed.

[0372] When the second operation is detected, the dark light recording interface is displayed, so that when the camera application is started, the dark light recording mode is started by default. The dark light recording mode can be the default startup mode of the camera application or the last time the dark light recording mode was started, thereby simplifying the steps of starting the dark light recording and improving the convenience of the dark light recording.

[0373] It should be noted that for the description of the above implementation method, please refer to the above Figure 7 , Fig. 8A , Figure 8B , Fig. 9A , Fig. 9B , Fig. 9C and Fig.9D The corresponding parts will not be repeated here.

[0374] In S1320, during video recording, the first image is denoised according to the second image to obtain a video frame.

[0375] According to an embodiment of the present application, a video frame may be a frame in a recorded video. The first image and the second image may be taken using a plurality of cameras. The shooting time of the first image and the shooting time of the second image may overlap. The lens orientation of the first camera may be the same as the lens orientation of the second camera. The first camera may be a camera that shoots the first image. The second camera may be a camera that shoots the second image. The exposure time of the first image may be less than the exposure time of the second image.

[0376] As an implementation manner, the definition of the first image may be higher than the definition of the second image.

[0377] By making the clarity of the first image higher than that of the second image, the video noise reduction effect is further improved.

[0378] As another implementation, the exposure time of the second image may be greater than the frame interval of the first image. The exposure time of the first image may be less than or equal to the frame interval of the first image.

[0379] By setting the exposure time of the reference image to be less than or equal to the frame interval of the reference image, a balanced frame rate is achieved, and by setting the exposure time of the auxiliary image to be greater than the frame interval of the reference image, an improved signal-to-noise ratio is achieved. Thus, the video noise reduction effect is improved while balancing the frame rate.

[0380] As another implementation, the definition of the first image may be higher than that of the second image. Furthermore, the exposure time of the second image may be greater than the frame interval of the first image. The exposure time of the first image may be less than or equal to the frame interval of the first image.

[0381] As an implementation manner, the first image may be a short frame image, and the second image may be a long frame image.

[0382] Since the first image is a short-frame image, the short-frame image can balance the frame rate, maintain the texture edge and highlight area information of the scene, and has high definition and low signal-to-noise ratio. The second image is a long-frame image, which has a high signal-to-noise ratio, low definition, richer dark light area information, and richer color information. Therefore, it can be shown that the long-frame image and the short-frame image can complement each other to a certain extent. For example, the high-noise sharp texture of the short-frame image and the low-noise fuzzy texture of the long-frame image can constrain each other, which is conducive to detail recovery. The highlight area information of the short-frame image can repair the overexposed cutoff area of ​​the long-frame image. In addition, the color information of the long-frame image can correctly restore the color of the scene. Therefore, the short-frame image has the advantage of balancing the frame rate and / or high definition, and the long-frame image has the advantage of improving the signal-to-noise ratio. By using the long-frame image as a supplement to the short-frame image, the quality of the short-frame image is improved, thereby achieving the improvement of the video noise reduction effect under the condition of balancing the frame rate.

[0383] As an implementation manner, the exposure time of the second image may be greater than or equal to 100 ms.

[0384] Since the exposure time of the second image is greater than or equal to 100 ms, the exposure time of the second image can improve the signal-to-noise ratio, thereby improving the video noise reduction effect.

[0385] It should be noted that, for the description of the exposure time of the first image, the exposure time of the second image, the shooting time of the first image, the shooting time of the second image, the frame interval of the first image, the clarity of the first image and the clarity of the second image, please refer to the corresponding parts above and will not be repeated here.

[0386] As an implementation manner, the image brightness of the first image and the second image are aligned, and / or the image brightness of the first image and the second image are aligned and the field of view angles are aligned.

[0387] By aligning the image brightness of the first image and the second image, the brightness range of the first image and the second image is made the same, thereby improving the accuracy of image registration and image fusion, thereby improving the video noise reduction effect. By aligning the field of view of the first image and the second image, the accuracy of image registration and image fusion is improved, thereby improving the video noise reduction effect.

[0388] As an implementation method, the image brightness of the second image can be determined based on the original image brightness of the second image, the amount of light entering the second image, and the amount of light entering the first image. The amount of light entering the first image can be determined based on the exposure time and sensitivity of the first image. The amount of light entering the second image can be determined based on the exposure time and sensitivity of the second image.

[0389] For example, the image brightness of the second image may be determined according to the ratio and the original image brightness of the second image. The ratio may be the ratio between the amount of light entering the second image and the amount of light entering the first image.

[0390] By adjusting the image brightness, the image brightness ranges of multiple images are made the same, that is, the image brightness of multiple images are aligned, thereby improving the accuracy of image registration and image fusion, and further improving the video noise reduction effect.

[0391] It should be noted that, for the description of how to determine the image brightness of the second image, please refer to the corresponding part above, for example, the description of how to determine the image brightness of the cropped second image above is not repeated here.

[0392] As an implementation method, the target image may be obtained by image cropping. The target image may be an image captured by a target camera among the first image and the second image. The target camera may be a camera with a large field of view among multiple cameras. Optionally, the target camera may be a camera with a long exposure time among multiple cameras. Optionally, the target camera may be a camera with a large field of view and a long exposure time among multiple cameras. For example, the target camera may be the second camera described above.

[0393] By using image cropping to achieve the field of view alignment of multiple images, the accuracy of image registration and image fusion is improved, thereby improving the video noise reduction effect. In addition, the amount of data processing is reduced and the consumption of computing resources is reduced.

[0394] It should be noted that for the description of image cropping, please refer to the corresponding part above, which will not be repeated here.

[0395] As an implementation manner, the first image and the second image may be RAW images.

[0396] Since the RAW image is not processed by the image signal processor, the ISP processing will make the noise complicated. Therefore, the first image and the second image are RAW images, which reduces the difficulty of video noise reduction and improves the video noise reduction effect.

[0397] It should be noted that for the description of RAW images, please refer to the corresponding part above and will not be repeated here.

[0398] Regarding how to reduce the noise of the first image according to the second image to obtain a video frame, it can be achieved in the following manner.

[0399] As an implementation manner, the first image and the second image are fused to obtain a video frame.

[0400] Since the video frame is obtained by fusing the first image and the second image, the first image is taken by a camera that uses a shorter exposure time to balance the frame rate, and the second image is taken by a camera that uses a longer exposure time to improve the signal-to-noise ratio. Therefore, the video noise reduction effect is improved while balancing the frame rate.

[0401] Regarding how to fuse the first image and the second image to obtain a video frame, it can be achieved in the following manner.

[0402] As an implementation method, the first image and the registered second image are fused to obtain a video frame. The registered second image is obtained by registering the second image with the first image.

[0403] Since the video frame is obtained by fusing the first image and the registered second image, and the registered second image is obtained by registering the second image and the first image, image-level denoising is achieved. Since image-level denoising can make full use of pixel information, retain image detail information as much as possible, and has good interpretability, the video denoising effect is improved.

[0404] Regarding how to register the first image and the second image, it can be achieved in the following manner.

[0405] As an implementation, the registered second image may be obtained by registering the first image and the second image using an optical flow method. For example, the registered second image may be obtained by registering the second image and the first image using optical flow information. The optical flow information may be pixel-by-pixel relative displacement information between the first image and the second image.

[0406] The image registration between the second image and the first image is realized by using the optical flow method, thereby improving the efficiency of image registration.

[0407] As another implementation, the first image and the image converted by histogram are fused to obtain a video frame. The image converted by histogram can be obtained by transferring the color distribution of the second image to the first image based on a histogram conversion function.

[0408] Since there is no spatial misalignment problem between the image obtained by histogram conversion and the first image, the probability of ghosting effect caused by the movement of the camera or the object is reduced. In addition, since the image registration step is not performed, the video noise reduction process is simplified.

[0409] As another implementation, the tenth feature data and the eleventh feature data are fused to obtain a video frame. The tenth feature data and the eleventh feature data may be obtained based on the first feature data of the first image and the second feature data of the second image.

[0410] Regarding how to obtain the tenth characteristic data and the eleventh characteristic data, it can be achieved in the following manner.

[0411] As an implementation, the tenth feature data may be obtained by processing the first feature data of the first image and the second feature data of the second image using a cross attention mechanism. The first feature data may be used for query. The second feature data may be used for keys and values. The eleventh feature data may be obtained by processing the first feature data of the first image and the second feature data of the second image using a cross attention mechanism. The second feature data may be used for query. The first feature data may be used for keys and values.

[0412] Regarding how to use the cross attention mechanism to obtain the tenth feature data and the eleventh feature data, it can be achieved in the following way.

[0413] As an implementation, the tenth feature data may be obtained based on the seventh attention weight matrix and the sixth value matrix. The seventh attention weight matrix may be obtained based on the seventh query matrix and the seventh key matrix. The seventh query matrix may be obtained by performing a fourteenth linear transformation on the first feature data of the first image. The seventh key matrix may be obtained by performing a fifteenth linear transformation on the second feature data of the second image. The seventh attention weight matrix may be used to indicate the similarity between the second feature and the second feature data.

[0414] The eleventh feature data may be obtained based on the eighth attention weight matrix and the seventh value matrix. The eighth attention weight matrix may be obtained based on the eighth query matrix and the eighth key matrix. The eighth query matrix may be obtained by performing a sixteenth linear transformation on the second feature data of the second image. The eighth key matrix may be obtained by performing a seventeenth linear transformation on the first feature data of the first image. The eighth attention weight matrix may be used to indicate the similarity between the first feature and the second feature data.

[0415] Since the tenth feature data is obtained by processing the first feature data of the first image and the second feature data of the second image using the cross attention mechanism, the first feature data is used for query, and the second feature data is used for key and value, therefore, the tenth feature in the tenth feature data is obtained by fusing the second feature data with the first feature data. The eleventh feature data is obtained by processing the first feature data of the first image and the second feature data of the second image using the cross attention mechanism, the second feature data is used for query, and the first feature data can be used for key and value, therefore, the eleventh feature in the eleventh feature data is obtained by fusing the first feature data with the second feature data, thereby achieving more effective use of the first feature data and the second feature data. On this basis, the video frame is obtained by fusing the tenth feature data and the eleventh feature data, thereby improving the video noise reduction effect.

[0416] As another implementation, the first feature data of the first image and the registered second feature data are fused to obtain a video frame. The registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image.

[0417] Since the first feature data abstracts the first image while retaining the detail information of the first image, and the second feature data abstracts the second image while retaining the detail information of the second image, the first feature data and the second feature data have higher robustness and discriminability. On this basis, the video frame is obtained by fusing the first feature data of the first image and the registered second feature data, and the registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image, thereby improving the reliability and robustness of video denoising and improving the video denoising effect.

[0418] Regarding how to align the second feature data of the second image with the first feature data of the first image, it can be achieved in the following manner.

[0419] As an implementation, the registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image using a cross attention mechanism. The first feature data can be used for query. The second feature data can be used for key and value.

[0420] Since the cross-attention mechanism can mine the association between the second feature data and the first feature data, realize the interaction and matching of the first image and the second image at the feature level, thereby capturing the similarities and differences between the features, focusing on the similarities with high weights, suppressing the differences with low weights, and adaptively adjusting the weights, it has higher robustness and scalability. Therefore, by using the cross-attention mechanism to align the second feature data of the second image with the first feature data of the first image, the feature weight of the second feature in the second feature data can be adjusted. For example, a larger weight is assigned to the second feature in the second feature data that has a large similarity with the first feature data, and a smaller weight is assigned to the second feature in the second feature data that has a small similarity with the first feature data, thereby improving the accuracy of feature alignment and thus improving the video denoising effect.

[0421] As an implementation, the cross-attention mechanism may include at least one of the following: a dot product attention mechanism, a scaled dot product attention mechanism, an additive attention mechanism, a linear attention mechanism, or a sparse attention mechanism.

[0422] Since the video processing method described in the embodiment of the present application can be applied to resource-constrained electronic devices, such as terminal devices, the computational complexity of the cross-attention mechanism is proportional to the square of the number of feature data included in the feature data. Therefore, by adopting an optimization strategy based on optimizing the cross-attention mechanism, such as a linear attention mechanism and a sparse attention mechanism, the desired ability to reduce the computational complexity while maintaining or improving the accuracy of feature registration is achieved.

[0423] Regarding how to utilize the cross-attention mechanism to obtain the aligned second feature data, it can be achieved in the following manner.

[0424] As an implementation, the registered second feature data may be obtained according to the first query matrix, the first intermediate matrix and the first value matrix. The first value matrix may be obtained according to the first intermediate matrix, the first key matrix and the second value matrix.

[0425] The first query matrix may be obtained by performing a first linear transformation on the first feature data of the first image. The first intermediate matrix may be obtained based on the first query matrix. The first key matrix may be obtained by performing a second linear transformation on the second feature data of the second image. The second value matrix may be obtained by performing a third linear transformation on the second feature data.

[0426] Since the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix, and the first intermediate matrix is ​​used as a query, the first intermediate matrix can be used to aggregate information of the first key matrix and the second value matrix. Since the registered second feature data is obtained according to the first query matrix, the first intermediate matrix and the first value matrix, and the first query matrix is ​​used as a query and the first intermediate matrix is ​​used as a key, it is explained that the first intermediate matrix can pass the aggregated information to the first query matrix.

[0427] Since the first intermediate matrix is ​​obtained according to the first query matrix, the data volume of the first intermediate matrix can be less than the data volume of the first query matrix. Therefore, in the process of determining the first value matrix, the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix, and the first intermediate matrix is ​​used as a query. Therefore, compared with the first query matrix as a query, the data processing volume is reduced, the computing resource effect is reduced, and the first value matrix aggregates the information of the first key matrix and the second value matrix. In the process of determining the second feature data after registration, the second feature data after registration is obtained according to the first value matrix obtained by the first query matrix, the first intermediate matrix and the first value matrix. The first query matrix is ​​used as a query, the first intermediate matrix is ​​used as a key, and the first value matrix is ​​used as a value. Therefore, the first value matrix can pass the aggregated information to the first query matrix, thereby realizing global modeling. Among them, the first intermediate matrix can be used to aggregate the information of the first key matrix and the second value matrix, and pass the aggregated information to the first query matrix.

[0428] How to obtain the registered second feature data according to the first query matrix, the first intermediate matrix and the first value matrix can be achieved in the following manner.

[0429] As an implementation, the registered second feature data may be obtained according to the second intermediate matrix and the third value matrix. The second intermediate matrix may be obtained according to the first query matrix, the first intermediate matrix and the first value matrix. The third value matrix may be obtained by performing a depth-separable convolution on the first value matrix.

[0430] Since the third value matrix is ​​obtained by performing a first depth-wise separable convolution on the first value matrix, feature diversity is improved.

[0431] As another implementation, the registered second feature data may be obtained based on the third intermediate matrix and the second query matrix. The third intermediate matrix may be obtained based on the second key matrix and the fourth value matrix. The second query matrix and the second key matrix may be obtained by processing the third query matrix and the third key matrix respectively using a focusing function. The focusing function may be used to adjust the proximity between the third query matrix and the third key matrix. The third query matrix may be obtained by performing a fourth linear transformation on the first feature data of the first image. The third key matrix may be obtained by performing a fifth linear transformation on the second feature data of the second image. The third intermediate matrix may be obtained based on the second key matrix and the fourth value matrix. The fourth value matrix may be obtained by performing a sixth linear transformation on the second feature data of the second image.

[0432] Since the aggregation function can be used to adjust the proximity between the third query matrix and the third key matrix, that is, the aggregation function can be used to make similar elements in the third query matrix and the third key matrix closer and dissimilar elements farther away, it is achieved that the second feature with a large similarity between the aligned second feature data and the first feature data is given a larger weight, and the second feature with a small similarity between the second feature data and the first feature data is given a smaller weight, thereby improving the accuracy of feature alignment and further improving the video noise reduction effect. In addition, since the third intermediate matrix is ​​obtained based on the second key matrix and the fourth value matrix, the aligned second feature data is obtained based on the second query matrix and the third intermediate matrix, that is, the key and value are calculated first, and the result is then calculated with the query, thereby reducing the amount of data processing and reducing the consumption of computing resources.

[0433] It should be noted that for the description of this part, please refer to the corresponding part above and will not be repeated here.

[0434] Regarding how to fuse the first feature data of the first image and the registered second feature data to obtain a video frame, it can be achieved in the following manner.

[0435] As an implementation manner, the fused feature data is decoded to obtain a video frame. The fused feature data may be obtained by fusing the first feature of the first image and the registered second feature data.

[0436] Since the fused feature data is obtained by fusing the first feature data and the registered second feature data, the fused feature data can reflect the fusion characteristics of the first feature data and the registered second feature data, thereby making the information carried by the fused feature more comprehensive and accurate. On this basis, the fused feature data is decoded to obtain a video frame, thereby improving the quality of the video frame, thereby improving the video noise reduction effect.

[0437] Regarding how to obtain fused feature data, it can be achieved in the following ways.

[0438] As an implementation, the fused feature data may be obtained based on the first feature data and the third feature data. The third feature data may be obtained based on the fourth feature data and the registered second feature data. The fourth feature data may be obtained based on the first feature data and the registered second feature data.

[0439] Since the fused feature data is obtained based on the first feature data and the third feature data, the third feature data is obtained based on the fourth feature data and the registered second feature data, and the fourth feature data is obtained based on the first feature data and the registered second feature data, a relatively complete fusion of the first feature data and the second feature data is achieved, thereby making the information carried by the fused feature data more comprehensive and accurate.

[0440] It should be noted that for the description of this part, please refer to the corresponding part above and will not be repeated here.

[0441] According to an embodiment of the present application, in the case of video recording, the first image is taken by a camera that uses a shorter exposure time to balance the frame rate, and the second image is taken by a camera that uses a longer exposure time to improve the signal-to-noise ratio, and the shooting time of the first image and the shooting time of the second image overlap, which indicates that the contents of the first image and the second image are similar. Therefore, the second image is used to reduce the noise of the first image to obtain a video frame in the video, thereby achieving an improved video noise reduction effect while balancing the frame rate.

[0442] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0443] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0444] The video processing method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above video processing method.

[0445] Fig.14 This is a structural block diagram of a video processing device provided in an embodiment of the present application.

[0446] like Fig.14 As shown, the video processing device 1400 may include a recording module 1410 and a processing module 1420 .

[0447] The recording module 1410 is configured to perform video recording in response to a first operation.

[0448] Processing module 1420 is used to reduce noise of the first image according to the second image to obtain a video frame during video recording. The video frame is a frame in the recorded video. The first image and the second image are captured by multiple cameras. The shooting time of the first image and the shooting time of the second image overlap. The lens direction of the first camera is the same as the lens direction of the second camera. The first camera is a camera that captures the first image. The second camera is a camera that captures the second image. The exposure time of the first image is less than the exposure time of the second image.

[0449] The video processing method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can refer to the above related descriptions, which will not be repeated here.

[0450] The embodiment of the present application provides an electronic device, which includes a processor and a memory. The memory stores computer-executable instructions. The processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the video processing method in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here.

[0451] The embodiment of the present application provides a chip or a chip system. The chip or chip system includes at least one processor and a communication interface. The communication interface and the at least one processor are interconnected by a line. The at least one processor is used to run a computer program or instruction to execute the video processing method in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here.

[0452] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on an electronic device, the electronic device executes the video processing method in the above embodiment. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0453] In one possible implementation, a computer-readable medium may include RAM, ROM, Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (e.g., infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. Disks and optical disks as used herein include optical disks, laser disks, magnetic disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where magnetic disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0454] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0455] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0456] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A video processing method, characterized in that: include: In response to the first operation, performing video recording; During the video recording process, the first image is denoised according to the second image to obtain a video frame, wherein the video frame is a frame in the recorded video, the first image and the second image are shot by multiple cameras, the shooting time of the first image and the shooting time of the second image overlap, the lens direction of the first camera is the same as the lens direction of the second camera, the first camera is a camera for shooting the first image, the second camera is a camera for shooting the second image, and the exposure time of the first image is less than the exposure time of the second image.

2. The method according to claim 1, characterized in that The step of recording a video in response to the first operation includes: In response to the second operation, displaying a camera application interface, wherein the camera application interface includes a low-light recording control; in response to a third operation on the low-light recording control, displaying a low-light recording interface; or in response to detecting that the camera is in a low-light scene, displaying the low-light recording interface; or in response to the second operation, displaying the low-light recording interface, wherein the low-light recording interface includes a start control; In response to a first operation on the start control, video recording is performed.

3. The method according to claim 1 or 2, characterized in that: The clarity of the first image is higher than that of the second image, and / or the exposure time of the second image is greater than the frame interval of the first image, and the exposure time of the first image is less than or equal to the frame interval.

4. The method according to claim 1 or 2, characterized in that: The first image is a short frame image, and the second image is a long frame image.

5. The method according to claim 4, characterized in that The exposure time of the second image is greater than or equal to 100 ms.

6. The method according to claim 1 or 2, characterized in that: The step of reducing noise of the first image according to the second image to obtain a video frame includes: The first image and the second image are fused to obtain the video frame.

7. The method according to claim 6, characterized in that The fusing the first image and the second image to obtain the video frame includes: fusing the first image and the registered second image to obtain the video frame, wherein the registered second image is obtained by registering the second image and the first image; or The video frame is obtained by fusing the first feature data of the first image and the registered second feature data, wherein the registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image.

8. The method according to claim 7, characterized in that The registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image, and includes: The registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image using a cross-attention mechanism, wherein the first feature data is used for query and the second feature data is used for key and value.

9. The method according to claim 8, characterized in that The cross-attention mechanism includes at least one of the following: a dot product attention mechanism, a scaled dot product attention mechanism, an additive attention mechanism, a linear attention mechanism, or a sparse attention mechanism.

10. The method according to claim 8, characterized in that The registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image using a cross attention mechanism, and includes: The registered second feature data is obtained according to the first query matrix, the first intermediate matrix and the first value matrix, and the first value matrix is ​​obtained according to the first intermediate matrix, the first key matrix and the second value matrix; Among them, the first query matrix is ​​obtained by performing a first linear transformation on the first feature data of the first image, the first intermediate matrix is ​​obtained based on the first query matrix, the first key matrix is ​​obtained by performing a second linear transformation on the second feature data of the second image, and the second value matrix is ​​obtained by performing a third linear transformation on the second feature data.

11. The method according to claim 10, characterized in that The registered second feature data is obtained according to the first query matrix, the first intermediate matrix and the first value matrix, including: The registered second feature data is obtained based on a second intermediate matrix and a third value matrix, the second intermediate matrix is ​​obtained based on the first query matrix, the first intermediate matrix and the first value matrix, and the third value matrix is ​​obtained by performing a depth-separable convolution on the first value matrix.

12. The method according to claim 8, characterized in that The registered second feature data is obtained by registering the second feature data of the second image and the first feature data of the first image using a cross attention mechanism, and includes: The registered second feature data is obtained based on the third intermediate matrix and the second query matrix, the third intermediate matrix is ​​obtained based on the second key matrix and the fourth value matrix, the second query matrix and the second key matrix are obtained by processing the third query matrix and the third key matrix respectively using a focusing function, the focusing function is used to adjust the proximity between the third query matrix and the third key matrix, the third query matrix is ​​obtained by performing a fourth linear transformation on the first feature data of the first image, the third key matrix is ​​obtained by performing a fifth linear transformation on the second feature data of the second image, the third intermediate matrix is ​​obtained based on the second key matrix and the fourth value matrix, and the fourth value matrix is ​​obtained by performing a sixth linear transformation on the second feature data of the second image.

13. The method according to claim 7, characterized in that The fusing the first feature data of the first image and the registered second feature data to obtain the video frame includes: The fused feature data is decoded to obtain the video frame, wherein the fused feature data is obtained by fusing the first feature of the first image and the registered second feature data.

14. The method according to claim 13, characterized in that The fused feature data is obtained by fusing the first feature of the first image and the registered second feature data, and includes: The fused feature data is obtained based on the first feature data and the third feature data, the third feature data is obtained based on the fourth feature data and the registered second feature data, and the fourth feature data is obtained based on the first feature data and the registered second feature data.

15. The method according to claim 1 or 2, characterized in that: The image brightness of the first image and the second image are aligned, and / or the image brightness of the first image and the second image are aligned and the field of view angles are aligned.

16. The method according to claim 15, characterized in that The image brightness of the second image is determined according to the original image brightness of the second image, the amount of light entering the second image, and the amount of light entering the first image, the amount of light entering the first image is determined according to the exposure time and sensitivity of the first image, and the amount of light entering the second image is determined according to the exposure time and sensitivity of the second image; and / or The target image is obtained by image cropping, wherein the target image is an image among the first image and the second image taken by a target camera, the target camera is a camera with a large field of view angle among the multiple cameras, and / or the target camera is a camera with a long exposure time among the multiple cameras.

17. The method according to claim 1 or 2, characterized in that: The first image and the second image are RAW images.

18. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 17.

20. A chip system, characterized in that: It comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method according to any one of claims 1 to 17.

21. A computer program product, characterized in that The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 17.

Images