Video splicing method, device and electronic device

Abstract

The present application provides a video stitching method, apparatus, and electronic device. When obtaining K currently to-be-stitched images from different perspectives, for each currently to-be-stitched image, the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on the current color compensation coefficient of the currently to-be-stitched image and the historical color compensation coefficient. In this way, the influence of the color information of the historical to-be-stitched images on the color compensation coefficient of the currently to-be-stitched image is fully considered; further, the currently to-be-stitched image is color-corrected according to the target color compensation coefficient of the currently to-be-stitched image to obtain the target color image corresponding to the currently to-be-stitched image, and the target color images corresponding to the K currently to-be-stitched images are stitched to obtain the target stitched image for constructing the target video, achieving color balance in the video stitching process.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and particularly to a video stitching method, apparatus, and electronic device. Background Art

[0002] In an actual scenario, multiple frames of images collected by multiple scattered video capture devices are usually stitched together to provide a larger, wider, and more comprehensive view. During the video stitching process, in order to avoid misalignment at the stitching seam, image alignment processing is usually performed on the current image to be stitched. However, during the video stitching process, due to reasons such as changes in scene illumination and differences between photosensitive devices, even after image alignment processing, problems such as uneven color in the stitched image and unnatural image transition still occur. Summary of the Invention

[0003] In view of this, the present application provides a video stitching method, apparatus, and electronic device to achieve color balance during the video stitching process.

[0004] The technical solutions provided by the present application are as follows:

[0005] According to an embodiment of the first aspect of the present application, a video stitching method is provided, and the method includes:

[0006] Obtain K current images to be stitched with different perspectives, where K is not less than 2;

[0007] For each current image to be stitched, determine the target color compensation coefficient of the current image to be stitched based on the historical color compensation coefficient of the corresponding historical image to be stitched; wherein, the historical image to be stitched and the current image to be stitched are images in the same video and at the same perspective; wherein, the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient;

[0008] Perform color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain the target color image corresponding to the current image to be stitched;

[0009] Stitch the target color images corresponding to the K current images to be stitched to obtain a target stitched image; the target stitched image is used to form a target video.

[0010] Optionally, the determining the historical color compensation coefficient as the target color compensation coefficient includes:

[0011] If the difference between the current color distribution of each current image to be stitched and the historical color distribution of the historical image to be stitched is within a preset range, the historical color compensation coefficient of the historical image to be stitched is used as the target color compensation coefficient of the current image to be stitched; wherein, the historical image to be stitched is the previous frame of the current image to be stitched.

[0012] Optionally, determining the target color compensation coefficient based on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient includes:

[0013] If the difference between the current color distribution of any current image to be stitched and the historical color distribution of the historical image to be stitched is not within the preset range, a weighted operation is performed on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient of the historical image to be stitched to obtain the target color compensation coefficient; wherein, the historical image to be stitched is the previous M frames of the current image to be stitched, and M is greater than or equal to 1.

[0014] Optionally, the K current images to be stitched include a reference image and at least one non-reference image, and the reference image is any one of the K current images to be stitched that is selected; the current color compensation coefficient of the current image to be stitched is determined by the following method:

[0015] For the reference image, obtain the configured default color compensation coefficient, and determine the default color compensation coefficient as the current color compensation coefficient of the reference image;

[0016] For each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image.

[0017] Optionally, the current color distribution of the non-reference image and the current color distribution of the reference image include the first-channel color distribution, the second-channel color distribution, and the third-channel color distribution; determining the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-reference image and the current color distribution of the reference image includes:

[0018] Determine the current color compensation coefficient corresponding to each channel of the non-reference image relative to the reference image according to the following formula:

[0019]

[0020]

[0021]

[0022] Among them, the represents the current color compensation coefficient corresponding to the nth non-reference image in the first channel, and the represents the color distribution of the first channel of the nth non-reference image, and the represents the color distribution of the first channel of the reference image; the represents the current color compensation coefficient corresponding to the nth non-reference image in the second channel, and the represents the color distribution of the second channel of the nth non-reference image, and the represents the color distribution of the second channel of the reference image; the represents the current color compensation coefficient corresponding to the nth non-reference image in the third channel, and the represents the color distribution of the third channel of the nth non-reference image, and the represents the color distribution of the third channel of the reference image.

[0023] Optionally, for each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, determining the current color compensation coefficient of the non-reference image relative to the reference image includes:

[0024] For each current image to be stitched among the K current images to be stitched, determine the non-moving area in the current image to be stitched according to the current image to be stitched and the historical image to be stitched corresponding to the current image to be stitched; the non-moving area refers to the background area in the current image to be stitched except for moving objects;

[0025] For each non-reference image, based on the current color distribution of the non-moving area in the non-reference image and the current color distribution of the non-moving area in the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image.

[0026] Optionally, for each current image to be stitched among the K current images to be stitched, determining the non-moving area in the current image to be stitched according to the current image to be stitched and the historical image to be stitched corresponding to the current image to be stitched includes:

[0027] For each current image to be stitched among the K current images to be stitched, determine the velocity vector of each pixel point in the current image to be stitched according to the current image to be stitched and the historical image to be stitched corresponding to the current image to be stitched, and the velocity vector includes the change in motion rate and the change in motion direction of the pixel point between the acquisition times of the historical image to be stitched and the current image to be stitched;

[0028] For each pixel in the current image to be stitched, if it is determined that the change in the motion rate of the pixel is less than a first threshold and the change in the motion direction of the pixel is less than a second threshold, then the pixel is used as a candidate pixel;

[0029] Based on the set of all candidate pixels, determine the non-moving area in the current image to be stitched.

[0030] According to an embodiment of the second aspect of the present application, there is provided a video stitching device, the device includes:

[0031] An acquisition module, configured to acquire K current images to be stitched with different perspectives, where K is not less than 2;

[0032] A determination module, configured to, for each current image to be stitched, determine a target color compensation coefficient of the current image to be stitched based on a historical color compensation coefficient of a historical image to be stitched corresponding to the current image to be stitched; where the historical image to be stitched and the current image to be stitched are images in the same video and at the same perspective; where the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on a current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient;

[0033] A correction module, configured to perform color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain a target color image corresponding to the current image to be stitched;

[0034] A stitching module, configured to stitch the target color images corresponding to the K current images to be stitched to obtain a target stitched image; the target stitched image is used to form a target video.

[0035] Optionally, the determination module is specifically configured to:

[0036] If the difference between the current color distribution of each current image to be stitched and the historical color distribution of the historical image to be stitched is within a preset range, then use the historical color compensation coefficient of the historical image to be stitched as the target color compensation coefficient of the current image to be stitched; where the historical image to be stitched is the previous frame of the historical image to be stitched of the current image to be stitched;

[0037] And / or, the determination module is specifically configured to:

[0038] If the difference between the current color distribution of any current image to be stitched and the historical color distribution of the historical image to be stitched is not within the preset range, a weighted operation is performed on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient of the historical image to be stitched to obtain the target color compensation coefficient; wherein, the historical image to be stitched is the first M historical images to be stitched before the current image to be stitched, and M is greater than or equal to 1;

[0039] And / or, the K current images to be stitched include a reference image and at least one non-reference image, and the reference image is any one of the K current images to be stitched that is selected; the determining module is further configured to:

[0040] For the reference image, obtain the configured default color compensation coefficient, and determine the default color compensation coefficient as the current color compensation coefficient of the reference image;

[0041] For each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image;

[0042] And / or, the current color distribution of the non-reference image and the current color distribution of the reference image include a first-channel color distribution, a second-channel color distribution, and a third-channel color distribution; the determining module is specifically configured to:

[0043] Determine the current color compensation coefficient corresponding to each channel of the non-reference image relative to the reference image according to the following formula:

[0044]

[0045]

[0046]

[0047] Wherein, the represents the current color compensation coefficient corresponding to the first channel of the nth non-reference image, the represents the first-channel color distribution of the nth non-reference image, and the represents the first-channel color distribution of the reference image; the represents the current color compensation coefficient corresponding to the second channel of the nth non-reference image, the represents the second-channel color distribution of the nth non-reference image, and the represents the second-channel color distribution of the reference image; the represents the current color compensation coefficient corresponding to the third channel of the nth non-reference image, and the represents the color distribution of the third channel of the nth non-reference image, and the represents the color distribution of the third channel of the reference image;

[0048] And / or, the determining module is specifically configured to:

[0049] For each current image to be stitched among the K current images to be stitched, determine the non-moving area in the current image to be stitched according to the current image to be stitched and the corresponding historical image to be stitched; the non-moving area refers to the background area in the current image to be stitched except for moving objects;

[0050] For each non-reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-moving area in the non-reference image and the current color distribution of the non-moving area in the reference image;

[0051] And / or, the determining module is specifically configured to:

[0052] For each current image to be stitched among the K current images to be stitched, determine the velocity vector of each pixel point in the current image to be stitched according to the current image to be stitched and the corresponding historical image to be stitched, where the velocity vector includes the change in motion rate and the change in motion direction of the pixel point between the acquisition times of the historical image to be stitched and the current image to be stitched;

[0053] For each pixel point in the current image to be stitched, if it is determined that the change in motion rate of the pixel point is less than the first threshold and the change in motion direction of the pixel point is less than the second threshold, then use the pixel point as a candidate pixel point;

[0054] Determine the non-moving area in the current image to be stitched based on the set of all candidate pixel points.

[0055] According to an embodiment of the third aspect of the present application, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the method as described in the first aspect.

[0056] According to an embodiment of the fourth aspect of the present application, there is provided a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the method as described in the first aspect.

[0057] According to an embodiment of the fifth aspect of the present application, a machine-readable storage medium is provided. The machine-readable storage medium stores machine-executable instructions that can be executed by a processor. Wherein, the processor is configured to execute the machine-executable instructions to implement the method as described in the first aspect.

[0058] As can be seen from the above technical solutions, in the present application, when K current images to be stitched with different perspectives are obtained, for each current image to be stitched, based on the historical color compensation coefficient of the historical image to be stitched corresponding to the current image to be stitched, the target color compensation coefficient of the current image to be stitched is determined. Specifically, the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient. In this way, the influence of the color information of the historical image to be stitched on the color compensation coefficient of the current image to be stitched is fully considered. Further, the current image to be stitched is color-corrected according to the target color compensation coefficient of the current image to be stitched to obtain the target color image corresponding to the current image to be stitched, and the target color images corresponding to the K current images to be stitched are stitched to obtain the target stitched image for constituting the target video, realizing color balance in the video stitching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0060] Figure 1 It is a flowchart of the video stitching method provided by the embodiment of the present application;

[0061] Figure 2 It is a schematic diagram of the overall process provided by the embodiment of the present application;

[0062] Figure 3 It is a schematic diagram of the specific process of the spatio-temporal color balance stage provided by the embodiment of the present application;

[0063] Figure 4 It is a schematic diagram of the specific process of the image alignment stage provided by the embodiment of the present application;

[0064] Figure 5 It is a schematic diagram of the specific process of the transition zone processing stage provided by the embodiment of the present application;

[0065] Figure 6A It is an example diagram of spatial color balance provided by the embodiment of the present application;

[0066] Figure 6B It is an example diagram of spatio-temporal color balance provided by the embodiment of the present application;

[0067] Figure 7 Schematic diagram of a structure of an electronic device provided by an embodiment of the present application;

[0068] Figure 8 Structural diagram of a video splicing device provided by an embodiment of the present application. Detailed implementation manners

[0069] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned objects, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0070] In an actual scenario, multiple frames of images collected by multiple scattered video capture devices at the same moment are usually spliced to obtain a target spliced image for constructing a target video, so as to provide a larger, wider, and more comprehensive perspective. During the video splicing process, in order to avoid misalignment at the splicing seam, image alignment processing is usually performed on the currently to-be-spliced image. However, during the video splicing process, due to reasons such as changes in scene illumination and differences between photosensitive devices, even if the image alignment processing is performed, problems such as uneven color of the spliced image and unnatural image transition still occur.

[0071] It can be seen that during the video splicing process, in addition to aligning the images, it is also necessary to consider the color balance of the spliced image to make the image transition natural. For example, color balance in the spatial dimension is performed on each path of images at each moment. Performing color balance in the spatial dimension means performing color balance on multiple currently to-be-spliced images collected at the same moment, so that the color transition of the spliced image obtained after splicing is relatively natural.

[0072] However, although the spliced image obtained by splicing multiple to-be-spliced videos with color balance in the spatial dimension is color balanced in the current frame, it does not consider the color jumps in the time dimension (between different frames) due to reasons such as changes in illumination or the passing of moving objects. That is, each frame in the spliced video stream with color balance processing in the spatial dimension is color balanced when viewed alone, but there may be obvious color jumps between the front and rear frames, lacking temporal smoothness.

[0073] Please refer to Figure 1 , Figure 1 Flow chart of a video splicing method provided by an embodiment of the present application.

[0074] As Figure 1 shown, the method may include the following steps:

[0075] Step 101, obtain K currently to-be-spliced images from different perspectives.

[0076] In this embodiment, the K current images to be stitched refer to multiple images captured from different perspectives at the current moment, where K is not less than 2.

[0077] As an embodiment, the K current images to be stitched can be target scene images at the current moment captured by multiple image acquisition devices from different perspectives, or K aligned images obtained after image alignment of the target scene images at the current moment captured by multiple image acquisition devices from different perspectives. This application does not limit this.

[0078] So far, the description of step 101 ends. Next, execute step 102.

[0079] Step 102: For each current image to be stitched, determine the target color compensation coefficient of the current image to be stitched based on the historical color compensation coefficient of the corresponding historical image to be stitched.

[0080] In this embodiment, the historical image to be stitched and the current image to be stitched are images in the same video from the same perspective. For example, the current images to be stitched at different times captured by the same imaging device from the same perspective are the historical images to be stitched of the current image to be stitched from the same perspective.

[0081] In this embodiment, the historical image to be stitched can be one or multiple. Each historical image to be stitched corresponds to a historical color compensation coefficient.

[0082] In this embodiment, the method for determining the target color compensation coefficient of the current image to be stitched based on the historical color compensation coefficient of the corresponding historical image to be stitched includes the following two cases:

[0083] Determine the historical color compensation coefficient as the target color compensation coefficient, or determine the target color compensation coefficient based on the current color compensation coefficient and the historical color compensation coefficient of the current image to be stitched.

[0084] The following describes the above two cases separately.

[0085] (1) Determine the historical color compensation coefficient as the target color compensation coefficient.

[0086] As an embodiment, the specific method for determining the historical color compensation coefficient as the target color compensation coefficient can include:

[0087] If the difference between the current color distribution of each currently to-be-stitched image and the historical color distribution of the historical to-be-stitched image is within a preset range, then the historical color compensation coefficient of the historical to-be-stitched image is used as the target color compensation coefficient of the currently to-be-stitched image; wherein, the historical to-be-stitched image is the previous-frame historical to-be-stitched image of the currently to-be-stitched image.

[0088] In this embodiment, the historical to-be-stitched image can be the previous-frame historical to-be-stitched image of the currently to-be-stitched image. Specifically, if the number of historical to-be-stitched images corresponding to the currently to-be-stitched image is one, then directly determine the difference between the historical color distribution of the historical to-be-stitched image and the current color distribution of the currently to-be-stitched image; if the number of historical to-be-stitched images corresponding to the currently to-be-stitched image is multiple, then select the historical color distribution of the historical to-be-stitched image with the closest image acquisition time to the currently to-be-stitched image (i.e., the previous-frame historical to-be-stitched image of the currently to-be-stitched image) and perform a difference operation with the current color distribution of the currently to-be-stitched image.

[0089] In this embodiment, if the difference between the current color distribution of each currently to-be-stitched image among the K currently to-be-stitched images with different perspectives and the historical color distribution of the historical to-be-stitched image corresponding to the currently to-be-stitched image is within a preset range, it indicates that at this time, the color distributions of the currently to-be-stitched image and the corresponding historical to-be-stitched image are relatively similar, that is, there is no obvious color jump in time series.

[0090] In this case, the historical color compensation coefficient of the previous-frame historical to-be-stitched image of the currently to-be-stitched image can be directly used as the target color compensation coefficient of the currently to-be-stitched image, that is, the historical color compensation coefficient of the previous-frame historical to-be-stitched image is followed, and there is no need to recalculate the target color compensation coefficient of the currently to-be-stitched image. In this way, computing resources can be saved and power consumption can be reduced.

[0091] Specifically, if the number of historical to-be-stitched images corresponding to each currently to-be-stitched image is one, then the historical color compensation coefficient of the historical to-be-stitched image corresponding to each currently to-be-stitched image can be directly used as the target color compensation coefficient of the currently to-be-stitched image.

[0092] If the number of historical to-be-stitched images corresponding to each currently to-be-stitched image is multiple, then the historical color compensation coefficient of the historical to-be-stitched image with the closest image acquisition time to the currently to-be-stitched image (i.e., the previous-frame historical to-be-stitched image of the currently to-be-stitched image) is used as the target color compensation coefficient of the currently to-be-stitched image.

[0093] So far, the description of case (1), that is, determining the historical color compensation coefficient as the target color compensation coefficient, is completed.

[0094] (2) Determine the target color compensation coefficient based on the current color compensation coefficient and the historical color compensation coefficient of the current image to be stitched.

[0095] As an embodiment, the method for determining the target color compensation coefficient based on the current color compensation coefficient and the historical color compensation coefficient of the current image to be stitched may include:

[0096] If the difference between the current color distribution of any current image to be stitched and the historical color distribution of the historical image to be stitched is not within the preset range, perform a weighted operation on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient of the historical image to be stitched to obtain the target color compensation coefficient; wherein, the historical image to be stitched is the first M frames of historical images to be stitched of the current image to be stitched, and M is greater than or equal to 1.

[0097] In this embodiment, if the difference between the current color distribution of any current image to be stitched and the historical color distribution of the historical image to be stitched is not within the preset range, it indicates that the color distribution of the current image to be stitched and the corresponding historical image to be stitched is quite different at this time, that is, there is an obvious color jump in the time series.

[0098] In this case, a weighted operation can be performed on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient of the historical image to be stitched to obtain the target color compensation coefficient. In this way, when determining the target color compensation coefficient of the current image to be stitched, the influence of the color information of the historical image to be stitched is fully considered, so that the target stitched image determined by the target color compensation coefficient can have a more balanced color in the time series and enhance the visual experience of the video.

[0099] In this case, different from case (1), the historical image to be stitched can be the first M frames of historical images to be stitched of the current image to be stitched, and M is greater than or equal to 1.

[0100] Since the historical image to be stitched can be one or more, and the obtained historical color compensation coefficients are also one or more, at this time, according to the weights set for each historical color compensation coefficient and the current color compensation coefficient of the current image to be stitched, a weighted operation can be performed on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficients of the historical images to be stitched to obtain the target color compensation coefficient.

[0101] The specific weighted operation method will be described below and will not be elaborated here.

[0102] So far, the description of case (2), that is, determining the target color compensation coefficient based on the current color compensation coefficient and the historical color compensation coefficient of the current image to be stitched, is completed.

[0103] It should be noted that the current color compensation coefficient of the current image to be stitched in the above case (2) can be determined by the following method.

[0104] As an embodiment, the K current images to be stitched include a reference image and at least one non-reference image, and the reference image is any one of the K current images to be stitched selected; the current color compensation coefficient of the current image to be stitched is determined by the following method:

[0105] For the reference image, obtain the configured default color compensation coefficient, and determine the default color compensation coefficient as the current color compensation coefficient of the reference image;

[0106] For each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image.

[0107] In this embodiment, since the color compensation coefficient is a coefficient used to adjust the color effect of the current image to be unified with the color effect of the reference image, therefore, any one of the K current images to be stitched is selected as the reference image, and the remaining images are determined as non-reference images.

[0108] For the reference image, as the color standard of the non-reference image, there is no need to perform color compensation in the spatial dimension of the K current images to be stitched collected at the current moment, and its default color compensation coefficient can be 1 (that is, the compensated image is itself), and the default color compensation coefficient can be directly determined as the current color compensation coefficient of the reference image, and the present application does not limit the setting of the default color compensation coefficient.

[0109] For each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, the current color compensation coefficient of the non-reference image relative to the reference image can be determined.

[0110] In this embodiment, the current color distribution of the non-reference image and the current color distribution of the reference image include the first-channel color distribution, the second-channel color distribution, and the third-channel color distribution; based on the current color distribution of the non-reference image and the current color distribution of the reference image, determining the current color compensation coefficient of the non-reference image relative to the reference image includes:

[0111] Determine the current color compensation coefficient corresponding to each channel of the non-reference image relative to the reference image according to the following formula:

[0112]

[0113]

[0114]

[0115] Among them, represents the current color compensation coefficient corresponding to the nth non-reference image in the first channel, represents the color distribution of the first channel of the nth non-reference image, represents the color distribution of the first channel of the reference image; represents the current color compensation coefficient corresponding to the nth non-reference image in the second channel, represents the color distribution of the second channel of the nth non-reference image, represents the color distribution of the second channel of the reference image; represents the current color compensation coefficient corresponding to the nth non-reference image in the third channel, represents the color distribution of the third channel of the nth non-reference image, represents the color distribution of the third channel of the reference image.

[0116] Next, the method for determining the color distribution of each channel will be introduced first.

[0117] As an example, the current color distribution of the current image to be stitched can be determined by statistically calculating the mean values of the YUV three channels.

[0118] Among them, the YUV three channels is a color representation model in image and video processing, which can decompose the colors in an image into three components: Y, U, and V.

[0119] Specifically, the Y channel represents the brightness of the image (the intensity of light), and can be obtained by weighted calculation based on the red, green, and blue color channels of the image.

[0120] The U channel and the V channel respectively represent chrominance (the component of color), which determine the hue and color saturation of the image.

[0121] The U channel represents the difference between the blue part of the color and the brightness, and is usually a combination of blue and green.

[0122] The V channel represents the difference between the red part of the color and the brightness, and is usually a combination of red and green.

[0123] Exemplarily, for the input current image to be stitched, the calculation method of the color distribution can be: for input images, statistically calculate the mean values of the YUV three channels of different images,

[0124]

[0125]

[0126]

[0127] Among them, represents the Y-channel color distribution of the i-th image, represents the Y-channel data of the i-th image, represents the U-channel color distribution of the i-th image, represents the U-channel data of the i-th image, represents the V-channel color distribution of the i-th image, represents the V-channel data of the i-th image, represents calculating the average value.

[0128] Obtain: , , , , , , …, , , . That is, the average values of the YUV three channels of each current image to be stitched.

[0129] As an example, the current color distribution of the current image to be stitched can also be described by other methods such as RGB, HSV and other parameters or more complex models, and the present application does not limit this.

[0130] As an example, for the color distribution of multiple channels, each channel corresponds to a color compensation coefficient. Specifically, the calculation method of the color compensation coefficient may include: taking any one of the K current images to be stitched as the reference image, calculating the color compensation coefficients of the remaining non-reference images relative to the reference image. Taking the first image to be stitched as the reference image as an example, the color compensation coefficient of the th image is calculated as follows:

[0131]

[0132]

[0133]

[0134] Among them, represents the current color compensation coefficient corresponding to the Y channel of the K-th image, represents the Y-channel color distribution of the K-th image, represents the Y-channel color distribution of the first image (i.e., the reference image); represents the current color compensation coefficient corresponding to the U channel of the K-th image, represents the U-channel color distribution of the K-th image, Represents the U-channel color distribution of the first image (i.e., the reference image); Represents the current color compensation coefficient corresponding to the V-channel of the Kth image, Represents the V-channel color distribution of the Kth image, Represents the V-channel color distribution of the first image (i.e., the reference image).

[0135] In this embodiment, for the case where the current image to be stitched corresponds to the first frame image in the target video, the historical image to be stitched corresponding to the current image to be stitched is the current image to be stitched itself.

[0136] For the reference image itself, it is pre-configured with a default color compensation coefficient. The current color compensation coefficient of the reference image is this default color compensation coefficient, and this default color compensation coefficient can be 1. This application does not limit this.

[0137] Next, a method for obtaining the target color compensation coefficient through weighted operation will be briefly introduced.

[0138] As an embodiment, the target color compensation coefficient of each current image to be stitched can be determined according to the current color compensation coefficient of each current image to be stitched and the historical color compensation coefficient of the historical image to be stitched corresponding to this current image to be stitched.

[0139] Specifically, for each current image to be stitched, a weighted average operation is performed on the current color compensation coefficient and the historical color compensation coefficient of this current image to be stitched according to a preset weight coefficient to obtain the target color compensation coefficient of this current image to be stitched.

[0140] In this embodiment, before executing the video stitching method proposed in this application, the number of recorded historical images to be stitched can be determined first. For example, the latest three historical images to be stitched are retained as the calculation basis for the target color compensation coefficient.

[0141] In this embodiment, if only the historical color compensation coefficient of one historical image to be stitched of the current image to be stitched is recorded, then a weighted average operation can be performed on this historical color compensation coefficient and the current color compensation coefficient of this current image to be stitched according to the preset weight coefficient of the historical color compensation coefficient (such as 0.4) and the weight coefficient of the current color compensation coefficient (such as 0.6) to obtain the target color compensation coefficient of this current image to be stitched. Exemplarily, the sum of the weight coefficients of the historical color compensation coefficient and the current color compensation coefficient can be 1.

[0142] In this embodiment, if the historical color compensation coefficients of multiple historical images to be stitched of the current image to be stitched are recorded, weighted operations can be performed according to the weight coefficients respectively configured for the multiple historical color compensation coefficients and the current color compensation coefficient. For example, if the latest three historical images to be stitched are retained, weighted average operations can be performed on the historical color compensation coefficients and the current color compensation coefficient of the current image to be stitched according to the weight coefficients respectively corresponding to the historical color compensation coefficients of the latest three historical images to be stitched preset, and the weight coefficient of the current color compensation coefficient, to obtain the target color compensation coefficient of the current image to be stitched. Exemplarily, the sum of the weight coefficients of each historical color compensation coefficient and the current color compensation coefficient can be 1.

[0143] As an embodiment, the calculation method of the target color compensation coefficient of the current image to be stitched can be calculated according to the following formula:

[0144]

[0145] Wherein, represents the target color compensation coefficient of the t-th current image to be stitched (taking one channel as an example here, and the other channels are in the same way); represents the product of the current color compensation coefficient of the t-th current image to be stitched and the corresponding weight coefficient; represents the product of the historical color compensation coefficient of the r-th historical image to be stitched corresponding to the current image to be stitched and the corresponding weight coefficient; represents the sum of the products of the historical color compensation coefficients of s historical images to be stitched corresponding to the current image to be stitched and the corresponding weight coefficients, and s represents the number of historical images to be stitched participating in the operation of the target color compensation coefficient.

[0146] So far, the description of the weighted operation method ends.

[0147] In this embodiment, for each non-reference image, the method for determining the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-reference image and the current color distribution of the reference image can include:

[0148] For each of the K current images to be stitched, according to the current image to be stitched and the historical images to be stitched corresponding to the current image to be stitched, determine the non-moving area in the current image to be stitched; the non-moving area refers to the background area in the current image to be stitched except for the moving object;

[0149] For each non-reference image, based on the current color distribution of the non-moving area in the non-reference image and the current color distribution of the non-moving area in the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image.

[0150] In this embodiment, considering that when there are moving objects in the current image to be stitched, it will greatly affect the color distribution difference between the current image to be stitched and the corresponding historical image to be stitched. If the moving objects are taken into account, it will lead to a relatively large difference in the color distribution at the current moment compared with the previous one, causing a color mutation in the time sequence of the stitched video stream.

[0151] Therefore, the non-moving area in the current image to be stitched can be determined according to the current image to be stitched and the corresponding historical image to be stitched, and the current color compensation coefficient can be further determined according to the current color distribution of the non-moving area. The non-moving area refers to the background area in the current image to be stitched except for the moving objects.

[0152] As an embodiment, the method for specifically determining the non-moving area may include:

[0153] For each of the K current images to be stitched, according to the current image to be stitched and the corresponding historical image to be stitched, determine the velocity vector of each pixel point in the current image to be stitched. The velocity vector includes the change amount of the motion rate and the change amount of the motion direction of the pixel point between the acquisition times of the historical image to be stitched and the current image to be stitched;

[0154] For each pixel point in the current image to be stitched, if it is determined that the change amount of the motion rate of the pixel point is less than the first threshold and the change amount of the motion direction of the pixel point is less than the second threshold, then the pixel point is used as a candidate pixel point;

[0155] Based on the set of all candidate pixel points, determine the non-moving area in the current image to be stitched.

[0156] As an embodiment, when calculating the current color distribution, it is necessary to remove the moving object area that may cause an impact (the non-moving area after removing the moving object area can be denoted as Mask), and determine the target color compensation coefficient only according to the color distribution of the non-moving area.

[0157]

[0158]

[0159]

[0160] Among them, represents the Y-channel color distribution of the i-th image, represents the Y-channel data of the i-th image, represents the U-channel color distribution of the i-th image, represents the U-channel data of the i-th image, Represents the V-channel color distribution of the i-th image, Represents the V-channel data of the i-th image, Represents the operation of calculating the average value, and Mask represents the non-moving area.

[0161] Thus, the description of step 102 ends, and step 103 is executed below.

[0162] Step 103: Perform color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched, so as to obtain the target color image corresponding to the current image to be stitched.

[0163] Specifically, the method of performing color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain the target color image corresponding to the current image to be stitched may include:

[0164] Determine the target color distribution of the current image to be stitched according to the target color compensation coefficient of the current image to be stitched and the current color distribution of the current image to be stitched, and use the image corresponding to the target color distribution as the target color image.

[0165] In this embodiment, taking the K-th image as an example, the method of determining the target color distribution may be:

[0166]

[0167]

[0168]

[0169] Among them, Represents the target color compensation coefficient corresponding to the Y channel of the K-th image, Represents the target color distribution of the Y channel of the K-th image, Represents the current color distribution of the Y channel of the K-th image; Represents the target color compensation coefficient corresponding to the U channel of the K-th image, Represents the target color distribution of the U channel of the K-th image, Represents the current color distribution of the U channel of the K-th image; Represents the target color compensation coefficient corresponding to the V channel of the K-th image, Represents the target color distribution of the V channel of the K-th image, Represents the current color distribution of the V channel of the K-th image.

[0170] Thus, the description of step 103 ends, and step 104 is executed below.

[0171] Step 104: Stitch the target color images corresponding to the K current images to be stitched to obtain a target stitched image.

[0172] The target stitched image is used to form a target video.

[0173] In this embodiment, for each current image to be stitched, according to Step 103, the target color image of the current image to be stitched after color correction is obtained. Further, the target color images corresponding to the K current images to be stitched can be stitched to obtain a target stitched image for forming a target video.

[0174] So far, the description of Step 104 ends.

[0175] In this embodiment, the method for determining the target color compensation coefficient corresponding to each current image to be stitched can also be determined based on a deep learning network. For example, the current image to be stitched and the historical image to be stitched corresponding to the current image to be stitched are input into a trained deep learning network, and the target color compensation coefficient corresponding to the current image to be stitched is directly obtained end-to-end. Further, the step of performing color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain the target color image corresponding to the current image to be stitched is executed.

[0176] Alternatively, the target color image corresponding to the current image to be stitched can be directly obtained through a trained deep learning network. For example, the current image to be stitched and the historical image to be stitched corresponding to the current image to be stitched are input into a trained deep learning network, and the target color image corresponding to the current image to be stitched is directly obtained end-to-end. The present application does not limit this.

[0177] So far, the description of Figure 1 the video stitching method ends.

[0178] In the case where the present application obtains K current images to be stitched from different perspectives, for each current image to be stitched, the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on the current color compensation coefficient and the historical color compensation coefficient of the current image to be stitched. In this way, the influence of the color information of the historical image to be stitched on the color compensation coefficient of the current image to be stitched is fully considered. Further, color correction is performed on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain the target color image corresponding to the current image to be stitched, and the target color images corresponding to the K current images to be stitched are stitched to obtain a target stitched image for forming a target video, realizing color balance in the video stitching process.

[0179] Next, in combination with Figure 2Introduce the overall process of the video stitching method.

[0180] Please refer to Figure 2 , Figure 2 which is the overall process schematic diagram provided by the embodiment of the present application.

[0181] As Figure 2 shown, this solution proposes a video stitching method that can synchronously achieve spatio-temporal color balance.

[0182] Specifically, first obtain K current images to be stitched at the current moment. Secondly, perform stitching processing on the K current images to be stitched to obtain a stitched image. This stitching processing can be divided into three stages, namely the image alignment stage, the spatio-temporal color balance stage, and the transition zone processing stage.

[0183] Through the above three stages, a stitched image obtained by stitching K current images to be stitched can be obtained. If it is found that after completing the stitching of the current images to be stitched, there are still current images to be stitched at the next moment, enter the processing of the next moment, obtain the current images to be stitched corresponding to the next moment, until there are no current images to be stitched at the next moment, and determine that the stitched images of all frames of the video to be stitched have been obtained.

[0184] After obtaining all the stitched images, use the stitched images as each frame in the video to be stitched, and perform video stitching to obtain the target video.

[0185] In this embodiment, the overall video stitching process can be carried out online. For example, during video playback or recording, the stitching operation is performed so that the video data is processed in real time and the stitching result is immediately output or displayed.

[0186] In this embodiment, the overall video stitching process can also be carried out offline. For example, the stitching operation is performed after the video recording is completed. The present application does not limit this.

[0187] The following will introduce each stage separately.

[0188] Please refer to Figure 3 , Figure 3 which is the specific process schematic diagram of the spatio-temporal color balance stage provided by the embodiment of the present application.

[0189] As Figure 3 shown, the spatio-temporal color balance stage realizes the spatio-temporal color balance of the video stream to be stitched.

[0190] Specifically, obtain k current images to be stitched. Here, the current images to be stitched can be original images or already aligned images obtained through the image alignment stage.

[0191] First, determine the target color compensation coefficient of the currently to-be-stitched image based on the historical color compensation coefficient corresponding to the currently to-be-stitched image, and update the historical color compensation coefficient (update it to the target color compensation coefficient of the currently to-be-stitched image).

[0192] Secondly, based on the target color compensation coefficient corresponding to each currently to-be-stitched image, perform color correction on each currently to-be-stitched image respectively to obtain k target color images with color balance.

[0193] So far, the description of Figure 3 ends.

[0194] Please refer to Figure 4 , Figure 4 which is the schematic diagram of the specific process of the image alignment stage provided by the embodiment of this application.

[0195] As Figure 4 shown, the image alignment stage realizes the image alignment of each moment of the to-be-stitched video stream. At a specific moment, its input can be the currently to-be-stitched image, or can also include a mapping matrix (that is, used to map the to-be-stitched image from one coordinate system to another coordinate system for precise alignment of different images). In the case where the input includes a mapping matrix, the mapping matrix can be directly applied to perform image transformation on the currently to-be-stitched image. Otherwise, first calculate the mapping matrix and then perform image transformation, and finally output the currently to-be-stitched image after alignment, and can also output the mapping matrix at the same time.

[0196] Specifically, the alignment stage can include the following processes:

[0197] 1. Initialization processing, calculate the mapping matrix based on the currently to-be-stitched image of the initial frame.

[0198] In this embodiment, since the position of the camera in video stitching is usually fixed, the mapping matrix calculated from the first frame can be used for subsequent frames to improve efficiency. For the currently to-be-stitched image (taking a total of T frames as an example), it can first be determined whether the currently to-be-stitched image (that is, the t-th frame image) is the first frame image of the video. If it is the first frame image (that is, t = 1), then the mapping matrix can be calculated.

[0199] The calculation process of the mapping matrix is as follows:

[0200] (1) Feature detection and matching. First, perform feature point detection on the input currently to-be-stitched image, for example, use methods such as SIFT (Scale Invariant Feature Transform) to perform feature extraction to obtain the feature points of each image.

[0201] Feature matching is performed by methods such as NN (Nearest Neighbor: neural network) to obtain matching point pairs between each current image to be stitched. This application does not limit this.

[0202] To further improve the accuracy of the matching point pairs, algorithms such as RANSAC (RANdom SAmpleConsensus: RANdom SAC sampling consistency algorithm) can be applied to filter out outliers and obtain matching point pairs with higher consistency.

[0203] (2) Mapping matrix calculation. The homography matrix can be solved as the mapping matrix. The homography matrix can realize the projection of feature points from one plane to another plane, as shown below:

[0204]

[0205] Among them, the homography matrix H = , h11, h12, and h13 describe the projection transformation in the x-axis direction from the source plane to the target plane, h21, h22, and h23 describe the projection transformation in the y-axis direction from the source plane to the target plane, and the three parameters h31, h32, and h33 are usually related to the perspective transformation. They control the transformation of the third coordinate so that the mapping takes into account the perspective effect; is the coordinate of the point on the target plane obtained after projection, is the coordinate of the point on the source plane.

[0206] The above matrix describes the geometric transformation between the plane of one image and the plane of another image. Through this matrix, a point (x1, y1) on the image can be mapped to a point (x2, y2) on another image through a homography transformation.

[0207] Since the homography matrix H is a homogeneous matrix containing 9 parameters, these parameters are calculated from 4 pairs of matching points. To solve these parameters, at least 4 pairs of matching points are required for the solution. The least squares method can be used for the solution. The method for solving the homography matrix is a common method in related technologies and will not be elaborated here.

[0208] So far, the mapping matrix between the current images to be stitched has been obtained.

[0209] 2. Perform transformation processing on the current images to be stitched at each moment to output the aligned current images to be stitched.

[0210] If it is determined that the current image to be stitched is not the first frame image in the video (i.e., t ≠ 1), then image transformation can be performed based on the determined mapping matrix.

[0211] In this embodiment, the method of inverse warping can be adopted, that is, for each pixel of the target aligned image, its corresponding coordinates in the original image are solved, and the pixel value is obtained by interpolation methods such as bilinear interpolation. That is, for each pixel point of the target image, the inverse matrix of the homography matrix is used to find the corresponding position in the original image. Specifically, it can be calculated by the following formula:

[0212]

[0213] where the homography matrix H = , is the inverse matrix of the homography matrix, is the position of each pixel in the target image; is the corresponding coordinate of the point in the target image in the original image.

[0214] The above embodiments can achieve image alignment, and the specific method of image alignment in this application is not limited.

[0215] After completing the alignment of the current to-be-stitched image corresponding to the current frame, image transformation can be performed on the next frame of image based on the above mapping matrix for image alignment (i.e., t = t + 1) until the T-frame images included in the video are all aligned (i.e., t = T).

[0216] So far, the description of Figure 4 ends.

[0217] Please refer to Figure 5 , Figure 5 which is a schematic diagram of the specific process of the transition zone processing stage provided by the embodiment of this application.

[0218] As Figure 5 shown, the transition zone processing stage realizes the transition zone fusion at each moment of the to-be-stitched video stream. The input of the transition zone processing stage is T target stitched images after alignment and color balance (taking a total of T frames as an example, each frame corresponds to a target stitched image), and the above T target stitched images are processed in the transition zone to make the transition zone transition naturally.

[0219] Specifically, the transition zone processing stage can perform the following steps for each target stitched image (i.e., t = 1 to t = T, the 1st to the Tth target stitched images):

[0220] 1. Seam finding, which is used to determine the position of the seam.

[0221] First, the aligned current to-be-stitched image can be obtained through the image alignment stage, where there is an overlapping area between adjacent images. The seam is a line found in the overlapping area, which can be simply set as the center line of the area, or methods such as graph cut can be used to find the line with the smallest brightness difference between the left and right images.

[0222] 2. Fusion of the transition zone. The transition zone can be the entire overlapping area or can be set as an area within a specific range from the seam. Simple methods include feathering fusion, etc., and complex multi-band fusion methods can also be used to improve the naturalness of the transition.

[0223] Performing the above method on each of the T target stitched images can achieve a natural and smooth transition in the transition zone area. The above embodiments can achieve transition zone processing and obtain the final stitched video.

[0224] So far, the description of Figure 5 ends.

[0225] Please refer to Figure 6A and Figure 6B , Figure 6A which is an example diagram of spatial color balance provided by the embodiments of this application; Figure 6B which is an example diagram of spatio-temporal color balance provided by the embodiments of this application.

[0226] As Figure 6A shown, since there are many dynamic targets in this scene and the lighting conditions change with the weather and street lights, the color differences between the images at the front and back moments are relatively large. According to the traditional method of only performing spatial color balance, there are relatively large color changes in the stitched images at the front and back moments of the stitched video, presenting unnaturalness. When playing the images at these two moments as adjacent frames, there will be a visual perception of color jumping.

[0227] As Figure 6B shown, applying the above spatio-temporal color balance video stitching method, the color differences between the images at the front and back moments are relatively small, presenting a better visual effect, effectively alleviating the problem of inconsistent temporal color, and being more conducive to the natural smoothness of the stitched video.

[0228] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an electronic device proposed by the embodiments of this application. At the hardware level, this electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a terminal interaction device at the logical level. Of course, in addition to the software implementation method, this application does not exclude other implementation methods, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0229] Please refer to Figure 8 , Figure 8 which is a structural diagram of a video stitching device proposed by the embodiments of this application. As Figure 8As shown, the device may include an acquisition module 801, a determination module 802, a correction module 803, and a splicing module 804. Specifically, the device includes:

[0230] The acquisition module 801 is configured to acquire K current images to be spliced from different perspectives, where K is not less than 2;

[0231] The determination module 802 is configured to, for each current image to be spliced, determine a target color compensation coefficient of the current image to be spliced based on a historical color compensation coefficient of a historical image to be spliced corresponding to the current image to be spliced; where the historical image to be spliced and the current image to be spliced are images in the same perspective within the same video; and the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on a current color compensation coefficient and a historical color compensation coefficient of the current image to be spliced;

[0232] The correction module 803 is configured to perform color correction on the current image to be spliced according to the target color compensation coefficient of the current image to be spliced to obtain a target color image corresponding to the current image to be spliced;

[0233] The splicing module 804 is configured to splice the target color images corresponding to the K current images to be spliced to obtain a target spliced image; the target spliced image is used to form a target video.

[0234] Optionally, the determination module 802 is specifically configured to:

[0235] If the difference between the current color distribution of each current image to be spliced and the historical color distribution of the historical image to be spliced is within a preset range, the historical color compensation coefficient of the historical image to be spliced is used as the target color compensation coefficient of the current image to be spliced; where the historical image to be spliced is the previous-frame historical image to be spliced of the current image to be spliced;

[0236] And / or, the determination module 802 is specifically configured to:

[0237] If the difference between the current color distribution of any current image to be spliced and the historical color distribution of the historical image to be spliced is not within the preset range, a weighted operation is performed on the current color compensation coefficient of the current image to be spliced and the historical color compensation coefficient of the historical image to be spliced to obtain a target color compensation coefficient; where the historical image to be spliced is the previous M-frame historical images to be spliced of the current image to be spliced, and M is greater than or equal to 1;

[0238] And / or, the K current images to be spliced include a reference image and at least one non-reference image, and the reference image is any one of the K current images to be spliced that is selected; the determination module 802 is further configured to:

[0239] For the reference image, obtain the configured default color compensation coefficient, and determine the default color compensation coefficient as the current color compensation coefficient of the reference image;

[0240] For each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image;

[0241] And / or, the current color distribution of the non-reference image and the current color distribution of the reference image include the first-channel color distribution, the second-channel color distribution, and the third-channel color distribution; specifically, the determination module 802 is configured to:

[0242] Determine the current color compensation coefficient corresponding to each channel of the non-reference image relative to the reference image according to the following formula:

[0243]

[0244]

[0245]

[0246] Wherein, represents the current color compensation coefficient corresponding to the first channel of the nth non-reference image, represents the first-channel color distribution of the nth non-reference image, represents the first-channel color distribution of the reference image; represents the current color compensation coefficient corresponding to the second channel of the nth non-reference image, represents the second-channel color distribution of the nth non-reference image, represents the second-channel color distribution of the reference image; represents the current color compensation coefficient corresponding to the third channel of the nth non-reference image, represents the third-channel color distribution of the nth non-reference image, represents the third-channel color distribution of the reference image;

[0247] And / or, the determination module 802 is specifically configured to:

[0248] For each current image to be stitched among the K current images to be stitched, determine the non-moving area in the current image to be stitched according to the current image to be stitched and the corresponding historical image to be stitched; the non-moving area refers to the background area in the current image to be stitched except for the moving objects;

[0249] For each non-reference image, based on the current color distribution of the non-moving region in the non-reference image and the current color distribution of the non-moving region in the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image;

[0250] And / or, the determining module 802 is specifically configured to:

[0251] For each of the K currently to-be-stitched images, according to the currently to-be-stitched image and the corresponding historically to-be-stitched image, determine the velocity vector of each pixel point in the currently to-be-stitched image, where the velocity vector includes the change in the motion rate and the change in the motion direction of the pixel point between the acquisition times of the historically to-be-stitched image and the currently to-be-stitched image;

[0252] For each pixel point in the currently to-be-stitched image, if it is determined that the change in the motion rate of the pixel point is less than the first threshold and the change in the motion direction of the pixel point is less than the second threshold, then use the pixel point as a candidate pixel point;

[0253] Based on the set of all candidate pixel points, determine the non-moving region in the currently to-be-stitched image.

[0254] So far, the description of Figure 8 the video stitching device in

[0255] Correspondingly, based on the same application concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, on which several computer instructions are stored. When the computer instructions are executed by a processor, the video stitching method disclosed in the above examples of the present application can be implemented.

[0256] Among them, the above machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0257] Based on the same application concept as the above method, an embodiment of the present application further provides a computer program product, which may include a computer program. Among them, when the computer program is executed by a processor, the video stitching method disclosed in the above examples of the present application is implemented.

[0258] The above are only the preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of protection of the present application.

Claims

1. A video splicing method, characterized in that, The method includes: Obtaining current images to be stitched at K different perspectives, where K is not less than 2; For each current image to be stitched, determining a target color compensation coefficient for the current image to be stitched based on the historical color compensation coefficient of the historical image to be stitched corresponding to the current image to be stitched; wherein, the historical image to be stitched and the current image to be stitched are images at the same perspective within the same video; wherein, the historical color compensation coefficient is determined as the target color compensation coefficient, or the target color compensation coefficient is determined based on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient; Performing color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain a target color image corresponding to the current image to be stitched; Stitching the target color images corresponding to the current images to be stitched at the K different perspectives to obtain a target stitched image; the target stitched image is used to form a target video.

2. The method according to claim 1, wherein The determining the historical color compensation coefficient as the target color compensation coefficient includes: If the difference between the current color distribution of each current image to be stitched and the historical color distribution of the historical image to be stitched is within a preset range, then taking the historical color compensation coefficient of the historical image to be stitched as the target color compensation coefficient of the current image to be stitched; wherein, the historical image to be stitched is the previous-frame historical image to be stitched of the current image to be stitched.

3. The method according to claim 1, characterized in that, The determining the target color compensation coefficient based on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient includes: If the difference between the current color distribution of any current image to be stitched and the historical color distribution of the historical image to be stitched is not within the preset range, then performing a weighted operation on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient of the historical image to be stitched to obtain the target color compensation coefficient; wherein, the historical image to be stitched is the previous M-frame historical images to be stitched of the current image to be stitched, and M is greater than or equal to 1.

4. The method according to claim 1 or 3, characterized in that The current images to be stitched at the K different perspectives include a reference image and at least one non-reference image, the reference image is any one of the current images to be stitched selected from the current images to be stitched at the K different perspectives; the current color compensation coefficient of the current image to be stitched is determined by the following method: For the reference image, obtaining a configured default color compensation coefficient and determining the default color compensation coefficient as the current color compensation coefficient of the reference image; For each non-reference image, determining the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-reference image and the current color distribution of the reference image.

5. The method according to claim 4, wherein The current color distribution of the non-reference image and the current color distribution of the reference image include a first-channel color distribution, a second-channel color distribution, and a third-channel color distribution; the determining the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-reference image and the current color distribution of the reference image includes: Determine the current color compensation coefficient corresponding to each channel of the non-reference image relative to the reference image according to the following formula: Among them, the represents the current color compensation coefficient corresponding to the nth non-reference image in the first channel, and the represents the color distribution of the first channel of the nth non-reference image, and the represents the color distribution of the first channel of the reference image; the represents the current color compensation coefficient corresponding to the nth non-reference image in the second channel, and the represents the color distribution of the second channel of the nth non-reference image, and the represents the color distribution of the second channel of the reference image; the represents the current color compensation coefficient corresponding to the nth non-reference image in the third channel, and the represents the color distribution of the third channel of the nth non-reference image, and the represents the color distribution of the third channel of the reference image.

6. The method according to claim 4, characterized in that, For each non-reference image, determining the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-reference image and the current color distribution of the reference image includes: For each of the K current images to be stitched, determine the non-moving region in the current image to be stitched according to the current image to be stitched and the corresponding historical image to be stitched; the non-moving region refers to the background region in the current image to be stitched except for the moving objects; For each non-reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image based on the current color distribution of the non-moving region in the non-reference image and the current color distribution of the non-moving region in the reference image.

7. The method according to claim 6, characterized in that, The step of determining the non-moving region in the current image to be stitched according to the current image to be stitched and the corresponding historical image to be stitched for each of the K current images to be stitched includes: For each of the K current images to be stitched, determine the velocity vector of each pixel point in the current image to be stitched according to the current image to be stitched and the corresponding historical image to be stitched, where the velocity vector includes the change in the motion rate and the change in the motion direction of the pixel point between the acquisition times of the historical image to be stitched and the current image to be stitched; For each pixel point in the current image to be stitched, if it is determined that the change in the motion rate of the pixel point is less than the first threshold and the change in the motion direction of the pixel point is less than the second threshold, then regard the pixel point as a candidate pixel point; Determine the non-moving region in the current image to be stitched based on the set of all candidate pixel points.

8. A video splicing device, characterized in that, The device includes: An acquisition module, configured to acquire K current images to be stitched with different perspectives, where K is not less than 2; A determination module, configured to determine the target color compensation coefficient of each current image to be stitched based on the historical color compensation coefficient of the corresponding historical image to be stitched; wherein, the historical image to be stitched and the current image to be stitched are images in the same video and at the same perspective; wherein, determine the historical color compensation coefficient as the target color compensation coefficient, or determine the target color compensation coefficient based on the current color compensation coefficient of the current image to be stitched and the historical color compensation coefficient; A correction module, configured to perform color correction on the current image to be stitched according to the target color compensation coefficient of the current image to be stitched to obtain the target color image corresponding to the current image to be stitched; A stitching module, configured to stitch the target color images corresponding to the K current images to be stitched with different perspectives to obtain a target stitched image; the target stitched image is used to form a target video.

9. The device according to claim 8, characterized in that, The determination module is specifically configured to: If the difference between the current color distribution of each currently to-be-stitched image and the historical color distribution of the historical to-be-stitched image is within a preset range, then use the historical color compensation coefficient of the historical to-be-stitched image as the target color compensation coefficient of the currently to-be-stitched image; wherein, the historical to-be-stitched image is the previous-frame historical to-be-stitched image of the currently to-be-stitched image. And / or, the determining module is specifically configured to: If the difference between the current color distribution of any currently to-be-stitched image and the historical color distribution of the historical to-be-stitched image is not within the preset range, then perform a weighted operation on the current color compensation coefficient of the currently to-be-stitched image and the historical color compensation coefficient of the historical to-be-stitched image to obtain the target color compensation coefficient; wherein, the historical to-be-stitched image is the previous M-frame historical to-be-stitched images of the currently to-be-stitched image, and M is greater than or equal to 1. And / or, the K currently to-be-stitched images with different perspectives include a reference image and at least one non-reference image, and the reference image is any currently to-be-stitched image selected from the K currently to-be-stitched images with different perspectives; the determining module is further configured to: For the reference image, obtain the configured default color compensation coefficient, and determine the default color compensation coefficient as the current color compensation coefficient of the reference image. For each non-reference image, based on the current color distribution of the non-reference image and the current color distribution of the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image. And / or, the current color distribution of the non-reference image and the current color distribution of the reference image include a first-channel color distribution, a second-channel color distribution, and a third-channel color distribution; the determining module is specifically configured to: Determine the current color compensation coefficient corresponding to each channel of the non-reference image relative to the reference image according to the following formula: Among them, the represents the current color compensation coefficient corresponding to the nth non-reference image in the first channel, and the represents the color distribution of the first channel of the nth non-reference image, and the represents the color distribution of the first channel of the reference image; the represents the current color compensation coefficient corresponding to the nth non-reference image in the second channel, and the represents the color distribution of the second channel of the nth non-reference image, and the represents the color distribution of the second channel of the reference image; the represents the current color compensation coefficient corresponding to the nth non-reference image in the third channel, and the represents the color distribution of the third channel of the nth non-reference image, and the represents the color distribution of the third channel of the reference image; And / or, the determining module is specifically configured to: For each currently to-be-stitched image among the K currently to-be-stitched images, determine the non-moving region in the currently to-be-stitched image according to the currently to-be-stitched image and the corresponding historical to-be-stitched image; the non-moving region refers to the background region in the currently to-be-stitched image except for moving objects. For each non-reference image, based on the current color distribution of the non-moving region in the non-reference image and the current color distribution of the non-moving region in the reference image, determine the current color compensation coefficient of the non-reference image relative to the reference image. And / or, the determining module is specifically configured to: For each currently to-be-stitched image among the K currently to-be-stitched images, determine the velocity vector of each pixel point in the currently to-be-stitched image according to the currently to-be-stitched image and the corresponding historical to-be-stitched image, and the velocity vector includes the change amount of the motion rate and the change amount of the motion direction of the pixel point between the acquisition moments of the historical to-be-stitched image and the currently to-be-stitched image. For each pixel point in the current image to be stitched, if it is determined that the change in the motion rate of the pixel point is less than the first threshold and the change in the motion direction of the pixel point is less than the second threshold, then the pixel point is used as a candidate pixel point; Based on the set of all candidate pixel points, determine the non-moving area in the current image to be stitched.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images