Video synthesis method, device, computer equipment and storage medium
By performing color feature analysis and selection of color tuning reference images on the first video, the color tone of the second video is automatically adjusted, and the problem of automatic color tuning of video synthesis in the prior art is solved, simplifying the process and improving the video quality.
Patent Information
- Application Number
- CN202110558939.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-21
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-05-21
AI Technical Summary
The prior art is difficult to achieve automatic color tuning during video synthesis, which leads to users needing professional photography knowledge to produce high-quality combined videos.
By performing color characteristics analysis on the frame image in the first video, selecting a color tone reference image, and adjusting the second video based on the tone, and finally performing video synthesis.
Automatic color tuning is realized, the video synthesis process is simplified, and the quality of combined videos is improved, so that users can produce high-quality combined videos without having professional knowledge.
Smart Images

Figure CN113762058B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a video synthesis method, apparatus, computer equipment and storage medium. Background Art
[0002] At present, short videos have become the mainstream form of video consumption and one of the important operating methods of content platforms. Many content platforms provide users with video synthesis functions, where users can synthesize their own videos with other videos on the content platform into a co-produced video for publication.
[0003] Many videos on content platforms are shot by professional users or teams, and the color performance of these videos is generally excellent. If you want to make the co-shot videos better, users need to have professional photography knowledge and be able to perform operations such as color adjustment on the videos they shoot. Summary of the invention
[0004] The embodiments of the present invention provide a video synthesis method, apparatus, computer equipment and storage medium, which can automatically adjust the color of a second video based on a first video, thereby improving the video synthesis effect.
[0005] An embodiment of the present invention provides a video synthesis method, the method comprising:
[0006] Performing color feature analysis on at least two frames of first images in the first video to obtain color features of the first images;
[0007] comparing the color richness of the first image according to the color feature, and selecting a color adjustment reference image from the first image based on the comparison result;
[0008] Based on the hue of the color-adjusting reference image, adjusting the hue of each second image of the second video to obtain an adjusted second video;
[0009] A synthesis operation is performed on the first video and the adjusted second video to obtain a combined video.
[0010] The embodiment of the present invention further provides a video synthesis device, the device comprising:
[0011] A color feature extraction unit, configured to perform color feature analysis on at least two frames of first images in the first video to obtain color features of the first images;
[0012] an image selection unit, configured to compare the color richness of the first image according to the color feature, and select a color adjustment reference image from the first image based on the comparison result;
[0013] A tone matching unit, configured to adjust the tone of each second image of the second video based on the tone of the color-adjusting reference image to obtain an adjusted second video;
[0014] The synthesis unit is used to perform a synthesis operation on the first video and the adjusted second video to obtain a synthesized video.
[0015] In an optional example, the color feature extraction unit is used to:
[0016] Performing color mapping on at least two frames of first images in the first video, mapping a first color of a pixel point in each of the first images to a second color, wherein the first color is represented in a triplet form and the second color is represented in a binary form;
[0017] Count the number of pixels of each second color in the same first image, and obtain the color characteristics of the first image based on the second color and the number of pixels of the second color, wherein the number of pixels of a second color is the number of pixels of the first color mapped to the second color.
[0018] In an optional example, the image selection unit is used to:
[0019] Based on the second color of the first image and the number of pixels of the second color, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
[0020] In an optional example, the triplet is an RGB three-primary color triplet, and the color feature extraction unit is used to:
[0021] Obtain a color lookup table, wherein the color lookup table is used to map a color representation in the form of an RGB three-primary color triplet to a color representation in the form of a two-tuple;
[0022] Acquire at least two frames of first images in a first video;
[0023] In the color lookup table, a second color corresponding to the first color of a pixel in the first image is searched.
[0024] In an optional example, the image selection unit is used to:
[0025] Filtering out the second color whose number of pixels is lower than a preset number threshold from the color features of the first image to obtain filtered color features;
[0026] The color richness of the first images is compared according to the number of second colors in the filtered color features of each of the first images, and a color adjustment reference image is selected from the first images based on the comparison result.
[0027] In an optional example, the color feature includes a color distribution histogram, and the color feature extraction unit is used to:
[0028] Acquire at least two frames of first images in a first video;
[0029] A color distribution histogram analysis is performed on each of the first images to obtain a color distribution histogram of each of the first images.
[0030] In an optional example, the image selection unit is used to:
[0031] According to the proportion of each color component in the color distribution histogram of the first image in the first image, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
[0032] In an optional example, the tone matching unit is used to:
[0033] Acquire a color distribution histogram of the color-adjusting reference image as a reference histogram;
[0034] The color distribution histogram of each second image in the second video is adjusted according to the reference histogram, and the second video after the histogram is adjusted is used as the adjusted second video.
[0035] In an optional example, the number of the color reference images is at least two, and the tone matching unit is used to:
[0036] During the shooting of the second video, at least two color tone selection controls are displayed on a video preview interface of the second video, wherein the video preview interface includes a captured video image, and one color tone selection control corresponds to one color reference image;
[0037] In response to a preview operation on a hue selection control, adjusting the hue of a video image displayed on the video preview interface based on a color reference image corresponding to the hue selection control selected by the preview operation;
[0038] In response to a selection operation on a target color tone selection control in the color tone selection controls, determining a target color reference image corresponding to the target color tone selection control;
[0039] The color tone of each second image of the second video is adjusted based on the target color reference image to obtain an adjusted second video.
[0040] In an optional example, the device further includes a composition prompting unit, configured to:
[0041] Performing composition analysis on the first image of the first video to determine a target display area of the first photographed object in the first image;
[0042] During the shooting of the second video, obtaining an actual display area of the second shooting object in the second image of the second video;
[0043] Based on the actual display area and the target display area, shooting prompt information is output, and the shooting prompt information is used to prompt adjustment of shooting parameters of the second video so that the position of the actual display area in the second image matches the position of the target display area in the first image.
[0044] In some embodiments of the present invention, a computer device may be provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0045] In some embodiments of the present invention, a storage medium may be provided, on which a computer program is stored. When the computer program is executed on a computer, the computer is caused to execute the steps of the method as described above.
[0046] By using the embodiments of the present application, color feature analysis can be performed on at least two frames of first images in a first video to obtain color features of the first images; the color richness of the first images can be compared according to the color features, and a color reference image can be selected from the first images based on the comparison results; based on the hue of the color reference image, the hue of each second image of the second video can be adjusted to obtain an adjusted second video; the first video and the adjusted second video can be synthesized to obtain a combined video, thereby quickly selecting a more representative color reference image in the first video, thereby quickly adjusting the hue of the image of the second video. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0048] Figure 1 is a schematic diagram of the structure of a video synthesis system provided by an embodiment of the present invention;
[0049] Figure 2a is a flow chart of a video synthesis method provided by an embodiment of the present invention;
[0050] Figure 2b is a schematic diagram of a color distribution histogram provided by an embodiment of the present invention;
[0051] Figure 2c is a schematic diagram of a LUT provided by an embodiment of the present invention;
[0052] Figure 2d is a schematic diagram of color tone selection provided by an embodiment of the present invention;
[0053] Figure 3 is a schematic diagram of the structure of a video synthesis device provided by an embodiment of the present invention;
[0054] Figure 4 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0056] The embodiments of the present invention provide a video synthesis method, apparatus, computer equipment and storage medium.
[0057] The embodiment of the present invention provides a video synthesis system, which includes a video synthesis device suitable for a computer device, wherein the computer device may be a terminal or a server.
[0058] The terminal may be a mobile phone, tablet computer, laptop computer or other terminal device, or a wearable device, smart TV or other smart terminal with a display module.
[0059] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, but is not limited to these.
[0060] The video synthesis device of this embodiment may be integrated into a terminal or a server. Optionally, it may be integrated into the terminal or the server in the form of an application program or the like.
[0061] refer to Figure 1 The video synthesis system provided in this embodiment includes a terminal 10 and a server 20, etc.
[0062] Terminal 10 can be used to perform color feature analysis on at least two frames of first images in a first video to obtain color features of the first images; compare the color richness of the first images according to the color features, and select a color reference image from the first images based on the comparison results; adjust the hue of each second image of the second video based on the hue of the color reference image to obtain an adjusted second video; and perform a synthesis operation on the first video and the adjusted second video to obtain a combined video.
[0063] The server 20 may be configured to send the first video to the terminal 10 according to a request of the terminal 10 .
[0064] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0065] The embodiment of the present invention will be described from the perspective of a video synthesis device, which may be integrated into a terminal or a server, for example, in the form of an application program.
[0066] like Figure 2a As shown, the process of the video synthesis method can be as follows:
[0067] 201. Perform color feature analysis on at least two frames of first images in a first video to obtain color features of the first images;
[0068] In this embodiment, the source of the first video is not limited, and it can be a video in the form of a TV series, a movie, an animation, etc. The length of the first video is not limited, and can be determined by the producer of the first video according to demand.
[0069] In this embodiment, the form of color features and the method of extracting color features are not limited. For example, in one example, color features include but are not limited to: color feature vectors extracted by a neural network, color distribution histograms, and feature information statistically calculated from the color perspective of pixels, etc.
[0070] At least three color feature extraction methods of this embodiment are introduced below:
[0071] Method (1), a scheme for extracting color feature vectors through a neural network. Optionally, the step of "performing color feature analysis on at least two frames of first images in the first video to obtain color features of the first images" may include:
[0072] Obtain the trained image analysis model;
[0073] Acquire at least two first image frames of a first video;
[0074] Color features of the first image are extracted based on the image analysis model to obtain a color feature vector of the first image.
[0075] The training samples of the image analysis model include sample video images with labels, wherein the value in the label of the sample video image represents the color richness of the sample video image, and optionally, the value in the label ranges from 0 to 1, indicating that the color richness is 0 to 1.
[0076] Optionally, the image analysis model can be deployed in a terminal or a server.
[0077] In one example, the image analysis model may be trained based on the training samples to obtain a trained image analysis model for use.
[0078] In this embodiment, considering that there are many repeated frames in the video sequence of the first video, the first image can be extracted by frame sampling to reduce the calculation of repeated images and reduce the complexity of the algorithm. Optionally, the video segment to be matched, that is, the first video, is sampled, wherein the interval is a preset sampling interval, for example, one frame is sampled every 0.5s, which is recorded as I i , all sampling results are recorded as:
[0079]
[0080] Among them, I orig represents the first image in the first video, is the i-th frame image sampled.
[0081] In this embodiment, color feature extraction of the first image based on the image analysis model to obtain a color feature vector of the first image may include: color feature extraction of pixel values of each pixel in the first image based on the image analysis model to obtain a color feature vector of the first image.
[0082] The image analysis model of this embodiment is an application based on computer vision technology. Computer vision technology (CV) is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0083] The image analysis model of this embodiment uses image processing, image recognition and other technologies. The training of the image analysis model is based on AI (Artificial Intelligence) technology, especially machine learning (ML) technology in artificial intelligence technology. More specifically, it can be achieved through deep learning in machine learning.
[0084] The image analysis model in this embodiment can be constructed based on the structure of any artificial neural network that can be used for image recognition. For example, the image analysis model can be CNN (Convolutional Neural Network) and the like.
[0085] Method (2): The color feature is a color distribution histogram.
[0086] Optionally, the step of “performing color feature analysis on at least two frames of first images in the first video to obtain color features of the first images” may include:
[0087] Acquire at least two frames of first images in a first video;
[0088] A color distribution histogram analysis is performed on each of the first images to obtain a color distribution histogram of each of the first images.
[0089] Wherein, obtaining at least two frames of first images in the first video includes: obtaining a sampling interval, and performing frame sampling on the first video based on the sampling interval to obtain at least two frames of first images.
[0090] The sampling interval is set according to actual needs.
[0091] It can be understood that, in this embodiment, the same histogram acquisition algorithm is used to acquire the color distribution histogram for different first images.
[0092] In this embodiment, the color distribution histogram includes multiple color components of the first image and the proportion of each color component in the first image.
[0093] Among them, a reference to the color distribution histogram Figure 2b As shown, Figure 2b In the histogram, each rectangle represents a color component, and the height of the rectangle indicates the proportion of the color component in the first image.
[0094] Method (3) statistically analyzes the feature information from the color perspective of the pixel points.
[0095] In one example, there is similarity between the colors of different pixels. From the perspective of the colors of the pixels, certain statistics can be performed on the colors of the pixels to obtain color features that can characterize the first image.
[0096] Optionally, the step of “performing color feature analysis on at least two frames of first images in the first video to obtain color features of the first images” may include:
[0097] Performing color mapping on at least two frames of first images in the first video, mapping a first color of a pixel point in each of the first images to a second color, wherein the first color is represented in a triplet form and the second color is represented in a binary form;
[0098] Count the number of pixels of each second color in the same first image, and obtain the color characteristics of the first image based on the second color and the number of pixels of the second color, wherein the number of pixels of a second color is the number of pixels of the first color mapped to the second color.
[0099] Among them, the triplet form of the first color includes but is not limited to the RGB three-primary color triplet, that is, the first color is expressed as (R, G, B), and the first color of the pixel point in the first image is represented by the red component value R, the green component value G, and the blue component value B.
[0100] Optionally, the color mapping in this embodiment may map the color represented by the RGB color triplet into a two-dimensional space and represent it with a binary color triplet.
[0101] The following example illustrates the meaning of the number of pixels of the second color.
[0102] For example, the first image A contains 160,000 pixels of 400*400. The first colors (R, G, B) of these 160,000 pixels are mapped to (xn, yn) after color mapping, where n is an integer not less than 0. Assuming n is 4000, it means that the first colors of the 160,000 pixels are mapped to 4000 second colors, namely (x1, y1), (x2, y2)...(x4000, y4000). Among them, if the first colors (R, G, B) of 100 pixels are mapped to (x1, y1), then for the second color (x1, y1), the number of pixels is 100; if the first colors (R, G, B) of 250 pixels are mapped to (x2, y2), then for the second color (x2, y2), the number of pixels is 250.
[0103] In one example, the mapping can be implemented based on a LUT lookup table method, which is optional. The step of "color mapping at least two frames of the first image in the first video, mapping the first color of each pixel in the first image to the second color" may include:
[0104] Obtain a color lookup table, wherein the color lookup table is used to map a color representation in the form of an RGB triplet to a color representation in the form of a binary;
[0105] Acquire at least two frames of first images in a first video;
[0106] In the color lookup table, a second color corresponding to the first color of a pixel in the first image is searched.
[0107] Among them, the LUT (Look-Up-Table) lookup table method can encode RGB into 512x512 coordinates.
[0108] In actual engineering, an information space of n*n*n is usually used to roughly represent an information space of 256 (red channel)*256 (green channel)*256 (blue channel). n is set as needed. For example, according to 4 as the level, 256 of each channel is compressed to 64, and an information space of 256*256*256 can be represented by an information space of 64*64*64. In an example, in order to realize the mapping from three-dimensional space to two-dimensional space, 64*64*64 is decomposed into 64*64*8*8, that is, an image size of 512*512.
[0109] In an example, according to 4 as the classification, the component values 0-255 are divided into 64 equal parts in the LUT map. Assume that the mapping relationship between the color component values 0-255 and the mapping values in the LUT lookup table is:
[0110] Component value interval in channel -> mapping value:
[0111]
[0112] That is, the color component values 0-4 in the three channels of the three primary colors are mapped to 0, 4-8 are mapped to 4, and so on.
[0113] Among them, see Figure 2c The LUT image shown in the figure includes 8*8 sub-images, each of which stores a 64*64 operation result. The 64*64 operation result in each sub-image represents the operation result of the 64 component values of each of the two channels after mapping, such as the 64*64 operation result of the 64 component values of the red channel and the 64 component values of the green channel. The index of each sub-image is determined by the component values of the remaining channels, such as the blue channel.
[0114] In this embodiment, optionally, the component value of the B channel in RGB can be used as an index to determine the position of the second color (x, y) in the target sub-image to which the pixel is mapped in the LUT image, and the component value of the RG channel is used as the local coordinates of the second color (x, y) in the target sub-image of the LUT image. The present invention expresses a similar mapping as follows:
[0115]
[0116] In such Figure 2c In the LUT diagram shown, the horizontal and vertical coordinates (x, y) of the image are mapped pairs, which can be expressed as (x1*64+x2, y1*64+y2), where the value of the blue channel (Blue) can be obtained through (x1, y1), there are 64 (x1, y1), corresponding to the 64 components of the blue channel, the component value of the red channel (Red) can be obtained through x2, and the component value of the green channel (Green) can be obtained through y2.
[0117] For example, (x1, y1) indicates Figure 2c In the y1th column, the x1th sub-image, the value range of x1 and y1 can be eight integers from 1 to 8. The component value of the blue channel (Blue) corresponding to this sub-image is (y1-1)*8+x1. In an example, ((y1-1)*8+x1, x2, y2) is the value of the triplet of the first color in the first image after being mapped according to 4 as the graded value. It can be understood that each sub-image in the LUT image has color, but Figure 2c The colors are not shown.
[0118] The (x1*8+y1, x2, y2) is input into the LUT lookup table, and the LUT lookup table can find the above-mentioned (x, y) coordinates, and the (x, y) is the second color represented by the binary group in this embodiment.
[0119] The statistical formula for the number of pixels of the second color (x, y) is as follows:
[0120]
[0121] Among them, I orig represents the original image (i.e., the first image), w and h represent the width and height of the original image, step() is a step function, and when the input value in () is greater than 0, it is 1, and when it is less than zero, it is 0. orig (x, y) represents the pixel point mapped to (x, y) in the original image. x,y Indicates the number of pixels in the original image whose first color is mapped to (x, y).
[0122] 202. Compare the color richness of the first image according to the color feature, and select a color adjustment reference image from the first image based on the comparison result;
[0123] In one example, for the above method (1) of extracting color feature vectors through a neural network, the step of selecting a color reference image includes:
[0124] Performing color richness classification on the color features through the image analysis model to obtain classification probabilities of the image analysis model under at least two preset color richness classifications;
[0125] A toning reference image is determined based on the classification probability of the first image under each color richness classification.
[0126] For example, the image analysis model may be provided with 10 color richness classifications, which respectively represent classifications with color richness of 0.1, 0.2, 0.3, ... 1. The image analysis model may output classification results under the 10 classifications for the first image, and the sum of the probabilities of the 10 classification results is 1.
[0127] Optionally, determining the color reference image based on the classification probability of the first image in each color richness classification may include: selecting the first image with the highest classification probability in the classification with the highest color richness as the color reference image. For example, selecting the first image with the highest classification probability in the classification with a color richness of 1 as the color reference image.
[0128] In one example, for the above method (2) in which the color feature includes a color distribution histogram, the step of selecting a color reference image includes:
[0129] According to the proportion of each color component in the color distribution histogram of the first image in the first image, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
[0130] Optionally, the color richness may be determined by the types of color components and the proportion of each color component in the first image. The more color components with larger proportions there are, the higher the color richness is.
[0131] Optionally, the step of “comparing the color richness of the first image according to the proportion of each color component in the color distribution histogram of the first image in the first image, and selecting a color adjustment reference image from the first image based on the comparison result” may include:
[0132] Counting the proportion of each color component in the first image in the color distribution histogram of each first image;
[0133] Determine a target color component whose ratio is not less than a preset ratio threshold;
[0134] The number of target color components in each first image is counted, and a color adjustment reference image is selected in descending order of the number of target color components.
[0135] If only one color reference image is needed, the first image with the largest number of target color components is selected as the target color component. If multiple color reference images are needed, multiple color reference images are selected in descending order of the number of target color components.
[0136] In one example, for the feature information obtained from the color angle of the pixel points in the above method (3), the step of "comparing the color richness of the first image according to the color feature, and selecting a color reference image from the first image based on the comparison result" includes:
[0137] Based on the second color of the first image and the number of pixels of the second color, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
[0138] In one example, if the number of second colors with a higher number of pixels in the first image is greater, the color richness of the first image is higher.
[0139] In order to eliminate the noise in the image and reduce the number of noise pixels, the first image may be filtered first. If the number of pixels mapped to the same second color is small (e.g., only one or two pixels of the first color are mapped to a second color (x, y), etc.), the color mapping result is that the pixel of the second color (x, y) is a noise pixel.
[0140] Optionally, the step of “comparing the color richness of the first image based on the second color of the first image and the number of pixels of the second color, and selecting a color adjustment reference image from the first image based on the comparison result” may include:
[0141] Filtering out the second color whose number of pixels is lower than a preset number threshold from the color features of the first image to obtain filtered color features;
[0142] The color richness of the first images is compared according to the number of second colors in the filtered color features of each of the first images, and a color adjustment reference image is selected from the first images based on the comparison result.
[0143] The preset quantity threshold is set as required, and this embodiment has no limitation on this.
[0144] For example, assuming that the quantity threshold is 5, and taking the first image A mentioned above as an example, the first image A contains 160,000 pixels of 400*400. The first colors (R, G, B) of these 160,000 pixels are mapped to (xn, yn) after color mapping, where n is 4000, and (xn, yn) are (x1, y1), (x2, y2)···(x4000, y4000) respectively.
[0145] Among them, during the filtering process, it was found that the number of pixels of the 10 second colors (x1, y1)-(x10, y10) was less than 5, then the second colors in the color features after filtering were (x11, y11)-(x4000, y4000), and the number of second colors (x2, y2) in the color features after filtering was 3990.
[0146] In an optional example, the filtering process is expressed as:
[0147] I′x,y=filter(I x,y )
[0148] Among them, filter represents filtering processing, I′ x,y (x, y) in represents the second color whose number of pixels is not less than the number threshold.
[0149] Optionally, the number of the second color in the filtered color feature of the first image can be calculated using the following formula:
[0150]
[0151] Here, k may be equal to 512.
[0152] Among them, the LUT lookup table can be used to simply and quickly map the first color (R, G, B) to a two-dimensional space to obtain a binary group (x, y), which can achieve simple, fast and accurate selection of a color reference image. In addition, the solution based on the LUT lookup table can be quickly deployed on the terminal, which is conducive to the promotion and popularization of video synthesis methods and to improving the user's video co-shooting experience.
[0153] 203. Adjust the color tone of each second image of the second video based on the color tone of the color adjustment reference image to obtain an adjusted second video;
[0154] In this embodiment, hue refers to the overall tendency of the color of a picture, which is a large color effect. In nature, we often see such a phenomenon: objects of different colors are shrouded in a golden sun, or in a light blue moonlight like a veil of mist; or in the charming golden color of autumn; or unified in the silvery white world of winter. This color phenomenon of objects of different colors being shrouded in a certain color, so that objects of different colors have the same color tendency, is hue.
[0155] Optionally, in this embodiment, before color adjustment, certain video corrections may be performed on the adjusted second video to correct problems that may occur when the user shoots the second video. For example, white balance calibration may be performed on the adjusted second video.
[0156] In one example, the hue adjustment may be implemented through a built-in plug-in of the video software. This embodiment has no limitation on the algorithm used for the hue adjustment.
[0157] In an optional example, the step of “adjusting the hue of each second image of the second video based on the hue of the color-adjusting reference image to obtain an adjusted second video” may include:
[0158] Acquire a color distribution histogram of the color-adjusting reference image as a reference histogram;
[0159] The color distribution histogram of each second image in the second video is adjusted according to the reference histogram, and the second video after the histogram is adjusted is used as the adjusted second video.
[0160] The method of adjusting the color distribution histogram of each second image in the second video according to the reference histogram to obtain the second video after the histogram is adjusted as the adjusted second video may include:
[0161] Acquire a second image of a second video;
[0162] According to the ratio of each color component in the reference histogram, adjusting the ratio of each color component in the color distribution histogram of the second image to obtain an adjusted second image;
[0163] Based on the adjusted second image, an adjusted second video is obtained.
[0164] Among them, when adjusting the proportion of each color component in the color distribution histogram of the second image according to the proportion of each color component in the reference histogram, the first relative ratio of each color component in the reference histogram to the reference color component (a specific component among the color components) in the image can be calculated, and the proportion of the color component in the second image in the second image is adjusted so that the second relative ratio of the color component in the second image to the reference color component in the second image matches (such as being the same as) the first relative ratio.
[0165] 204. Perform a synthesis operation on the first video and the adjusted second video to obtain a combined video.
[0166] The synthesis may include synthesizing image frames at the same frame position in the first video and the adjusted second video into one frame to obtain a combined video, wherein the method of synthesizing one frame includes: splicing the first image of the first video and the second image of the second video to obtain a composite image.
[0167] In one example, optionally, the number of the color reference images may be at least two, and the user may select the desired color effect. Optionally, the step of "adjusting the color tone of each second image of the second video based on the color tone of the color reference image to obtain the adjusted second video" may include:
[0168] During the shooting of the second video, at least two color tone selection controls are displayed on a video preview interface of the second video, wherein the video preview interface includes a captured video image, and one color tone selection control corresponds to one color reference image;
[0169] In response to a preview operation on a hue selection control, adjusting the hue of a video image displayed on the video preview interface based on a color reference image corresponding to the hue selection control selected by the preview operation;
[0170] In response to a selection operation on a target color tone selection control in the color tone selection controls, determining a target color reference image corresponding to the target color tone selection control;
[0171] The color tone of each second image of the second video is adjusted based on the target color reference image to obtain an adjusted second video.
[0172] For example, see Figure 2d , Figure 2dThe display mode of the first video and the second video in the co-shooting process is shown. The first video is displayed in the left area of the screen, and the video preview interface of the second video is displayed in the right area of the screen. The first video is a video that has been shot, so the above scheme can be used to select at least two color reference images from the first video, and generate corresponding color selection controls to be displayed in the video preview interface, as shown in FIG. Figure 2d There are two controls in the menu: "Tone 1" and "Tone 2".
[0173] In response to a preview operation such as a click operation for "Tone 1", the tone of the video image displayed on the video preview interface is adjusted based on the color reference image corresponding to the tone selection control selected by the preview operation. Thus, the user can preview the color effect of "Tone 1". If the user is not satisfied, he can also click "Tone 2" to preview the corresponding effect. If the user triggers a selection operation "such as a long press operation" for the target tone selection control in the tone selection control, the target color reference image corresponding to the target tone selection control is determined, and then the second image in the second video can be adjusted based on the target color reference image. Optionally, the adjustment of the second image can be performed after each frame of the second image is captured, or it can be performed after the second video is shot. This embodiment has no limitation on this.
[0174] Among them, considering the similarity of the composition of the first video and the second video, in order to improve the effect of the synthesized video, this embodiment can also guide the user of the second video to adjust the shooting parameters during the shooting process of the second video to improve the similarity of the composition of the second video and the first video.
[0175] Optionally, this embodiment may further include:
[0176] Performing composition analysis on the first image of the first video to determine a target display area of the first photographed object in the first image;
[0177] During the shooting of the second video, obtaining an actual display area of the second shooting object in the second image of the second video;
[0178] Based on the actual display area and the target display area, shooting prompt information is output, and the shooting prompt information is used to prompt adjustment of shooting parameters of the second video so that the position of the actual display area in the second image matches the position of the target display area in the first image.
[0179] Optionally, in this embodiment, the shooting parameters include but are not limited to parameters such as focal length, position and size of the second shot object in the second image, etc.
[0180] There may be multiple first shooting objects, or the first shooting object may be the main shooting object in the first video (the shooting object occupying the largest area).
[0181] There may be multiple second shooting objects, or the second shooting object may be the main shooting object in the second video (the shooting object occupying the largest area).
[0182] By using the embodiments of the present application, color feature analysis can be performed on at least two frames of first images in a first video to obtain color features of the first images; the color richness of the first images can be compared according to the color features, and a color reference image can be selected from the first images based on the comparison results; based on the hue of the color reference image, the hue of each second image of the second video can be adjusted to obtain an adjusted second video; the first video and the adjusted second video can be synthesized to obtain a combined video, thereby quickly selecting a more representative color reference image in the first video, thereby quickly adjusting the hue of the image of the second video.
[0183] In order to better implement the above method, accordingly, an embodiment of the present invention further provides a video synthesis device, which is specifically integrated in a terminal or a server.
[0184] refer to Figure 3 , the device comprises:
[0185] A color feature extraction unit 301 is used to perform color feature analysis on at least two frames of first images in a first video to obtain color features of the first images;
[0186] An image selection unit 302, configured to compare the color richness of the first image according to the color feature, and select a color adjustment reference image from the first image based on the comparison result;
[0187] A tone matching unit 303 is used to adjust the tone of each second image of the second video based on the tone of the color reference image to obtain an adjusted second video;
[0188] The synthesis unit 304 is used to perform a synthesis operation on the first video and the adjusted second video to obtain a synthesized video.
[0189] In an optional example, the color feature extraction unit is used to:
[0190] Performing color mapping on at least two frames of first images in the first video, mapping a first color of a pixel point in each of the first images to a second color, wherein the first color is represented in a triplet form and the second color is represented in a binary form;
[0191] Count the number of pixels of each second color in the same first image, and obtain the color characteristics of the first image based on the second color and the number of pixels of the second color, wherein the number of pixels of a second color is the number of pixels of the first color mapped to the second color.
[0192] In an optional example, the image selection unit is used to:
[0193] Based on the second color of the first image and the number of pixels of the second color, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
[0194] In an optional example, the triplet is an RGB three-primary color triplet, and the color feature extraction unit is used to:
[0195] Obtain a color lookup table, wherein the color lookup table is used to map a color representation in the form of an RGB three-primary color triplet to a color representation in the form of a two-tuple;
[0196] Acquire at least two frames of first images in a first video;
[0197] In the color lookup table, a second color corresponding to the first color of a pixel in the first image is searched.
[0198] In an optional example, the image selection unit is used to:
[0199] Filtering out the second color whose number of pixels is lower than a preset number threshold from the color features of the first image to obtain filtered color features;
[0200] The color richness of the first images is compared according to the number of second colors in the filtered color features of each of the first images, and a color adjustment reference image is selected from the first images based on the comparison result.
[0201] In an optional example, the color feature includes a color distribution histogram, and the color feature extraction unit is used to:
[0202] Acquire at least two frames of first images in a first video;
[0203] A color distribution histogram analysis is performed on each of the first images to obtain a color distribution histogram of each of the first images.
[0204] In an optional example, the image selection unit is used to:
[0205] According to the proportion of each color component in the color distribution histogram of the first image in the first image, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
[0206] In an optional example, the tone matching unit is used to:
[0207] Acquire a color distribution histogram of the color-adjusting reference image as a reference histogram;
[0208] The color distribution histogram of each second image in the second video is adjusted according to the reference histogram, and the second video after the histogram adjustment is used as the adjusted second video. In an optional example, the number of the color reference images is at least two, and the tone matching unit is used to:
[0209] During the shooting of the second video, at least two color tone selection controls are displayed on a video preview interface of the second video, wherein the video preview interface includes a captured video image, and one color tone selection control corresponds to one color reference image;
[0210] In response to a preview operation on a hue selection control, adjusting the hue of a video image displayed on the video preview interface based on a color reference image corresponding to the hue selection control selected by the preview operation;
[0211] In response to a selection operation on a target color tone selection control in the color tone selection controls, determining a target color reference image corresponding to the target color tone selection control;
[0212] The color tone of each second image of the second video is adjusted based on the target color reference image to obtain an adjusted second video.
[0213] In an optional example, the device further includes a composition prompting unit, configured to:
[0214] Performing composition analysis on the first image of the first video to determine a target display area of the first photographed object in the first image;
[0215] During the shooting of the second video, obtaining an actual display area of the second shooting object in the second image of the second video;
[0216] Based on the actual display area and the target display area, shooting prompt information is output, and the shooting prompt information is used to prompt adjustment of shooting parameters of the second video so that the position of the actual display area in the second image matches the position of the target display area in the first image.
[0217] By using the device of the embodiment of the present application, color feature analysis can be performed on at least two frames of first images in a first video to obtain color features of the first images; the color richness of the first images can be compared according to the color features, and a color reference image can be selected from the first images based on the comparison results; based on the hue of the color reference image, the hue of each second image of the second video can be adjusted to obtain an adjusted second video; the first video and the adjusted second video can be synthesized to obtain a combined video, thereby quickly selecting a more representative color reference image in the first video, thereby quickly adjusting the hue of the image of the second video.
[0218] In addition, an embodiment of the present invention further provides a computer device, which may be a terminal or a server. Figure 4 As shown, it shows a schematic diagram of the structure of a computer device involved in an embodiment of the present invention, specifically:
[0219] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0220] The processor 401 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the computer device and processes data, thereby performing overall detection of the computer device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.
[0221] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
[0222] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0223] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0224] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:
[0225] Performing color feature analysis on at least two frames of first images in the first video to obtain color features of the first images;
[0226] comparing the color richness of the first image according to the color feature, and selecting a color adjustment reference image from the first image based on the comparison result;
[0227] Based on the hue of the color-adjusting reference image, adjusting the hue of each second image of the second video to obtain an adjusted second video;
[0228] A synthesis operation is performed on the first video and the adjusted second video to obtain a combined video.
[0229] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0230] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0231] To this end, an embodiment of the present invention further provides a storage medium, in which a plurality of instructions are stored. The instructions can be loaded by a processor to execute the video synthesis method provided by the embodiment of the present invention.
[0232] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.
[0233] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0234] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0235] Since the instructions stored in the storage medium can execute the steps in the video synthesis method provided in the embodiment of the present invention, the beneficial effects that can be achieved by the video synthesis method provided in the embodiment of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0236] The above is a detailed introduction to a video synthesis method, device, computer equipment and storage medium provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A video synthesis method, characterized in that: include: Performing color feature analysis on at least two frames of first images in a first video to obtain color features of the first images, including: performing color mapping on at least two frames of first images in the first video, mapping a first color of a pixel in each of the first images to a second color, wherein the first color is represented in a triplet form and the second color is represented in a binary form; counting the number of pixels of each second color in the same first image, and obtaining color features of the first image based on the second color and the number of pixels of the second color, wherein the number of pixels of a second color is the number of pixels of the first color mapped to the second color; comparing the color richness of the first image according to the color feature, and selecting a color adjustment reference image from the first image based on the comparison result; Based on the hue of the color-adjusting reference image, adjusting the hue of each second image of the second video to obtain an adjusted second video; A synthesis operation is performed on the first video and the adjusted second video to obtain a combined video.
2. The video synthesis method according to claim 1, characterized in that: The step of performing color feature analysis on at least two frames of first images in the first video to obtain color features of the first images further includes: Obtain the trained image analysis model; Acquire at least two first image frames of a first video; Extracting color features of the first image based on the image analysis model to obtain a color feature vector of the first image; The step of comparing the color richness of the first image according to the color feature and selecting a color adjustment reference image from the first image based on the comparison result includes: Performing color richness classification on the color feature vector by using the image analysis model to obtain classification probabilities of the image analysis model under at least two preset color richness classifications; The color adjustment reference image is determined based on the classification probability of the first image in each color richness classification.
3. The video synthesis method according to claim 1, characterized in that: The step of comparing the color richness of the first image according to the color feature and selecting a color adjustment reference image from the first image based on the comparison result includes: Based on the second color of the first image and the number of pixels of the second color, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
4. The video synthesis method according to claim 1, characterized in that: The triplet is an RGB three-primary color triplet, and the color mapping is performed on at least two frames of the first image in the first video to map the first color of the pixel points in each of the first images to a second color, including: Obtain a color lookup table, wherein the color lookup table is used to map a color representation in the form of an RGB three-primary color triplet to a color representation in the form of a two-tuple; Acquire at least two frames of first images in a first video; In the color lookup table, a second color corresponding to the first color of a pixel in the first image is searched.
5. The video synthesis method according to claim 3, characterized in that: The comparing the color richness of the first image based on the second color of the first image and the number of pixels of the second color, and selecting a color adjustment reference image from the first image based on the comparison result, comprises: Filtering out the second color whose number of pixels is lower than a preset number threshold from the color features of the first image to obtain filtered color features; The color richness of the first images is compared according to the number of second colors in the filtered color features of each of the first images, and a color adjustment reference image is selected from the first images based on the comparison result.
6. The video synthesis method according to claim 1, characterized in that: The color feature includes a color distribution histogram, and the color feature analysis of at least two frames of the first image in the first video to obtain the color feature of the first image also includes: Acquire at least two frames of first images in a first video; A color distribution histogram analysis is performed on each of the first images to obtain a color distribution histogram of each of the first images.
7. The video synthesis method according to claim 6, characterized in that: The step of comparing the color richness of the first image according to the color feature and selecting a color adjustment reference image from the first image based on the comparison result includes: According to the proportion of each color component in the color distribution histogram of the first image in the first image, the color richness of the first image is compared, and a color adjustment reference image is selected from the first image based on the comparison result.
8. The video synthesis method according to any one of claims 1 to 7, characterized in that: The step of adjusting the hue of each second image of the second video based on the hue of the color-adjusting reference image to obtain the adjusted second video includes: Acquire a color distribution histogram of the color-adjusting reference image as a reference histogram; The color distribution histogram of each second image in the second video is adjusted according to the reference histogram, and the second video after the histogram is adjusted is used as the adjusted second video.
9. The video synthesis method according to any one of claims 1 to 7, characterized in that: The number of the color-adjusting reference images is at least two, and the color-adjusting of each second image of the second video based on the color tone of the color-adjusting reference images to obtain the adjusted second video includes: During the shooting of the second video, at least two color tone selection controls are displayed on a video preview interface of the second video, wherein the video preview interface includes a captured video image, and one color tone selection control corresponds to one color reference image; In response to a preview operation on a hue selection control, adjusting the hue of a video image displayed on the video preview interface based on a color reference image corresponding to the hue selection control selected by the preview operation; In response to a selection operation on a target color tone selection control in the color tone selection controls, determining a target color reference image corresponding to the target color tone selection control; The color tone of each second image of the second video is adjusted based on the target color reference image to obtain an adjusted second video.
10. The video synthesis method according to any one of claims 1 to 7, characterized in that: Also includes: Performing composition analysis on the first image of the first video to determine a target display area of the first photographed object in the first image; During the shooting of the second video, obtaining an actual display area of the second shooting object in the second image of the second video; Based on the actual display area and the target display area, shooting prompt information is output, and the shooting prompt information is used to prompt adjustment of shooting parameters of the second video so that the position of the actual display area in the second image matches the position of the target display area in the first image.
11. A video synthesis device, characterized in that: include: A color feature extraction unit, configured to perform color feature analysis on at least two frames of first images in a first video to obtain color features of the first images, including: performing color mapping on at least two frames of first images in the first video, mapping a first color of a pixel in each of the first images to a second color, wherein the first color is represented in a triplet form and the second color is represented in a binary form; counting the number of pixels of each second color in the same first image, and obtaining the color features of the first image based on the second color and the number of pixels of the second color, wherein the number of pixels of a second color is the number of pixels of the first color mapped to the second color; an image selection unit, configured to compare the color richness of the first image according to the color feature, and select a color adjustment reference image from the first image based on the comparison result; A tone matching unit, configured to adjust the tone of each second image of the second video based on the tone of the color-adjusting reference image to obtain an adjusted second video; The synthesis unit is used to perform a synthesis operation on the first video and the adjusted second video to obtain a synthesized video.
12. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 10 when executing the computer program.
13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Image processing method and device, image synthesis method and device, and storage medium
CN111652123A
Video processing method and device, electronic equipment and storage medium
CN111935418A
Image processing method and device, electronic equipment and storage medium
CN111954058A
Reference image selection for motion ghost filtering
US20140307044A1
