Video processing method and device

Abstract

The embodiment of the invention provides a video processing method and equipment. The method comprises the following steps: acquiring a video file of a to-be-processed video; according to the metadata information and the offset information, performing background blurring processing on the subject to be subjected to image matting in the video to be processed to obtain the video to be processed after background blurring; performing image detection processing on the to-be-processed video after background blurring to obtain an image area corresponding to the to-be-keyed subject; and determining an image matting video corresponding to the subject to be subjected to image matting in the video to be processed according to the image region corresponding to the subject to be subjected to image matting. According to the technical scheme, the foreground and the background of the video can be accurately distinguished through the offset information, some errors possibly existing when the offset information is distinguished are avoided through the main body marks in the metadata information, the main body position determining accuracy is improved, and therefore the purpose of conducting main body matting from the video more accurately is achieved.

Description

TECHNICAL FIELD

[0001] The embodiment of the present disclosure relates to the technical field of computer and network communication, in particular to a video processing method and device. BACKGROUND

[0002] With the development of science and technology, video processing technology has been widely applied in various fields. Video matting, as a key technology of video processing, its main task is to separate a certain object from the background in the video, so as to carry out subsequent editing, synthesis or special effect processing, etc. Video matting technology has important application value in film production, advertisement design, virtual reality, game development and other fields.

[0003] The existing common matting scheme obtains the video, and takes the pixel information of the decoded frame image in the video as the difference between the to-be-cut region and the background region, so as to realize video matting.

[0004] However, the existing matting scheme only takes the size of the pixel information of the decoded frame image as the judgment element, which has the problem of insufficient edge detection accuracy, so that the video matting is not accurate. SUMMARY

[0005] The embodiment of the present disclosure provides a video processing method and device to solve the technical problem of inaccurate video matting.

[0006] In a first aspect, the embodiment of the present disclosure provides a video processing method, comprising:

[0007] obtaining a video file of a to-be-processed video, the video file comprising offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being a pixel offset of a same position between adjacent frame images of the to-be-processed video, and the metadata information being used for marking a to-be-matted subject of the to-be-processed video;

[0008] performing background blurring processing on the to-be-matted subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video;

[0009] performing image detection processing on the background-blurred to-be-processed video, to obtain an image region corresponding to the to-be-matted subject;

[0010] determining a to-be-matted video corresponding to the to-be-matted subject in the to-be-processed video according to the image region corresponding to the to-be-matted subject.

[0011] In a second aspect, the embodiment of the present disclosure provides a video processing device, comprising:

[0012] An acquisition unit is configured to acquire a video file of a to-be-processed video, the video file comprising offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being a pixel offset between the same position of adjacent frame images of the to-be-processed video, and the metadata information being used to mark a to-be-cut subject of the to-be-processed video.

[0013] A first processing unit is configured to perform background blurring processing on the to-be-cut subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video.

[0014] A second processing unit is configured to perform image detection processing on the background-blurred to-be-processed video, to obtain an image region corresponding to the to-be-cut subject.

[0015] A third processing unit is configured to determine a cut video corresponding to the to-be-cut subject in the to-be-processed video according to the image region corresponding to the to-be-cut subject.

[0016] In a third aspect, an electronic device is provided, comprising a processor and a memory.

[0017] The memory stores computer-executable instructions.

[0018] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video processing method as designed in the first aspect.

[0019] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the video processing method as designed in the first aspect is implemented.

[0020] In a fifth aspect, a computer program product is provided, and the computer program product comprises a computer program. When a processor executes the computer program, the video processing method as designed in the first aspect is implemented.

[0021] The video processing method and device provided in the embodiment, the method comprises the following steps: obtaining a video file of a to-be-processed video, the video file comprising offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being a pixel offset between the same position of adjacent frame images of the to-be-processed video, and the metadata information being used for marking a to-be-cut subject of the to-be-processed video; performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the metadata information and the offset information, to obtain a to-be-processed video after background blurring; performing image detection processing on the to-be-processed video after background blurring, to obtain an image region corresponding to the to-be-cut subject; and determining a cutout video corresponding to the to-be-cut subject in the to-be-processed video according to the image region corresponding to the to-be-cut subject. In the technical solution, the offset information can be used to accurately distinguish the foreground and the background of the video, and the metadata information is used for marking the subject to avoid some errors in the offset information when distinguishing the foreground and the background, so that the accuracy of determining the position of the subject is improved, and the purpose of more accurately cutting out the subject from the video is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0023] Figure 1 The video processing method flowchart provided in the embodiment of the present disclosure Figure One ;

[0024] Figure 2 The to-be-cut subject selection diagram in the movie mode provided in the embodiment of the present disclosure

[0025] Figure 3 The cutout video corresponding to the to-be-cut subject provided in the embodiment of the present disclosure

[0026] Figure 4 The target video diagram provided in the embodiment of the present disclosure

[0027] Figure 5 The video processing method flowchart provided in the embodiment of the present disclosure Figure Two ;

[0028] Figure 6 The structure diagram of the video processing device provided in the embodiment of the present disclosure

[0029] Figure 7 The structure diagram of the electronic device provided in the embodiment of the present disclosure DETAILED DESCRIPTION

[0030] So that the purposes, technical solutions and advantages of the embodiments of the present disclosure are more apparent, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present disclosure.

[0031] With the development of technology, video processing technology has been widely applied in various fields. Video matting, as a key technology in video processing, its main task is to separate a certain object from the background in the video, so as to carry out subsequent editing, synthesis or special effect processing, etc. Video matting technology has important application value in film production, advertisement design, virtual reality, game development and other fields.

[0032] The existing common matting scheme only obtains the pixel information of the decoded picture as the element for distinguishing the subject and the background.

[0033] However, in the above-mentioned scenario, the edge detection may not be accurate enough, resulting in an inaccurate subject video.

[0034] To solve the above-mentioned technical problems, the technical idea of the inventor is as follows: if the Disparity in the video can be obtained during the shooting process, the Disparity can assist in distinguishing the foreground and the background in the video, and then based on the movie mode, the subject can be selected on the shooting device, and the information is also retained during shooting, that is, MetaData, which contains the marking of the subject in the video, can improve the accuracy of identifying the subject, so as to realize more accurate matting of the subject in the video.

[0035] Among them, Disparity is a concept used in stereo vision, which refers to the displacement of corresponding pixels between two adjacent images. By calculating the Disparity between images under different viewing angles, the depth information of the object can be obtained, which can help distinguish the foreground and the background in the video; Metadata contains information such as the position, shape and motion trajectory of the subject, which can provide assistance for more accurate identification of the subject in subsequent steps.

[0036] The execution subject of the embodiments of the present disclosure is an electronic device, which can be a mobile phone, a notebook computer, a computer, a server, etc.

[0037] The following are related embodiments of the video processing method provided by the present disclosure.

[0038] Figure 1 A video processing method flow provided by an embodiment of the present disclosure Figure One As shown in the figure, the video processing method comprises: Figure 1

[0039] Step 11, obtaining a video file of a to-be-processed video;

[0040] The video file comprises offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being a pixel offset of a same position between adjacent frame images of the to-be-processed video, and the metadata information being used for marking a to-be-cut subject of the to-be-processed video;

[0041] In this step, the to-be-processed video for subject cutout, the pixel offset of each same position between adjacent frame images in the to-be-processed video, and the subject marked in each frame image of the to-be-processed video are obtained from a preset physical storage space, and the to-be-cut subject is as follows.

[0042] The above acquisition process can be a production link, based on a cinematic framework (English: Cinematic Framework), and specifically can be an application programming interface (Application Programming Interface, API) for editing a scene to process specific shooting materials.

[0043] That is, by invoking the API, the electronic device can perform corresponding processing on the to-be-processed file to obtain the offset information and the metadata information. Specifically, in the Cinematic Framework, the Metadata of the video file can be read by means of the AV Foundation framework, and the Metadata information in the video file can be obtained and analyzed by means of a related function class; by means of a related API in the Cinematic Framework, such as a depth (English: Depth) API, the pixel offset of each same position between adjacent frame images in the video can be accessed.

[0044] Optionally, one possible implementation of obtaining the metadata information of the to-be-processed video can be that, in the process of shooting the video, the metadata information is obtained in response to a preset operation on the to-be-cut subject, and the preset operation comprises clicking, selecting, and marking.

[0045] In some implementations, when the video is being shot, the user can manually click on a subject (i.e., the to-be-cut subject) in a graphical user interface that needs to be cut out later, and the device responds to the clicking operation to mark the to-be-cut subject in the shooting and generate corresponding metadata information.

[0046] In addition, in the process of shooting, the to-be-cut subject can also be transformed, for example,​Figure 2 The selection diagram of the subject to be cut out in the movie mode provided by the embodiments of the present disclosure is shown in FIG. 1. As shown in the diagram, the shooting starts at the first second, the user clicks to select the subject A, and at the 63rd second, the user clicks to select the subject B. Figure 2

[0047] The background C in the video is also included in the diagram.

[0048] It should be understood that the number of the subjects to be cut out selected at the same time can be one or more, or none.

[0049] In the above implementation, since the original information of multiple shooting subjects is retained during shooting, the focal point parallax and aperture can be flexibly adjusted during post-processing.

[0050] Step 12, performing background blurring processing on the subject to be cut out in the to-be-processed video according to the metadata information and the offset information, to obtain the background-blurred to-be-processed video;

[0051] In this step, according to the metadata information and the offset information, the subject to be cut out and the background part in the to-be-processed video can be distinguished, and then the background part is blurred to obtain the background-blurred to-be-processed video, which can be processed more easily and accurately in the subsequent step.

[0052] In the embodiments of the present disclosure, the following several possible implementations of step 12 are provided: the image gradient method, see the embodiment shown in mode one of step 52 in FIG. 2; the frequency domain distribution method, see the embodiment shown in mode two of step 52 in FIG. 3; and the picture contrast method, see the embodiment shown in mode three of step 52 in FIG. 4. Figure 5 Figure 5 Figure 5

[0053] Step 13, performing image detection processing on the background-blurred to-be-processed video to obtain the image region corresponding to the subject to be cut out;

[0054] In this step, the subject detection can be performed on each background-blurred image frame using an image segmentation algorithm (such as semantic segmentation, instance segmentation, etc.), and the above algorithm can be implemented through a deep learning model such as Mask R-CNN, U-Net, etc.

[0055] Then, according to the subject detection result, the image region corresponding to the subject to be cut out in each image frame is extracted, and the subject region can be extracted using binarization, contour detection, etc.

[0056] ​​​​Optionally, if the subject in the video to be processed is continuously moving, a target tracking algorithm (such as Kalman filtering, optical flow method, etc.) can be used to track the position of the subject, so as to accurately determine the image area corresponding to the subject to be cut out in different image frames.

[0057] Step 14. Determine the cut-out video corresponding to the subject to be cut out in the video to be processed according to the image area corresponding to the subject to be cut out.

[0058] In this step, the image corresponding to the subject to be cut out is determined in the video to be processed frame by frame according to the image area corresponding to the subject to be cut out in each frame, and then the cut-out video is obtained.

[0059] That is, each frame after processing is recombined into a complete video sequence to obtain the cut-out video.

[0060] Optionally, one specific implementation scenario under this embodiment is as follows:

[0061] Figure 3 The schematic diagram of the cut-out video corresponding to the subject to be cut out provided by the embodiment of the present disclosure is shown in FIG. 5, which includes frame images at 05 seconds and 1 minute 06 seconds as examples, including subject A and subject B. Figure 3

[0062] Further, the video processing method can also synthesize a target video according to the preset material and the cut-out video corresponding to the subject to be cut out, and the target video is a video in which the subject to be cut out is the foreground and the material is the background.

[0063] Under this implementation, the subject can be cut out when the video to be processed collected in the movie mode is edited again, and the preset material can be used as the background after the background processing.

[0064] For example, Figure 4 The schematic diagram of the target video provided by the embodiment of the present disclosure is shown in FIG. 6, which includes frame images at 05 seconds and 1 minute 06 seconds as examples, including subject A and subject B, and further including background D (i.e., the preset material) in the video. Figure 4

[0065] ​​The video processing method provided in this disclosure involves acquiring a video file of a video to be processed. The video file includes offset information and metadata information of the video to be processed. The offset information is the pixel offset between adjacent frames of the video at the same position, and the metadata information is used to mark the subject to be cut out in the video. Based on the metadata information and offset information, the subject to be cut out in the video is subjected to background blurring processing to obtain a blurred video. Image detection processing is then performed on the blurred video to obtain the image region corresponding to the subject to be cut out. Based on the image region corresponding to the subject to be cut out, the cut-out video corresponding to the subject is determined in the video to be processed. In this technical solution, the offset information can accurately distinguish between the foreground and background of the video, and the subject marker in the metadata information avoids some errors that may exist when using offset information for distinction, improving the accuracy of subject position determination, thereby achieving a more accurate subject cutout from the video.

[0066] Based on the above embodiments, Figure 5 Flowchart of the video processing method provided in the embodiments of this disclosure Figure Two .like Figure 5 As shown, step 12 above may include:

[0067] Step 51: Determine the depth information of the frame images in the video to be processed based on the offset information;

[0068] In this step, the Disparity information represents the pixel offset or displacement difference at the same position between adjacent frame images. In order to distinguish the foreground and background of frame images in the video to be processed, the Disparity information can be used to analyze the depth information of objects in the video to be processed.

[0069] This implementation could be as follows: for each frame of the video to be processed, use the AVDepthData class in the Cinematic Framework to parse the Disparity information and obtain the depth information.

[0070] Specifically, Disparity is the pixel offset that causes the difference between the left and right images. By calculating parameters such as Disparity and baseline distance, depth information can be obtained.

[0071] Step 52: Based on the depth information and the labeled subject to be cut out, perform background blurring on the subject to be cut out in the video to be processed, and obtain the video to be processed with the background blurred.

[0072] In this step, after obtaining the depth information, the subject in the video to be processed is marked or identified in combination with the Metadata information, and then the background blurring processing of the video to be processed is realized according to the depth information and the subject marking.

[0073] The background blurring method can be realized based on at least one of the following: image gradient, frequency domain distribution, and picture contrast.

[0074] The gradient refers to the direction and rate of change of pixel intensity in the image. The background is usually relatively static, while the foreground (the subject to be cut out) usually has greater gradient changes. By analyzing the gradient of the image, the foreground and background can be identified and segmented, thereby realizing background blurring.

[0075] The frequency domain representation of the image describes the distribution of different frequency components in the image. The background usually has lower frequency components, while the foreground usually has higher frequency components (such as edges and textures). By analyzing the frequency domain distribution of the image, the features of the foreground and background can be extracted to help realize background blurring.

[0076] Contrast refers to the degree of difference in pixel intensity in the image. The background usually has lower contrast, while the subject usually has higher contrast. By analyzing the contrast information of the image, the boundaries of the foreground and background can be determined, thereby realizing background blurring.

[0077] Method one: for image gradient, the implementation of step 52 can be: according to the depth information and the marked subject to be cut out, the edge information of the subject to be cut out is extracted in the video to be processed; according to the edge information, the non-subject area in the frame image of the video to be processed is determined; the non-subject area in the frame image of the video to be processed is blurred to obtain the background blurred video to be processed.

[0078] In this implementation, the position of the subject to be cut out can be located in each frame image of the video to be processed according to the subject marking and the depth information. Image segmentation algorithms or deep learning techniques can be used to extract the subject area. Once the subject position is determined, edge detection algorithms (such as Canny edge detection) can be used to extract the edge information of the subject to be cut out, which can help determine the boundary between the subject to be cut out and the background.

[0079] Further, for each frame image, according to the edge information of the subject to be cut out, the non-subject area can be determined, and the non-subject area is selected to be blurred, i.e. the background part is set to be transparent or other ways are used to distinguish the main image area, and then the background blurred video to be processed is obtained.

[0080] Secondly, for the frequency domain distribution, the implementation of the step 52 can be: determining the frequency domain distribution of the frame images in the video to be processed; adjusting the frequency domain distribution weight in the frequency domain distribution according to the labeled subject to be cut and the depth information, obtaining the frequency component of the subject, and performing the virtualization processing on the frequency component of the non-subject in the video to be processed to obtain the background-virtualized video to be processed.

[0081] In this implementation, the image in the video to be processed can be converted into a frequency domain representation using Fourier transform or other frequency domain transform methods (such as discrete cosine transform), the weight in the frequency domain distribution is adjusted according to the subject label and the depth information, the frequency component in the subject area can be adjusted by increasing or decreasing to adjust the frequency domain difference between the subject and the background, the weight of the subject area is adjusted to a higher value, and the weight of the background area is adjusted to a lower value to highlight the subject.

[0082] Further, the frequency domain distribution is weighted using the adjusted weight, the frequency component of the subject area is retained, the frequency component of the background area is suppressed, the frequency component of the background part is filtered or adjusted to virtualize the background, and the background-virtualized video to be processed is obtained.

[0083] Thirdly, for the picture contrast, the implementation of the step 52 can be: determining the subject area and the background area of the frame images in the video to be processed according to the labeled subject to be cut and the depth information; adjusting the contrast difference between the subject area and the background area until the contrast of the subject area reaches a preset threshold to obtain the background-virtualized video to be processed.

[0084] In this implementation, the position of the subject area and the background area in each frame image is determined according to the subject label and the depth information (this implementation is based on the foreground and the background, and the subject area is selected according to the subject label, and the others are the background area), and the subject label can be the subject position information obtained by an image segmentation algorithm, a deep learning technology, etc.; according to the depth information, the area between the subject and the background can be determined by threshold setting or depth range screening.

[0085] Further, the contrast difference between the subject area and the background area is adjusted using image processing techniques such as histogram equalization, contrast enhancement, etc. The contrast of the subject area is increased, and the contrast of the background area is reduced to highlight the subject and weaken the background content (this process can be implemented based on a preset threshold). For each frame image, the contrast difference is adjusted according to the adjusted contrast difference until the contrast of the subject area reaches a preset threshold (a threshold that can obviously distinguish the subject area and the background area is set) to obtain the background-virtualized video to be processed.

[0086] The video processing method provided by the embodiment of the present disclosure comprises: determining depth information of frame images in a to-be-processed video according to offset information; and performing background blurring processing on a to-be-cut subject in the to-be-processed video according to the depth information and the to-be-cut subject marked, to obtain a to-be-processed video after background blurring. In the technical solution, the non-cut subject part is subjected to background blurring, so as to facilitate subsequent extraction of images in the region corresponding to the cut subject.

[0087] On the basis of the method embodiment, Figure 6 The structural schematic diagram of the video processing device provided by the embodiment of the present disclosure is shown in the figure, Figure 6 The video processing device comprises:

[0088] The acquisition unit 61 is configured to acquire a video file of a to-be-processed video, wherein the video file comprises offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information is a pixel offset of a same position between adjacent frame images of the to-be-processed video, and the metadata information is used for marking a to-be-cut subject of the to-be-processed video;

[0089] The first processing unit 62 is configured to perform background blurring processing on the to-be-cut subject in the to-be-processed video according to the metadata information and the offset information, to obtain a to-be-processed video after background blurring;

[0090] The second processing unit 63 is configured to perform image detection processing on the to-be-processed video after background blurring, to obtain an image region corresponding to the to-be-cut subject;

[0091] The third processing unit 64 is configured to determine a cut video corresponding to the to-be-cut subject in the to-be-processed video according to the image region corresponding to the to-be-cut subject.

[0092] In one or more embodiments of the present disclosure, the first processing unit 62 is specifically configured to:

[0093] determine depth information of frame images in the to-be-processed video according to the offset information;

[0094] perform background blurring processing on the to-be-cut subject in the to-be-processed video according to the depth information and the to-be-cut subject marked, to obtain a to-be-processed video after background blurring.

[0095] In one or more embodiments of the present disclosure, the first processing unit 62 performs background blurring processing on the to-be-cut subject in the to-be-processed video according to the depth information and the to-be-cut subject marked, to obtain a to-be-processed video after background blurring, and the specific process is as follows:

[0096] extract edge information of the to-be-cut subject in the to-be-processed video according to the depth information and the to-be-cut subject marked;

[0097] According to the edge information, a non-subject region in a frame image of the video to be processed is determined;

[0098] The non-subject region in the frame image of the video to be processed is blurred to obtain the background-blurred video to be processed.

[0099] In one or more embodiments of the present disclosure, the first processing unit 62 performs background blurring processing on the subject to be cut in the video to be processed according to the depth information and the labeled subject to be cut, to obtain the background-blurred video to be processed, specifically:

[0100] The frequency domain distribution of the frame image in the video to be processed is determined;

[0101] According to the labeled subject to be cut and the depth information, the frequency domain distribution weight is adjusted in the frequency domain distribution to obtain the frequency component of the subject;

[0102] The frequency component of the non-subject in the video to be processed is blurred to obtain the background-blurred video to be processed.

[0103] In one or more embodiments of the present disclosure, the first processing unit 62 performs background blurring processing on the subject to be cut in the video to be processed according to the depth information and the labeled subject to be cut, to obtain the background-blurred video to be processed, specifically:

[0104] According to the labeled subject to be cut and the depth information, the subject region and the background region of the frame image in the video to be processed are determined;

[0105] The contrast difference between the subject region and the background region is adjusted until the contrast of the subject region reaches a preset threshold, to obtain the background-blurred video to be processed.

[0106] In one or more embodiments of the present disclosure, the acquisition unit 61 acquires the metadata information of the video to be processed, specifically for:

[0107] During the process of shooting the video, in response to a preset operation on the subject to be cut, the metadata information is acquired, and the preset operation includes: clicking, selecting, and labeling.

[0108] In one or more embodiments of the present disclosure, the third processing unit 64 is further configured to:

[0109] According to the preset material and the cutout video corresponding to the subject to be cut, the target video is synthesized, and the target video is a video in which the subject to be cut is the foreground and the material is the background.

[0110] The video processing device provided in the embodiments of the present disclosure has similar technical solutions and technical effects to the above-mentioned embodiments, and details are not repeated here.

[0111] To achieve the above-mentioned embodiments, the electronic device provided by the embodiments of the present disclosure also provides an electronic device. Figure 7 The structure schematic diagram of the electronic device provided by the embodiments of the present disclosure is shown in Figure 7 The electronic device can be a terminal device.

[0112] The terminal device can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDA), tablet computers (PAD), portable multimedia players (PMP), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0113] As shown in Figure 7 The electronic device can include a processing device (such as a central processor, a graphics processor, etc.) 71, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 72 or programs loaded from a storage device 78 into a random access memory (RAM) 73. Various programs and data required for the operation of the electronic device are also stored in the RAM 73. The processing device 71, the ROM 72, and the RAM 73 are connected to each other through a bus 74. An input / output (I / O) interface 75 is also connected to the bus 74.

[0114] Generally, the following devices can be connected to the I / O interface 75: input devices 76 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 77 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 78 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 79. The communication devices 79 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices can be alternatively implemented or provided.

[0115] In particular, in accordance with embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 79, or installed from the storage device 78, or installed from the ROM 72. When the computer program is executed by the processing device 71, the above-described functions defined in the methods of the embodiments of the present disclosure are executed.

[0116] Note that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example and without limitation, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used or used in conjunction with an instruction execution system, apparatus or device. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that can send, propagate or transmit a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the foregoing.

[0117] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0118] The computer readable medium described above carries one or more programs, which when executed by the electronic device, cause the electronic device to execute the methods shown in the embodiments described above.

[0119] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0120] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0121] The units described in the embodiments of the present disclosure can be implemented by hardware, software, or a combination thereof. In some cases, the names of the units do not constitute a limitation on the units themselves. For example, the first obtaining unit can also be described as a unit that obtains at least two Internet protocol addresses.

[0122] The functions described in this specification can be performed at least in part by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] In a first aspect, according to one or more embodiments of the present disclosure, a video processing method is provided, comprising:

[0125] obtaining a video file of a to-be-processed video, the video file comprising offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being a pixel offset between the same position of adjacent frame images of the to-be-processed video, and the metadata information being used for marking a to-be-chromakeyed subject of the to-be-processed video;

[0126] performing background blurring processing on the to-be-chromakeyed subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video;

[0127] performing image detection processing on the background-blurred to-be-processed video, to obtain an image region corresponding to the to-be-chromakeyed subject;

[0128] determining a chroma key video corresponding to the to-be-chromakeyed subject in the to-be-processed video according to the image region corresponding to the to-be-chromakeyed subject.

[0129] In one or more embodiments of the present disclosure, the performing background blurring processing on the to-be-chromakeyed subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video, comprises:

[0130] determining depth information of frame images in the to-be-processed video according to the offset information;

[0131] performing background blurring processing on the to-be-chromakeyed subject in the to-be-processed video according to the depth information and the to-be-chromakeyed subject marked, to obtain a background-blurred to-be-processed video.

[0132] In one or more embodiments of the present disclosure, the background blurring processing on the subject to be cut out in the to-be-processed video according to the depth information and the labeled subject to be cut out comprises:

[0133] According to the depth information and the labeled subject to be cut out, edge information of the subject to be cut out in the to-be-processed video is extracted;

[0134] According to the edge information, a non-subject region in a frame image of the to-be-processed video is determined;

[0135] The non-subject region in the frame image of the to-be-processed video is subjected to a blurring processing, and the background-blurred to-be-processed video is obtained.

[0136] In one or more embodiments of the present disclosure, the background blurring processing on the subject to be cut out in the to-be-processed video according to the depth information and the labeled subject to be cut out comprises:

[0137] The frequency domain distribution of a frame image in the to-be-processed video is determined;

[0138] According to the labeled subject to be cut out and the depth information, a frequency component of the subject is obtained by adjusting a frequency domain distribution weight in the frequency domain distribution;

[0139] The frequency component other than the subject in the to-be-processed video is subjected to a blurring processing, and the background-blurred to-be-processed video is obtained.

[0140] In one or more embodiments of the present disclosure, the background blurring processing on the subject to be cut out in the to-be-processed video according to the depth information and the labeled subject to be cut out comprises:

[0141] According to the labeled subject to be cut out and the depth information, a subject region and a background region of a frame image in the to-be-processed video are determined;

[0142] The contrast difference between the subject region and the background region is adjusted until the contrast of the subject region reaches a preset threshold, and the background-blurred to-be-processed video is obtained.

[0143] In one or more embodiments of the present disclosure, the metadata information of the to-be-processed video is obtained, comprising:

[0144] In the process of shooting a video, the metadata information is obtained in response to a preset operation on the subject to be cut out, and the preset operation comprises clicking, selecting, and labeling.

[0145] In one or more embodiments of the present disclosure, the method further comprises:

[0146] synthesize a target video according to the preset material and the green screen video corresponding to the green screen subject, the target video being a video in which the green screen subject is foreground and the preset material is background.

[0147] In a second aspect, according to one or more embodiments of the present disclosure, a video processing device is provided, including:

[0148] an acquisition unit configured to acquire a video file of a to-be-processed video, the video file including offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being pixel offsets of a same position between adjacent frame images of the to-be-processed video, and the metadata information being used to mark a green screen subject of the to-be-processed video;

[0149] a first processing unit configured to perform background blurring processing on the green screen subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video;

[0150] a second processing unit configured to perform image detection processing on the background-blurred to-be-processed video, to obtain an image region corresponding to the green screen subject;

[0151] a third processing unit configured to determine a green screen video corresponding to the green screen subject in the to-be-processed video according to the image region corresponding to the green screen subject.

[0152] In one or more embodiments of the present disclosure, the first processing unit is specifically configured to:

[0153] determine depth information of frame images in the to-be-processed video according to the offset information;

[0154] perform background blurring processing on the green screen subject in the to-be-processed video according to the depth information and the labeled green screen subject, to obtain a background-blurred to-be-processed video.

[0155] In one or more embodiments of the present disclosure, the first processing unit performs background blurring processing on the green screen subject in the to-be-processed video according to the depth information and the labeled green screen subject, to obtain a background-blurred to-be-processed video, specifically as follows:

[0156] extract edge information of the green screen subject in the to-be-processed video according to the depth information and the labeled green screen subject;

[0157] determine a non-subject region in frame images of the to-be-processed video according to the edge information;

[0158] The non-main body region in the frame image of the to-be-processed video is blurred to obtain the background-blurred to-be-processed video.

[0159] In one or more embodiments of the present disclosure, the first processing unit performs background blurring on the to-be-chromakeyed main body in the to-be-processed video according to the depth information and the labeled to-be-chromakeyed main body, to obtain a background-blurred to-be-processed video, specifically:

[0160] The frequency domain distribution of the frame image in the to-be-processed video is determined;

[0161] The frequency domain distribution weight is adjusted in the frequency domain distribution according to the labeled to-be-chromakeyed main body and the depth information, to obtain the frequency component of the main body;

[0162] The frequency component of the non-main body in the to-be-processed video is blurred to obtain the background-blurred to-be-processed video.

[0163] In one or more embodiments of the present disclosure, the first processing unit performs background blurring on the to-be-chromakeyed main body in the to-be-processed video according to the depth information and the labeled to-be-chromakeyed main body, to obtain a background-blurred to-be-processed video, specifically:

[0164] The main body region and the background region of the frame image in the to-be-processed video are determined according to the labeled to-be-chromakeyed main body and the depth information;

[0165] The contrast difference between the main body region and the background region is adjusted until the contrast of the main body region reaches a preset threshold, to obtain the background-blurred to-be-processed video.

[0166] In one or more embodiments of the present disclosure, the acquisition unit acquires metadata information of the to-be-processed video, specifically for:

[0167] In the process of shooting the video, the metadata information is acquired in response to a preset operation on the to-be-chromakeyed main body, and the preset operation includes clicking, selecting, and labeling.

[0168] In one or more embodiments of the present disclosure, the third processing unit is further configured to:

[0169] According to the preset material and the chromakeyed video corresponding to the to-be-chromakeyed main body, a target video is synthesized, and the target video is a video in which the to-be-chromakeyed main body is in the foreground and the material is in the background.

[0170] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including at least one processor and a memory;

[0171] The memory stores computer execution instructions;

[0172] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video processing method according to the first aspect and various possible designs of the first aspect.

[0173] In a fourth aspect, a computer-readable storage medium is provided according to one or more embodiments of the present disclosure, and the computer-readable storage medium has stored therein computer-executable instructions that, when executed by a processor, implement the video processing method according to the first aspect and various possible designs of the first aspect.

[0174] In a fifth aspect, a computer program product is provided according to one or more embodiments of the present disclosure, and the computer program product includes a computer program that, when executed by a processor, implements the video processing method according to the first aspect and various possible designs of the first aspect.

[0175] The above description is merely illustrative of the exemplary embodiments of the present disclosure and the principles of the technology involved. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0176] In addition, although each operation is depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments. The various features described in the context of the above embodiments can be combined in any suitable sub-combination.

[0177] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method of video processing, the method comprising: The method comprises the following steps: acquiring a video file of a to-be-processed video, the video file comprising offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being pixel offsets of the same position between adjacent frame images of the to-be-processed video, and the metadata information being used for marking a to-be-cut subject of the to-be-processed video; performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video; performing image detection processing on the background-blurred to-be-processed video, to obtain an image region corresponding to the to-be-cut subject; determining a cut video corresponding to the to-be-cut subject in the to-be-processed video according to the image region corresponding to the to-be-cut subject.

2. The method of claim 1, wherein, The step of performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video, comprises the following steps: determining depth information of frame images in the to-be-processed video according to the offset information; performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the depth information and the labeled to-be-cut subject, to obtain a background-blurred to-be-processed video.

3. The method of claim 2, wherein, The step of performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the depth information and the labeled to-be-cut subject, to obtain a background-blurred to-be-processed video, comprises the following steps: extracting edge information of the to-be-cut subject in the to-be-processed video according to the depth information and the labeled to-be-cut subject; determining a non-subject region in frame images of the to-be-processed video according to the edge information; performing blurring processing on the non-subject region in the frame images of the to-be-processed video, to obtain the background-blurred to-be-processed video.

4. The method of claim 2, wherein, The step of performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the depth information and the labeled to-be-cut subject, to obtain a background-blurred to-be-processed video, comprises the following steps: determining a frequency domain distribution of frame images in the to-be-processed video; adjusting a frequency domain distribution weight in the frequency domain distribution according to the labeled to-be-cut subject and the depth information, to obtain a frequency component of the subject; performing blurring processing on a frequency component other than the frequency component of the subject in the to-be-processed video, to obtain the background-blurred to-be-processed video.

5. The method of claim 2, wherein, The step of performing background blurring processing on the to-be-cut subject in the to-be-processed video according to the depth information and the labeled to-be-cut subject, to obtain a background-blurred to-be-processed video, comprises the following steps: determining a subject region and a background region of frame images in the to-be-processed video according to the labeled to-be-cut subject and the depth information; adjusting a contrast difference between the subject region and the background region until a contrast of the subject region reaches a preset threshold, to obtain the background-blurred to-be-processed video.

6. The method according to any one of claims 1 to 5, characterized in that, The step of acquiring metadata information of the to-be-processed video comprises the following steps: in a process of shooting a video, acquiring the metadata information in response to a preset operation on the to-be-cut subject, the preset operation comprising clicking, selecting, and labeling.

7. The method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps: According to the preset material and the matting video corresponding to the to-be-matted subject, a target video is synthesized, the target video being a video in which the to-be-matted subject is foreground and the material is background.

8. A video processing device, comprising: The method comprises the following steps: An acquisition unit is configured to acquire a video file of a to-be-processed video, the video file comprising offset information of the to-be-processed video and metadata information of the to-be-processed video, the offset information being a pixel offset of a same position between adjacent frame images of the to-be-processed video, and the metadata information being used to mark a to-be-matted subject of the to-be-processed video; A first processing unit is configured to perform background blurring processing on the to-be-matted subject in the to-be-processed video according to the metadata information and the offset information, to obtain a background-blurred to-be-processed video; A second processing unit is configured to perform image detection processing on the background-blurred to-be-processed video, to obtain an image region corresponding to the to-be-matted subject; A third processing unit is configured to determine a matting video corresponding to the to-be-matted subject in the to-be-processed video according to the image region corresponding to the to-be-matted subject.

9. An electronic device, comprising: The method comprises the following steps: A processor and a memory; The memory stores computer-executed instructions; The processor executes the computer-executed instructions stored in the memory, so that the processor executes the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executed instructions, and when the processor executes the computer-executed instructions, the video processing method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the video processing method according to any one of claims 1 to 7.

Images