Scene multi-objective video coding method

Abstract

The application provides a scene multi-target video coding method, comprising the following steps: determining a key target; collecting a video frame F 1t ; processing the video frame F 1t , taking all targets as non-key targets, and obtaining a video frame F 2t ; identifying and intercepting an image of the key target in the video frame F 1t ; performing image sharpening processing on each intercepted image of the key target; merging the processed images of the key targets into the video frame F 1t according to position information of the key targets in the video frame F 2t , and obtaining a merged video frame F' 2t ; adopting interpolation transition for a key target demarcation area of the merged video frame F' 2t , and obtaining a video frame F 3t in which the key target is clear and other targets are relatively blurred. The scene multi-target video coding method provided by the application can ensure the clarity of multiple key targets in a video stream when the video content is in a multi-scene large range.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and more specifically to a method for multi-objective video coding in a scene. Background Technology

[0002] Conventional video recording systems, due to the large amount of content and wide dynamic range of the scene in the video, often struggle to clearly display one or more specific targets of interest to the user in each frame, even with high resolution. Examples include a teacher giving a lesson and the blackboard, a dog moving on the grass, athletes in a sports competition, and a display board. Therefore, they have certain limitations in their use. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a multi-target video coding method for specific scenes, which can effectively solve the aforementioned problems.

[0004] The technical solution adopted in this invention is as follows:

[0005] This invention provides a method for encoding multi-object videos in a scene, comprising the following steps:

[0006] Step S1: Identify the key targets that require focused attention, obtain the key target feature representation vector for each key target, and form a key target feature matrix F = {F1, F2, ..., F...} m}; where F j The feature representation vector of the key target j; j = 1, 2, ..., m, where m is the number of key targets;

[0007] Step S2: Start the recording system to capture video, and acquire video frame F at acquisition time t. 1t ;

[0008] Step S3, for video frame F 1t Processing is performed to convert video frames F 1t All targets are treated as non-priority targets, resulting in reduced sharpness but maintained resolution for all targets, thus obtaining video frame F. 2t ;

[0009] Step S4, based on the key target feature matrix F = {F1, F2, ..., F...} m}, in video frame F 1t The system identifies and extracts images of m key targets of interest, and obtains the image of each identified key target of interest j in video frame F. 1t Image data and its location information;

[0010] Step S5: Perform image sharpening processing on the image of each extracted key target j to obtain the processed image of the key target j;

[0011] Step S6, based on the focus target j in video frame F 1t The location information in the image is used to merge the processed images of the key target j into video frame F. 2t In the process, the merged video frame F' is obtained. 2t ;

[0012] Step S7, for the merged video frame F' 2t Interpolation is used to transition the boundary area of ​​the key target, resulting in video frames F where the key target is clear and other targets are relatively blurry. 3t ;

[0013] Step S8, for video frame F 3t High-bitrate compression encoding is performed to obtain the encoded video frame F. 4t ;

[0014] Step S9, output the encoded video frame F 4t Let t = t + 1; return to step S2.

[0015] Preferably, in step S1, the focus is on the feature representation vector F of target j. j ={F j,visual ,F j,spatial ,F j,temporal};

[0016] in:

[0017] F j,visual To obtain the appearance feature vector of the target j, we extract the color histogram, texture features, and shape description features of the target j.

[0018] F j,spatial The feature vector of the target j is used to describe the positional distribution and movement trajectory pattern of the target j in the image.

[0019] F j,temporal This is the temporal feature vector of the target j, used to describe the temporal pattern and duration statistics of the target j.

[0020] Preferably, step S3 specifically includes:

[0021] Step S3.1, video frame F 1t The width is M pixels and the height is N pixels. The following formula is used to divide the video frame F... 1t Transformation from the spatial domain to the frequency domain:

[0022]

[0023] in:

[0024] T(x,y) represents video frame F 1t The pixel value at position (x,y) in the spatial domain; j is the imaginary unit in the formula; F(u,v) represents the complex coefficients in the frequency domain; the position (x,y) in the spatial domain corresponds to the position (u,v) in the frequency domain;

[0025] Step S3.2: For the complex coefficients F(u,v) in the frequency domain, perform the following filtering operation to retain low-frequency components and remove high-frequency components, obtaining the filtered image H(u,v):

[0026] if Then let H(u,v) = 0; otherwise, let H(u,v) = 1.

[0027] Where: ω_cutoff is the cutoff frequency threshold;

[0028] Step S3.3: Using the following formula, the filtered image H(u,v) and the complex coefficients F(u,v) in the frequency domain are multiplied element-wise to achieve frequency domain filtering, extracting low-frequency information from the image and obtaining the low-frequency information representation F_high(u,v):

[0029] F_high(u,v)=F(u,v)⊙H(u,v)

[0030] Where: ⊙ represents element-wise product;

[0031] Low-frequency information is represented by F_high(u,v), which is the video frame F. 2t .

[0032] Preferably, step S4 specifically includes:

[0033] Step S4.1, in video frame F 1t The target l is identified, and the feature representation vector F of target l is obtained. l ={F l,visual ,F l,spatial ,F l,temporal};

[0034] in:

[0035] F l,visual The appearance feature vector of target l is obtained by extracting the color histogram, texture features, and shape description features of target l.

[0036] F l,spatial is the position feature vector of target l, used to describe the position distribution and movement trajectory pattern of target l in the image;

[0037] F l,temporal This is the temporal feature vector of target l, used to describe the temporal pattern and duration statistics of target l's occurrence;

[0038] Step S4.2, use the following formula to calculate the comprehensive similarity between target j and target l:

[0039] Sim(j,l)=α·Simvisual(j,l)+β·Simspatial(j,l)+γ·Simtime(j,l)

[0040] Where: Simvisual(j,l), Simspatial(j,l), and Simtime(j,l) represent appearance similarity, positional similarity, and temporal similarity, respectively; α, β, and γ represent the appearance similarity weight, positional similarity weight, and temporal similarity weight, respectively; α+β+γ=1, α,β,γ>0; for scenarios where the target appearance changes significantly, increase the value of β; for scenarios where the target appearance is relatively stable, increase the value of α.

[0041] Step S4.3: If the overall similarity Sim(j,l) between target j and target l is greater than the matching threshold θ_match, it means that target j and target l are successfully matched, and target l is determined to be target j.

[0042] Preferably, step S5 specifically involves: sharpening the image of each extracted key target j and adjusting the contrast to obtain an image of the key target j after image sharpening.

[0043] Preferably, step S5 specifically includes:

[0044] Step S5.1: For each extracted image of the key target j, denoted as image I, image I is enhanced using the following formula to obtain the enhanced image I_enhanced:

[0045] I_enhanced=I+λ(II*G_σ)

[0046] Where: λ is the sharpening intensity parameter; G_σ is the Gaussian blur kernel;

[0047] Step S5.2: Perform contrast enhancement on the enhanced image I_enhanced to obtain the image of the focus target j after image sharpening, denoted as I_contrast:

[0048] I_contrast=255(I_enhanced / 255)γ"

[0049] Wherein: γ" is the gamma correction parameter, which is adaptively adjusted according to the target characteristics.

[0050] The multi-target video coding method for scenes provided by this invention has the following advantages:

[0051] The multi-target video coding method provided by this invention can ensure the clarity of multiple key targets in the video stream when the video content covers multiple scenes and a wide range. Attached Figure Description

[0052] Figure 1 The flowchart illustrates the multi-target video coding method for a scene provided by this invention. Detailed Implementation

[0053] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0054] This invention provides a multi-target video coding method for a scene, which is a video coding method that focuses on the adjustable sharpness of the target.

[0055] See Figure 1 This invention provides a method for encoding multi-target video in a scene, comprising the following steps:

[0056] Step S1: Identify the key targets that require focused attention, obtain the key target feature representation vector for each key target, and form a key target feature matrix F = {F1, F2, ..., F...} m}; where F j The feature representation vector of the key target j; j = 1, 2, ..., m, where m is the number of key targets;

[0057] In this step, the focus is on the feature representation vector F of target j. j ={F j,visual ,F j,spatial ,F j,temporal};

[0058] in:

[0059] F j,visual To obtain the appearance feature vector of the target j, we extract the color histogram, texture features, and shape description features of the target j.

[0060] F j,spatialThe feature vector of the target j is used to describe the positional distribution and movement trajectory pattern of the target j in the image.

[0061] F j,temporal This is the temporal feature vector of the target j, used to describe the temporal pattern and duration statistics of the target j.

[0062] Step S2: Start the recording system to capture video, and acquire video frame F at acquisition time t. 1t ;

[0063] Step S3, for video frame F 1t Processing is performed to convert video frames F 1t All targets are treated as non-priority targets, resulting in reduced sharpness but maintained resolution for all targets, thus obtaining video frame F. 2t ;

[0064] Step S3 is as follows:

[0065] Step S3.1, video frame F 1t The width is M pixels and the height is N pixels. The following formula is used to divide the video frame F... 1t Transformation from the spatial domain to the frequency domain:

[0066]

[0067] in:

[0068] T(x,y) represents video frame F 1t The pixel value at position (x,y) in the spatial domain; j is the imaginary unit in the formula; F(u,v) represents the complex coefficients in the frequency domain; the position (x,y) in the spatial domain corresponds to the position (u,v) in the frequency domain;

[0069] Step S3.2: For the complex coefficients F(u,v) in the frequency domain, perform the following filtering operation to retain low-frequency components and remove high-frequency components, obtaining the filtered image H(u,v):

[0070] if Then let H(u,v) = 0; otherwise, let H(u,v) = 1.

[0071] Where: ω_cutoff is the cutoff frequency threshold;

[0072] Step S3.3: Using the following formula, the filtered image H(u,v) and the complex coefficients F(u,v) in the frequency domain are multiplied element-wise to achieve frequency domain filtering, extracting low-frequency information from the image and obtaining the low-frequency information representation F_high(u,v):

[0073] F_high(u,v)=F(u,v)⊙H(u,v)

[0074] Where: ⊙ represents element-wise product;

[0075] Low-frequency information is represented by F_high(u,v), which is the video frame F. 2t .

[0076] Another specific implementation method for this step is:

[0077] For video frame F 1t Lossy compression is performed using a low bitrate compression algorithm, and then the compressed data is decompressed to obtain new image frame data D1. At this time, all targets in the new image frame data D1 are equivalent to being set as non-key targets.

[0078] The compression quality control parameter Q is defined as follows:

[0079] Q = Q_base × (1 - β × I(x,y))

[0080] Where I(x,y) represents the importance of the image region, and β is the adjustment coefficient.

[0081] Step S4, based on the key target feature matrix F = {F1, F2, ..., F...} m}, in video frame F 1t The system identifies and extracts images of m key targets of interest, and obtains the image of each identified key target of interest j in video frame F. 1t Image data and its location information;

[0082] Step S4 is as follows:

[0083] Step S4.1, in video frame F 1t The target l is identified, and the feature representation vector F of target l is obtained. l ={F l,visual ,F l,spatial ,F l,temporal};

[0084] in:

[0085] F l,visual The appearance feature vector of target l is obtained by extracting the color histogram, texture features, and shape description features of target l.

[0086] F l,spatial is the position feature vector of target l, used to describe the position distribution and movement trajectory pattern of target l in the image;

[0087] F l,temporal This is the temporal feature vector of target l, used to describe the temporal pattern and duration statistics of target l's occurrence;

[0088] Step S4.2, use the following formula to calculate the comprehensive similarity between target j and target l:

[0089] Sim(j,l)=α·Simvisual(j,l)+β·Simspatial(j,l)+γ·Simtime(j,l)

[0090] Where: Simvisual(j,l), Simspatial(j,l), and Simtime(j,l) represent appearance similarity, positional similarity, and temporal similarity, respectively; α, β, and γ represent the appearance similarity weight, positional similarity weight, and temporal similarity weight, respectively; α+β+γ=1, α,β,γ>0; for scenarios where the target appearance changes significantly, increase the value of β; for scenarios where the target appearance is relatively stable, increase the value of α.

[0091] Step S4.3: If the overall similarity Sim(j,l) between target j and target l is greater than the matching threshold θ_match, it means that target j and target l are successfully matched, and target l is determined to be target j.

[0092] Step S5: Perform image sharpening processing on the image of each extracted key target j to obtain the processed image of the key target j;

[0093] In this step, the image of each extracted key target j is sharpened and the contrast is adjusted to obtain the image of the key target j after image sharpening.

[0094] One specific implementation method is as follows:

[0095] Step S5.1: For each extracted image of the key target j, denoted as image I, image I is enhanced using the following formula to obtain the enhanced image I_enhanced:

[0096] I_enhanced=I+λ(II*G_σ)

[0097] Where: λ is the sharpening intensity parameter; G_σ is the Gaussian blur kernel;

[0098] Step S5.2: Perform contrast enhancement on the enhanced image I_enhanced to obtain the image of the focus target j after image sharpening, denoted as I_contrast:

[0099] I_contrast=255(I_enhanced / 255)γ"

[0100] Wherein: γ" is the gamma correction parameter, which is adaptively adjusted according to the target characteristics.

[0101] Step S6, based on the focus target j in video frame F 1t The location information in the image is used to merge the processed images of the key target j into video frame F. 2t In the process, the merged video frame F' is obtained. 2t ;

[0102] Step S7, for the merged video frame F' 2t Interpolation is used to transition the boundary area of ​​the key target, resulting in video frames F where the key target is clear and other targets are relatively blurry. 3t ;

[0103] Step S8, for video frame F 3t High-bitrate compression encoding is performed to obtain the encoded video frame F. 4t ;

[0104] Specifically, the video frame F 3t The input is a standardized video compressor, which outputs a streaming video stream. This stream features high-definition processing of key targets and low-definition processing of non-key data, making it easier for viewers to obtain the information that is of interest to the video terminal.

[0105] Step S9, output the encoded video frame F 4t Let t = t + 1; return to step S2.

[0106] This invention discloses a method for adjusting the sharpness of multiple key targets in video encoding. Video stream data, composed of multiple video frames, is obtained through recording, screenshotting, or other acquisition methods. The original video frames are subjected to lossy compression encoding and then decoded to obtain frame data with lower sharpness but maintained resolution. Then, using artificial intelligence methods, multiple pre-input key targets are extracted from the original video frames to obtain image data and location information of the key targets. This image data and location information of the key targets are then merged into the lower-sharp frame data. Due to the difference in sharpness, interpolation is used to transition the boundary areas of the key targets in the merged frame data. Finally, frame data with sharp key targets and relatively blurred other targets is obtained. These frame data are then subjected to high-bitrate compression encoding to obtain a video stream with sharp key targets. This invention can solve the problem of ensuring the sharpness of multiple key targets in a video stream when the video content is extensive and covers a wide range of scenes.

[0107] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of scene multi-objective video coding, characterized in that, The method comprises the following steps: Step S1: Identify the key targets that require focused attention, obtain the key target feature representation vector for each key target, and form a key target feature matrix F = {F1, F2, ..., F...} m }; where F j Let F represent the feature representation vector of the target j; j = 1, 2, ..., m, where m is the number of targets; where F is the feature representation vector of the target j. j ={F j,visual ,F j,spatial ,F j,temporal }; where: F j,visual To focus on the appearance feature vector of target j, it is obtained by extracting the color histogram, texture features, and shape description features of target j; F j,spatial F is the positional feature vector of the target j, used to describe the positional distribution and movement trajectory pattern of the target j in the image; j,temporal This is the temporal feature vector of the target j, used to describe the temporal pattern and duration statistics of the target j. Step S2, starting the video recording system to take a picture, and collecting the video frame F at the collection time t 1t ; Step S3, processing the video frame F 1t , all targets in the video frame F 1t are regarded as non-focus targets, the definition of all targets is reduced but the resolution is kept, and a video frame F 2t is obtained. Step S4, based on the key target feature matrix F = {F1, F2,..., Fm}, identify and intercept the images of m key targets in the video frame F m 1t , and obtain the image data and position information of each identified key target j in the video frame F 1t ;​ Step S5, image sharpening is performed on each intercepted image of the focus target j to obtain a processed image of the focus target j; Step S6, according to the position information of the focused target j in the video frame F 1t , the image of each processed focused target j is merged into the video frame F 2t , and a merged video frame F' 2t is obtained. Step S7, the merged video frame F 2t is interpolated to obtain a video frame F 3t in which the target of focus is clear and other targets are relatively blurred. Step S8, compressing and encoding the video frame F 3t at a high code rate to obtain an encoded video frame F 4t ; Step S9, output the encoded video frame F 4t t = t + 1; go back to step S2.

2. The scene multi-objective video coding method of claim 1, wherein, Step S3 is specifically: Step S3.1, video frame F 1t of width M pixels and height N pixels is converted from the spatial domain to the frequency domain using the following equation: 1t F (u, v) = F (x, y) * G (u, v) Wherein: T(x, y) represents the video frame F 1t Pixel value at (x, y) position in spatial domain; j is imaginary unit in the formula; F(u, v) represents complex coefficient in frequency domain; (x, y) position in spatial domain corresponds to (u, v) position in frequency domain; Step S3.2, for the complex coefficient F(u, v) in the frequency domain, the following filtering operation is performed to retain the low-frequency components and filter out the high-frequency components to obtain a filtered image H(u, v): If then let H(u, v) = 0; otherwise, let H(u, v) = 1; Wherein: ω_cutoff is a cutoff frequency threshold; Step S3.3, the filtered image H(u, v) and the complex coefficient F(u, v) in the frequency domain are multiplied element by element to realize frequency domain filtering and extract low-frequency information in the image to obtain low-frequency information representation F_high(u, v) according to the following formula: F_high(u, v) = F(u, v) e H(u, v) Wherein: e represents element by element multiplication; The low frequency information F_high(u,v) is represented by the video frame F 2t .

3. The scene multi-objective video coding method of claim 1, wherein, Step S4 is specifically: Step S4.1, in the video frame F 1t target I is identified, a feature representation vector F l l,visual l,spatial l,temporal is obtained for the target I;​​​ Wherein: F l,visual The appearance feature vector of the target l is obtained by extracting the color histogram, texture feature, and shape description feature of the target l. F l,spatial Position feature vector of target l, used to describe the position distribution and moving track pattern of target l in the picture; F l,temporal Time feature vector for target l, describing the temporal pattern and duration statistics of the appearance of target l; Step S4.2, the following formula is used to calculate the comprehensive similarity between the focus target j and the target l: Sim(j, l) = α · Simvisual(j, l) + β · Simspatial(j, l) + γ · Simtime(j, l) Wherein: Simvisual(j, l), Simspatial(j, l) and Simtime(j, l) are appearance similarity, position similarity and time similarity respectively; α, β and γ are appearance similarity weight, position similarity weight and time similarity weight respectively; α + β + γ = 1, α, β, γ > 0; for scenes with large target appearance changes, increase the value of β; for scenes with relatively stable target appearance, increase the value of α; Step S4.3, if the comprehensive similarity Sim(j, l) between the focus target j and the target l is greater than the matching threshold θ_match, it means that the focus target j and the target l match successfully, and the target l is determined as the focus target j.

4. The scene multi-objective video coding method of claim 1, wherein, Step S5 is specifically: sharpening is performed on each intercepted image of the focus target j to obtain an image of the focus target j after image sharpening.

5. The scene multi-objective video coding method of claim 1, wherein, Step S5 is specifically: Step S5.1, for each intercepted image of the focus target j, it is represented as image I; image enhancement is performed on the image I according to the following formula to obtain an image I_enhanced after image enhancement: I_enhanced = I + λ(I - I * G_σ) Wherein: λ is the sharpening intensity parameter; G_σ is the Gaussian blur kernel; Step S5.2, contrast enhancement is performed on the image I_enhanced after image enhancement to obtain an image of the focus target j after image sharpening, represented as I_contrast: I_contrast = 255(I_enhanced / 255)γ" Wherein: γ" is a gamma correction parameter, which is adaptively adjusted according to the target characteristics.

Images