Video processing method, apparatus, storage medium, device, and program product

Abstract

The application discloses a video processing method, device, storage medium, equipment and program product, and the method comprises the steps of determining a first image pair sequence and a horizontal mirror image pair sequence according to a video image sequence of a target video; determining a target difference value image sequence according to the first image pair sequence and the horizontal mirror image pair sequence; performing straight line detection on each target difference value image in the target difference value image sequence to obtain a first straight line corresponding to each target difference value image; determining an image effective area coordinate of the target video according to a cosine distance between each first straight line and a horizontal straight line; obtaining a face detection result corresponding to each video image; determining a video effective picture coordinate of the target video according to the face detection result and the video effective area coordinate, and determining a video picture frame based on the video effective picture coordinate, so that the calculation precision of the video picture frame is improved, and the experience of a user watching the video is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, in particular to a video processing method and device, a storage medium, equipment and a program product. BACKGROUND

[0002] The video picture frame refers to a frame in which all video picture contents in a whole video time span are within the frame. There is usually a certain interval space between the video picture frame and the whole video canvas (the size of the real picture), and the interval space can be filled with a pure color picture (such as the upper and lower black edge color bars of a movie) or a color picture (such as the current popular small video). With the rise of small videos, and the public spending more and more time on mobile devices such as mobile phones to watch videos, and being used to editing the upper and lower boundaries of the original video picture when publishing the video, adding a pure color or color changing picture and the like.

[0003] Due to the limited screen size of mobile devices such as mobile phones, the additional picture edited by the user will cause a "shrinking" effect on the presentation of the existing picture, which will affect the user's experience of watching the video. SUMMARY

[0004] The embodiments of the present application provide a video processing method and device, a storage medium, equipment and a program product, which can improve the calculation accuracy of the video picture frame and improve the user's experience of watching the video.

[0005] In one aspect, the embodiments of the present application provide a video processing method, which comprises:

[0006] obtaining a video image sequence of a target video, and obtaining a first image pair sequence and a horizontal mirror image pair sequence according to a first image pair composed of every two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair;

[0007] determining a target difference value image sequence according to the first image pair sequence and the horizontal mirror image pair sequence, wherein each target difference value image in the target difference value image sequence is determined based on a first difference value image of each first image pair and a second difference value image of the corresponding horizontal mirror image pair;

[0008] performing straight line detection on each target difference value image in the target difference value image sequence to obtain a first straight line corresponding to each target difference value image in the target difference value image sequence respectively;

[0009] determining an image effective area coordinate of the target video according to a cosine distance between each first straight line and a horizontal straight line;

[0010] perform face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence;

[0011] determine video valid picture coordinates of the target video according to the face detection result corresponding to each video image in the video image sequence and the video valid region coordinates, and determine a video picture frame based on the video valid picture coordinates.

[0012] In another aspect, an embodiment of the present application provides a video processing device, and the device comprises:

[0013] an acquisition unit configured to acquire a video image sequence of a target video, and obtain a first image pair sequence and a horizontal mirror image pair sequence according to a first image pair formed by every two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair;

[0014] a difference calculation unit configured to determine a target difference image sequence according to difference calculation of the first image pair sequence and the horizontal mirror image pair sequence, wherein each target difference image in the target difference image sequence is determined based on a first difference image of each first image pair and a second difference image of the corresponding horizontal mirror image pair;

[0015] a straight line detection unit configured to perform straight line detection on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence;

[0016] a first determination unit configured to determine image valid region coordinates of the target video according to a cosine distance between each first straight line and a horizontal straight line;

[0017] a face detection unit configured to perform face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence;

[0018] a second determination unit configured to determine video valid picture coordinates of the target video according to the face detection result corresponding to each video image in the video image sequence and the video valid region coordinates, and determine a video picture frame based on the video valid picture coordinates.

[0019] In another aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program, and the computer program is adapted to be loaded by a processor to execute the video processing method according to any one of the above embodiments.

[0020] In another aspect, an embodiment of the present application provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program stored in the memory to perform the video processing method according to any one of the preceding embodiments.

[0021] In another aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the video processing method according to any one of the preceding embodiments.

[0022] In an embodiment of the present application, the video processing method comprises the following steps: obtaining a video image sequence of a target video, and obtaining a first image pair sequence and a horizontal mirror image pair sequence according to each first image pair consisting of two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair; determining a target difference image sequence according to the first image pair sequence and the horizontal mirror image pair sequence, wherein each target difference image in the target difference image sequence is determined based on a first difference image of each first image pair and a second difference image of the corresponding horizontal mirror image pair; performing straight line detection on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence; determining an image effective region coordinate of the target video according to a cosine distance between each first straight line and a horizontal straight line; performing face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence; determining a video effective picture coordinate of the target video according to the face detection result corresponding to each video image in the video image sequence and the image effective region coordinate, and determining a video picture frame based on the video effective picture coordinate. In an embodiment of the present application, the video processing method combines a straight line detection branch and a face detection branch to calculate the coordinate position of the video effective picture, so as to determine the video picture frame, wherein the straight line detection branch randomly combines the video image sequence of the target video into two first image pairs and the corresponding horizontal mirror image pairs, then calculates the target difference image, and then obtains all the first straight lines of the video image sequence through straight line detection, and determines the image effective region coordinate based on the cosine distance between all the first straight lines and the horizontal straight line; the face detection branch determines the video effective picture coordinate, i.e., the video picture frame, based on the face detection result of the face detection and the image effective region coordinate, thereby improving the calculation accuracy of the video picture frame and improving the experience of the user watching the video. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of the video processing method provided by an embodiment of the present application is shown.

[0024] Figure 2 An application scenario diagram of the video processing method provided by an embodiment of the present application is shown.

[0025] Figure 3 A structural schematic diagram of a video processing apparatus provided by an embodiment of the present application is shown.

[0026] Figure 4 A structural schematic diagram of a computer device provided by an embodiment of the present application is shown.

[0027] Figure 5 A structural schematic diagram of a storage medium provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0029] The embodiments of the present application provide a video processing method, apparatus, storage medium, device and program product. Specifically, the video processing method of the embodiments of the present application can be executed by a computer device, where the computer device can be a terminal or a server, etc. The terminal can be a smart phone, a tablet computer, a notebook computer, a smart television, a smart speaker, a wearable smart device, a smart vehicle terminal, etc. The terminal can also include a client, which can be a video client, a browser client, an instant messaging client or an applet, etc. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, etc. The embodiments of the present application can be applied to various scenarios such as video processing, image processing, artificial intelligence and intelligent transportation, etc.

[0030] First, some of the nouns or terms appearing in the description of the embodiments of the present application are explained as follows:

[0031] Artificial Intelligence (AI): is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0032] Intelligent transportation is to fully utilize the Internet of Things, spatial perception, cloud computing, mobile Internet and other new generation information technologies in the entire transportation field, and comprehensively use traffic science, system method, artificial intelligence, knowledge mining and other theories and tools to achieve the goal of comprehensive perception, deep integration, active service and scientific decision-making. Through the construction of a real-time dynamic information service system, deep mining of transportation-related data, formation of problem analysis models, and realization of the optimization of industry resource allocation, public decision-making, industry management, and public service capabilities, the operation and development of transportation are promoted to be safer, more efficient, more convenient, more economical, more environmentally friendly, and more comfortable. It also drives the transformation and upgrading of transportation-related industries.

[0033] Picture frame: there is a certain distance between the picture content of some movies, TV series and other video images and the real picture boundary. There is a region (such as the black edge of the movie, the colored edge in the small video, etc.) unrelated to the video image picture in this distance space; the picture frame is the horizontal line that divides the video content picture and this region, and the effective picture is all the regions of the video content picture.

[0034] With the rise of small videos, and the public spending more and more time watching videos on mobile devices such as mobile phones, and the habit of editing, adding pure color or color-changing pictures to the upper and lower boundaries of the original video picture when publishing videos.

[0035] Due to the limited screen size of mobile devices such as mobile phones, the additional picture edited by the user will have a "shrinking" effect on the presentation of the existing picture, which will affect the user's experience of watching the video.

[0036] In the related art, the calculation of the picture frame is mainly to calculate the horizontal line detection result through the possible boundaries in the video image, and then to distinguish the image picture; or to distinguish the picture by using face and body detection technology. The calculation method for the boundary in the related art does not consider the editing of the upper and lower black edges, colored edges, and text of the video; and the face and body detection calculation does not consider the solution to the case where no face appears.

[0037] In view of the problems in the above related art, the embodiments of the present application provide a video processing method, device, storage medium, equipment and program product, the embodiments of the present application can not only solve the division of each picture area in the video concerned in the related art, but also can accurately calculate the video picture frame for the edited video with black edges, colored edges, text and the like on the upper and lower boundaries of the video. The embodiments of the present application calculate the coordinate position of the video effective picture by combining the straight line detection branch and the face detection branch, so as to determine the video picture frame, wherein the straight line detection branch is to randomly combine the video image sequence of the target video into a first image pair and a corresponding horizontal mirror image pair in each group, then calculate a target difference image, and then obtain all first straight lines of the entire video image sequence through straight line detection, and determine the image effective area coordinate based on the cosine distance of all first straight lines and the horizontal straight line; the face detection branch is to determine the video effective picture coordinate, i.e. the video picture frame, based on the face detection result of face detection and the image effective area coordinate, thereby improving the calculation precision of the video picture frame and improving the experience of users watching the video.

[0038] The following will be described in detail. It should be noted that the description order of the following embodiments is not limited as the priority order of the embodiments.

[0039] The embodiments of the present application provide a video processing method, which can be executed by a terminal or a server, or jointly executed by a terminal and a server; the embodiments of the present application take the execution of the video processing method by the server as an example for description.

[0040] Please refer to Figures 1 to 2 , Figure 1 The flowchart of the video processing method provided by the embodiments of the present application is shown in Figure 2 The application scenario diagram of the video processing method provided by the embodiments of the present application is shown in. The method can include the following steps 110 to 160.

[0041] Step 110, obtain the video image sequence of a target video, and obtain the first image pair sequence and the horizontal mirror image pair sequence according to the first image pair composed of every two video images in the video image sequence and the horizontal mirror image pair corresponding to each first image pair.

[0042] For example, first, the input target video is randomly sampled to form a video image sequence, and the sampling ensures that the content of the beginning, middle and end of the entire video content is sampled. Then, the first image pair of two video images is randomly formed from the sampled video image sequence. Meanwhile, in order to fully utilize the significant color difference between the effective picture and the non-effective picture (such as the black border) of the video image and the incomplete information of the picture at a certain time point in the target video, the horizontal mirror image pair corresponding to each first image pair is calculated.

[0043] In some embodiments, the first image pair sequence and the horizontal mirror image pair sequence are obtained from the first image pair formed by every two video images in the video image sequence and the horizontal mirror image pair corresponding to each first image pair, including:

[0044] The video images in the video image sequence are grouped into a first image pair of two video images to obtain a first image pair sequence;

[0045] Each first image pair in the first image pair sequence is processed by horizontal mirroring to obtain a horizontal mirror image pair sequence, wherein each horizontal mirror image pair in the horizontal mirror image pair sequence has a horizontal mirror relationship with the corresponding first image pair in the first image pair sequence.

[0046] For example, m frames of video images randomly sampled from the input target video form a video image sequence frames, such as frames=[f n1 ,f n2 ,…,f nm ].

[0047] Then, the first image pair of two video images is randomly formed from the video image sequence frames, such as the first image pair f piar is formed by the front and rear video images in the video image sequence frames, such as f piar =(f n1 ,f nm ), as shown in the schematic diagram (a) and the schematic diagram (b) in Figure 2 . Subsequently, the video images in the video image sequence are grouped into a first image pair of two video images to obtain a first image pair sequence, such as the first image pair sequence including a plurality of first image pairs, for example, the first image pair sequence is [(f n1 ,f nm ),(f n2 ,f nm-1 ),(f n3 ,f nm-2 ),…,(f ni+1 ,fnm-i )].

[0048] For example, by performing horizontal mirror processing on the first image pair, a horizontal mirror image pair fh corresponding to the first image pair is obtained piar , such as fh piar = (fh n1 , fh nm ). For example, by performing horizontal mirror processing on each first image pair in the first image pair sequence, a horizontal mirror image pair sequence is obtained, each horizontal mirror image pair in the horizontal mirror image pair sequence is in a horizontal mirror relationship with the corresponding first image pair in the first image pair sequence, such as the horizontal mirror image pair sequence [(fh n1 , fh nm ), (fh n2 , fh nm-1 ), (fh n3 , fh nm-2 ), …, (fh ni+1 , fh nm-i )].

[0049] Step 120, determining a target difference image sequence according to the first image pair sequence and the horizontal mirror image pair sequence, each target difference image in the target difference image sequence is determined based on a first difference image of each first image pair and a second difference image of the corresponding horizontal mirror image pair.

[0050] In some embodiments, the determining a target difference image sequence according to the first image pair sequence and the horizontal mirror image pair sequence comprises:

[0051] performing difference calculation on each first image pair in the first image pair sequence to obtain a first difference image of each first image pair;

[0052] performing difference calculation on each horizontal mirror image pair in the horizontal mirror image pair sequence to obtain a second difference image of each horizontal mirror image pair;

[0053] summing the first difference image of each first image pair and the second difference image of the corresponding horizontal mirror image pair to obtain a target difference image corresponding to each first image pair, and determining a target difference image sequence according to the target difference image corresponding to each first image pair.

[0054] For example, performing difference calculation on each first image pair in the first image pair sequence to obtain a first difference image of each first image pair. For example, taking the first image pair f piar = (f n1 , f nm) for example, the first difference image f diff As shown in the schematic diagram (c) in FIG. 1 Figure 2 The first difference image f diff may be represented as:

[0055] f diff = f n1 - f nm , 0 ≤ f diff (x, y) ≤ 255;

[0056] wherein f diff (x, y) represents the value of any spatial coordinate (x, y) in the first difference image f diff Since the difference calculation of the first image pair is performed without distinction, the value of any spatial coordinate (x, y) in the first difference image f diff may be set in [0, 255].

[0057] For example, the difference calculation is performed on each horizontal mirror image pair in the horizontal mirror image pair sequence to obtain a second difference image of each horizontal mirror image pair. For example, the difference calculation is performed on the horizontal mirror image pair fh piar = (fh n1 , fh nm ) to obtain a second difference image fh diff As shown in the schematic diagram (d) in FIG. 1 Figure 2 The second difference image fh diff may be represented as:

[0058] fh diff = fh n1 - fh nm , 0 ≤ fh diff (x, y) ≤ 255;

[0059] wherein fh diff (x, y) represents the value of any spatial coordinate (x, y) in the second difference image fh diff Since the difference calculation of the horizontal mirror image pair is performed without distinction, the value of any spatial coordinate (x, y) in the second difference image fh diff may be set in [0, 255].

[0060] For example, the first difference image of each first image pair is summed with the second difference image of the corresponding horizontal mirror image pair to obtain a target difference image corresponding to each first image pair, and a target difference image sequence is determined according to the target difference image corresponding to each first image pair. For example, the difference calculation is performed on one of the target difference images fh sf to obtain a target difference image fh sflike Figure 2 As shown in the schematic diagram (e), the target difference image fh sf It can be represented as:

[0061] fh sf =f diff +fh diff .

[0062] In order to make the edges, contours, and details of the target difference image clearer, the target difference image can be sharpened. For example, for the target difference image fh sf Each pixel value fh sf (x, y), if its value is greater than the second threshold (e.g., 240), its value is set to 0; if its value is less than the first threshold (e.g., 20), its value is set to 0; if its value 20 ≤ fh sf If (x,y)≤240, then no operation is performed.

[0063] Step 130: Perform line detection on each target difference image in the target difference image sequence to obtain the first straight line corresponding to each target difference image in the target difference image sequence.

[0064] In some embodiments, the step of performing line detection on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence includes:

[0065] Based on the Hough transform algorithm, straight line detection is performed on each target difference image in the target difference image sequence to obtain the first straight line corresponding to each target difference image in the target difference image sequence.

[0066] For example, for each target difference image fh sf Perform line detection to detect the difference image fh for each target. sf The first straight line L in the image is used to obtain the difference image fh for each target. sf The first baseline set hL consists of all the first straight lines. sf For example, the first set of straight lines hL sf It can be represented as:

[0067] hL sf =hough(fh) sf );

[0068] Here, hough() represents the Hough operator of the Hough transform algorithm. The first set of lines hL sf Given a set of first straight lines [L1, L2, ..., L n Composed of, i.e., the first set of straight lines hLsf =[L1,L2,…,L n ],like Figure 2 The first straight line shown in the schematic diagram (f) is n, which represents the number of first straight lines in each target difference image, and n is a natural number.

[0069] Step 140: Determine the effective region coordinates of the target video image based on the cosine distance between each of the first straight lines and the horizontal straight line.

[0070] In some embodiments, step 140 can be implemented by the following steps 141 to 143 (not shown in the figure):

[0071] Step 141: Calculate the cosine distance between each of the first straight lines and the horizontal line;

[0072] Step 142: Based on the cosine distance, determine the upper boundary line set and the lower boundary line set of the target video, wherein the upper boundary line set is used to represent the set of lines representing the upper boundary of the border of the target video frame that is initially judged as a valid video frame, and the lower boundary line set is used to represent the set of lines representing the lower boundary of the border of the target video frame that is initially judged as a valid video frame.

[0073] Step 143: Determine the effective region coordinates of the target video image based on the set of upper boundary lines and the set of lower boundary lines.

[0074] In some embodiments, step 142 can be implemented by the following steps 1421 to 1424 (not shown in the figure):

[0075] Step 1421: Obtain the width and height values ​​of the original video border in the target video;

[0076] Step 1422: Obtain the coordinate values ​​of each of the first straight lines, wherein the coordinate values ​​include the starting coordinates and the ending coordinates, the starting coordinates include the x-axis starting coordinate value and the y-axis starting coordinate value, and the ending coordinates include the x-axis ending coordinate value and the y-axis ending coordinate value;

[0077] Step 1423: The first target line in the first line corresponding to each target difference image is included in the upper boundary line set. The first target line is the first line in which half of the sum of the y-axis start coordinate value and the y-axis end coordinate value is less than or equal to half of the width value, and the corresponding cosine distance is greater than or equal to the horizontal line cosine distance threshold.

[0078] Step 1424, the second target straight line existing in the first straight line corresponding to each target difference image is classified into the lower boundary straight line set, wherein the second target straight line is the first straight line with the sum of the starting coordinate value of the y-axis and the ending coordinate value of the y-axis greater than half of the width value, and the corresponding cosine distance greater than or equal to the horizontal straight line cosine distance threshold.

[0079] In some embodiments, step 143 can be implemented by the following steps 1431 to 1433 (not shown in the figure), specifically:

[0080] Step 1431, determining the target edge frame upper boundary straight line according to the coordinate value of each first target straight line in the upper boundary straight line set;

[0081] Step 1432, determining the target edge frame lower boundary straight line according to the coordinate value of each second target straight line in the lower boundary straight line set;

[0082] Step 1433, determining the image effective area coordinate of the target video according to the target edge frame upper boundary straight line and the target edge frame lower boundary straight line.

[0083] For example, each first straight line L = [(x1, y1), (x2, y2)], (x1, y1) and (x2, y2) represent the starting coordinate and the ending coordinate of each first straight line respectively, the starting coordinate [(x1, y1)] includes the x-axis starting coordinate value x1 and the y-axis starting coordinate value y1, and the ending coordinate (x2, y2) includes the x-axis ending coordinate value x2 and the y-axis ending coordinate value y2. For each first straight line L, the relative position of the y direction in the image can be represented as (y1+y2) / 2; in the image coordinate space, a horizontal straight line can be represented as vector hori =[-0.5, 0.0, 0.5, 0.0], then the cosine distance cossim hori between the first straight line L and the horizontal straight line vector hori can be calculated by the following formula:

[0084] cossim hori =1-abs(cos(L,vector hori ));

[0085] Wherein, abs represents the absolute value calculation of a number, such as setting the horizontal straight line cosine distance threshold sim thre =0.98; cossim hori ≥sim thre The upper boundary straight line set and the lower boundary straight line set corresponding to the straight line obtained, such as the gray straight line shown in the schematic diagram (g) in Figure 2 .

[0086] For example, if the width and height of the original video border in the target video are represented as w and h respectively, and y0 = w / 2 (i.e., half the width of the original video border), then... Figure 2 The white circle shown in the schematic diagram (h) indicates that a first target line exists. This first target line is one where half of the sum of the y-axis start coordinate and the y-axis end coordinate is less than or equal to half of the width value, and the corresponding cosine distance is greater than or equal to the horizontal line cosine distance threshold. That is, there exists (y1+y2) / 2≤y0, and cossim hori ≥sim thre The first target line L up At that time, the first target line L up Included in the upper boundary line set hL sfup In the case of a second target line, where the sum of the initial and final y-axis coordinates is greater than half the width, and the corresponding cosine distance is greater than or equal to the horizontal line cosine distance threshold, then there exists a first line where (y1+y2) / 2>y0, and cossim hori ≥sim thre The second target line L down At that time, the second target line L down Included in the lower boundary line set hL sfdown In the middle. For example, for each target difference image fh sf The set of first lines hL, which consists of all the first lines in the first line. sf =[L1,L2,…,L n After the above calculations and judgments, the set of upper boundary lines hL is obtained for each first straight line in the [ ]. sfup (x1, y1, x2, y2) = [L up1 (x1,y1,x2,y2),L up2 (x1,y1,x2,y2),…,L upk [(x1, y1, x2, y2)], where k represents the set of upper boundary lines hL sfup The number of the first target lines, k is a natural number; the set of upper boundary lines hL sfup The set of lines representing the upper boundary of the border of the target video frame initially determined to be a valid video frame; and the set of lines representing the lower boundary hL. sfdown (x1, y1, x2, y2) = [L down1 (x1,y1,x2,y2),L down2 (x1,y1,x2,y2),…,L down k [x1, y1, x2, y2], where k represents the set of lower boundary lines hLsfdown wherein k is a natural number, the lower boundary line set hL sfdown wherein k is a natural number, the upper boundary line set hL sfup wherein k is a natural number, the upper boundary line set hL sfdown A lower boundary line set of a target video frame is used to represent the lower boundary of the bounding box of the video frame which is initially determined as a valid video frame.

[0087] For example, the upper boundary line of the bounding box of the valid video frame is defined as a target upper boundary line of the bounding box L up (x1,y1,x2,y2), the target upper boundary line of the bounding box L up (x1,y1,x2,y2) can be obtained from the upper boundary line set hL sfup Specifically, according to the coordinate values of each first target line in the upper boundary line set, the target upper boundary line of the bounding box is determined.

[0088] wherein the x-axis starting coordinate value x1 of the target upper boundary line of the bounding box L up (x1,y1,x2,y2) is represented as L up x1, and the calculation formula is as follows:

[0089] L up x1=min(L up1 x1,L up2 x1,…,L upk x1);

[0090] wherein L upk x1 is represented as L sfup x1 in the upper boundary line set hL upk (x1,y1,x2,y2); L up2 x1 is represented as L sfup x1 in the upper boundary line set hL up2 (x1,y1,x2,y2); L up1 x1 is represented as L sfup x1 in the upper boundary line set hL up1 (x1,y1,x2,y2).

[0091] wherein the x-axis ending coordinate value x2 of the target upper boundary line of the bounding box L up (x1,y1,x2,y2) is represented as L up x2, and the calculation formula is as follows:

[0092] L up x2=max(L up1 x2,L up2 x2,…,L upk x2);

[0093] wherein L upk x2 represents L sfup in the upper boundary line set hL upk x2 in (x1, y1, x2, y2); L up2 x2 represents L sfup in the upper boundary line set hL up2 x2 in (x1, y1, x2, y2); L up1 x2 represents L sfup in the upper boundary line set hL up1 x2 in (x1, y1, x2, y2).

[0094] wherein the y-axis starting coordinate value y1 of the target bounding box upper boundary line L up is represented as L up y1, and the calculation formula is as follows:

[0095] L up y1 = min(L up1 y1, L up2 y1,..., L upk y1).

[0096] wherein L upk y1 represents L sfup y1 in the upper boundary line set hL upk y1 in (x1, y1, x2, y2); L up2 y1 represents L sfup y1 in the upper boundary line set hL up2 y1 in (x1, y1, x2, y2); L up1 y1 represents L sfup y1 in the upper boundary line set hL up1 y1 in (x1, y1, x2, y2).

[0097] wherein the y-axis ending coordinate value y2 of the target bounding box upper boundary line L up is represented as L up y2, and the calculation formula is as follows:

[0098] L up y2 = min(L up1 y2, L up2 y2,..., L upk y2).

[0099] wherein L upk y2 represents L sfup y2 in the upper boundary line set hL upk y2 in (x1, y1, x2, y2); Lup2 y2 represents the set of lines on the upper boundary, hL. sfup L in up2 y2 in (x1, y1, x2, y2); L up1 y1 represents the set of lines on the upper boundary hL sfup L in up1 y2 in (x1,y1,x2,y2).

[0100] Therefore, the target bounding line L up (x1, y1, x2, y2) = [L up x1,L up y1,L up x2,L up y2).

[0101] For example, the lower boundary line of the border representing the effective frame of the video is defined as the lower boundary line L of the target border. down (x1, y1, x2, y2), the lower boundary line L of the target bounding box down (x1, y1, x2, y2) can be passed through the lower boundary line set hL. sfdown Specifically, the lower boundary line of the target border is determined based on the coordinate values ​​of each second target line in the set of lower boundary lines.

[0102] Among them, the lower boundary line L of the target bounding box down The initial x-axis coordinate value x1 is represented as L down x1 is calculated using the following formula:

[0103] L down x1=min(L down1 x1,L down2 x1,…,L downk x1);

[0104] Among them, L downk x1 represents the set of lower boundary lines hL sfdown L in downk x1 in (x1, y1, x2, y2); L down2 x1 represents the set of lower boundary lines hL sfdown L in down2 x1 in (x1, y1, x2, y2); L down1 x1 represents the set of lower boundary lines hL sfdown L in down1 x1 in (x1,y1,x2,y2).

[0105] Among them, the lower boundary line L of the target bounding box down The x-axis termination coordinate value x2 is represented as Ldown x2, whose calculation formula is as follows:

[0106] L down x2 = max(L down1 x2, L down2 x2, …, L downk x2);

[0107] wherein, L downk x2 represents x2 in L sfdown (x1, y1, x2, y2) in the lower boundary line set hL downk (x1, y1, x2, y2) in the lower boundary line set hL down2 (x1, y1, x2, y2) in the lower boundary line set hL sfdown (x1, y1, x2, y2) in the lower boundary line set hL down2 (x1, y1, x2, y2) in the lower boundary line set hL down1 (x1, y1, x2, y2) in the lower boundary line set hL sfdown (x1, y1, x2, y2) in the lower boundary line set hL down1 (x1, y1, x2, y2) in the lower boundary line set hL

[0108] wherein, y-axis starting coordinate value y1 of the target edge frame lower boundary line L down is represented as L down y1, whose calculation formula is as follows:

[0109] L down y1 = max(L down1 y1, L down2 y1, …, L downk y1);

[0110] wherein, L downk y1 represents y1 in L sfdown (x1, y1, x2, y2) in the lower boundary line set hL downk (x1, y1, x2, y2) in the lower boundary line set hL down2 (x1, y1, x2, y2) in the lower boundary line set hL sfdown (x1, y1, x2, y2) in the lower boundary line set hL down2 (x1, y1, x2, y2) in the lower boundary line set hL down1 (x1, y1, x2, y2) in the lower boundary line set hL sfdown (x1, y1, x2, y2) in the lower boundary line set hL down1 (x1, y1, x2, y2) in the lower boundary line set hL

[0111] wherein, y-axis ending coordinate value y2 of the target edge frame lower boundary line L down is represented as L down y2, whose calculation formula is as follows:

[0112] L down y2 = max(L down1 y2, L down2y2,..., L downk y2);

[0113] wherein, L downk y2 represents L sfdown in the lower boundary line set hL downk y2 in (x1, y1, x2, y2); L down2 y2 represents L sfdown in the lower boundary line set hL down2 y2 in (x1, y1, x2, y2); L down1 y2 represents L sfdown in the lower boundary line set hL down1 y2 in (x1, y1, x2, y2).

[0114] Therefore, the target frame lower boundary line L down (x1, y1, x2, y2) = [L down x1, L down y1, L down x2, L down y2].

[0115] For example, according to the target frame upper boundary line L up (x1, y1, x2, y2) and the target frame lower boundary line L down (x1, y1, x2, y2), if the image effective area coordinates of the target video are between the target frame upper boundary line L up (x1, y1, x2, y2) and the target frame lower boundary line L down (x1, y1, x2, y2), then the image effective area coordinates image hrect of the target video can be represented as:

[0116] image hrect = [image hx1 , image hy1 , image hx2 , image hy2 ];

[0117] wherein, image hx1 represents the x-axis upper left coordinate value of the video effective area coordinates, image hy1 represents the y-axis upper left coordinate value of the video effective area coordinates, image hx2 represents the x-axis right lower coordinate value of the video effective area coordinates, and image hy2 represents the y-axis right lower coordinate value of the video effective area coordinates. For example, the calculation formula of each coordinate value is as follows:

[0118] image hx1 = min(L up x1, L down x1) ;

[0119] image hy1 = min(L up y1, L up y2) ;

[0120] image hx2 = max(L up x2, L down x2) ;

[0121] image hy2 = max(L down y1, L down y2).

[0122] Step 150, performing face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence.

[0123] For example, by calculating whether there is a face in the video image sequence randomly sampled in the target video through the face detection branch, and the position of the face, the spatial position of the face coordinates is calculated to obtain a face detection result corresponding to each video image in the video image sequence, and the detected face coordinate information is used as a basis for determining the minimum picture area, such as the picture area between the gray straight lines above and below the picture in the schematic diagram (i) in the above. Figure 2

[0124] For example, when performing face detection, a CNN face detection model can be used for face detection. For example, other face detection models consistent with the output data format of the CNN face detection model can also be used.

[0125] Step 160, determining the video effective picture coordinates of the target video according to the face detection result corresponding to each video image in the video image sequence and the video effective area coordinates, and determining the video picture frame based on the video effective picture coordinates.

[0126] In some embodiments, the determination of the video effective picture coordinates of the target video according to the face detection result corresponding to each video image in the video image sequence and the video effective area coordinates comprises:

[0127] ​When multiple video images in the video image sequence have a detected face, the effective video frame coordinates of the target video are determined based on the face coordinate information in the multiple video images with detected faces and the effective video area coordinates.

[0128] In some embodiments, determining the effective video frame coordinates of the target video based on the face coordinate information in the multiple video images of the detected face includes:

[0129] Based on the face coordinate information in the multiple video images of the detected face, the location of the face region in the target video is determined;

[0130] The effective frame coordinates of the target video are determined based on the coordinates of the face region and the effective video region.

[0131] In some embodiments, determining the location of the face region in the target video based on the face coordinate information in the multiple video images of the detected face includes:

[0132] Obtain face coordinate information from multiple video images of the detected face, wherein the face coordinate information includes upper left coordinates and lower right coordinates, the upper left coordinates include upper left x-axis coordinates and upper left y-axis coordinates, and the lower right coordinates include lower right x-axis coordinates and lower right y-axis coordinates;

[0133] The x-axis upper left coordinate value of the face region in the target video is determined based on the minimum value among the upper left x-axis coordinate values ​​of the face coordinate information corresponding to the multiple video images.

[0134] The y-axis upper-left coordinate value of the face region in the target video is determined based on the minimum value among the upper-left y-axis coordinate values ​​of the face coordinate information corresponding to the multiple video images.

[0135] The x-axis lower right coordinate value of the face region in the target video is determined based on the maximum value among the x-axis lower right coordinate values ​​of the face coordinate information corresponding to the multiple video images.

[0136] The lower right y-axis coordinate of the face region in the target video is determined based on the maximum value among the lower right y-axis coordinates of the face coordinate information corresponding to the multiple video images.

[0137] When a face is detected in p video images in a video image sequence, face rect =[face1,face2,face3,…,face p ] represents the set of all face coordinate information, face p =[facep x1,face p y1,face p x2,face p y2] represents the upper left coordinates (x1, y1) and the lower right coordinates (x2, y2) of the pth face, wherein the upper left coordinates (x1, y1) include the x-axis upper left coordinate value x1 and the y-axis upper left coordinate value y1, and the lower right coordinates (x2, y2) include the x-axis lower right coordinate value x2 and the y-axis lower right coordinate value y2. The face region position image frect of the target video can be represented as:

[0138] image frect = [image fx1 , image fy1 , image fx2 , image fy2 ];

[0139] wherein image fx1 represents the x-axis upper left coordinate value of the face region position of the target video, image fy1 represents the y-axis upper left coordinate value of the face region position of the target video, image fx2 represents the x-axis lower right coordinate value of the face region position of the target video, and image fy2 represents the y-axis lower right coordinate value of the face region position of the target video. For example, the calculation formula of each coordinate value is as follows:

[0140] image fx1 = min ([face1 x1, face2 x1, …, face p x1];

[0141] image fy1 = min ([face1 y1, face2 y1, …, face p y1];

[0142] image fx2 = max ([face1 x2, face2 x2, …, face p x2];

[0143] image fy2 = max ([face1 y2, face2 y2, …, face p y2].

[0144] In some embodiments, the determining the video valid picture coordinates of the target video according to the coordinate values of the face region position and the coordinate values of the video valid region coordinates comprises:

[0145] obtaining coordinate values of the video active area coordinates, the coordinate values of the video active area coordinates including an x-axis upper-left coordinate value, a y-axis upper-left coordinate value, an x-axis lower-right coordinate value, and a y-axis lower-right coordinate value of the video active area coordinates;

[0146] an x-axis upper-left coordinate value of the video active picture coordinates of the target video according to a minimum value between the x-axis upper-left coordinate value of the face region position of the target video and the x-axis upper-left coordinate value of the video active area coordinates;

[0147] a y-axis upper-left coordinate value of the video active picture coordinates of the target video according to a minimum value between the y-axis upper-left coordinate value of the face region position of the target video and the y-axis upper-left coordinate value of the video active area coordinates;

[0148] an x-axis lower-right coordinate value of the video active picture coordinates of the target video according to a maximum value between the x-axis lower-right coordinate value of the face region position of the target video and the x-axis lower-right coordinate value of the video active area coordinates;

[0149] a y-axis lower-right coordinate value of the video active picture coordinates of the target video according to a minimum value between the y-axis lower-right coordinate value of the face region position of the target video and the y-axis lower-right coordinate value of the video active area coordinates.

[0150] For example, according to the coordinate values [image frect ,image fx1 ,image fy1 ,image fx2 ,image fy2 ] of the face region position image hrect and the coordinate values [image hx1 ,image hy1 ,image hx2 ,image hy2 ] of the video active area coordinates image fhrect , the video active picture coordinates image fhrect of the target video can be determined, and the video active picture coordinates image fhrect may be expressed as:

[0151] image fh x1,image fh y1,image fh x2,image fh y2];

[0152] wherein image fhx1 represents the left-up coordinate value of the x-axis of the video effective picture coordinate of the target video, image fh y1 represents the left-up coordinate value of the y-axis of the video effective picture coordinate of the target video, image fh x2 represents the right-down coordinate value of the x-axis of the video effective picture coordinate of the target video, image fh y2 represents the right-down coordinate value of the y-axis of the video effective picture coordinate of the target video. For example, the calculation formula of each coordinate value is as follows:

[0153] image fh x1 = min(image fx1 ,image hx1 );

[0154] image fh y1 = min(image fy1 ,image hy1 );

[0155] image fh x2 = max(image fx2 ,image hx2 );

[0156] image fh y2 = max(image fy2 ,image hy2 )。

[0157] Then, the video picture frame is determined based on the video effective picture coordinate, as shown in the box in the schematic diagram (j) in Figure 2

[0158] In some embodiments, the video effective picture coordinate of the target video is determined based on the face detection result corresponding to each video image in the video image sequence and the video effective area coordinate, and the method further comprises:

[0159] When the face detection result corresponding to each video image in the video image sequence is that no face is detected, the video effective area coordinate is determined as the video effective picture coordinate of the target video.

[0160] For example, when the face detection result corresponding to each video image in the video image sequence is that no face is detected, the video effective area coordinate image hrect is determined as the video effective picture coordinate image fhrect of the target video, i.e., image fhrect = image hrect .​

[0161] In some embodiments, the method further comprises:

[0162] playing the video frame of the target video based on the video frame border.

[0163] For example, after determining the video frame border, the video frame can be presented based on the video frame border.

[0164] The present application provides a simple and effective method for calculating and determining the video effective frame and the video frame border. The method does not require additional data for model training, nor does it require additional manual labor for time-consuming and laborious manual data labeling. That is, the model does not need to be trained, and the CNN detection model for face detection can be "plug and play" mode, easy to expand.

[0165] The present application can also not include a face detection calculation branch. When the image effective area coordinates of the target video are calculated based on the straight line detection branch, a good video effective frame border calculation result can be obtained.

[0166] The calculation method of the difference image sequence in the present application also provides a quick method for roughly determining the frame range in the video. For use scenarios with low requirements for effective frames and precision, the first straight line corresponding to each target difference image in the target difference image sequence can be used to preliminarily determine the video effective frame.

[0167] The present application provides a calculation method of a difference image of a video image sequence. The method comprises the following steps: randomly sampling video images to form a video image sequence and a first image pair, calculating a horizontal mirror image of the first image pair, and then calculating a target difference image based on the first image pair and the corresponding horizontal mirror image pair.

[0168] The present application provides a method for determining a horizontal straight line in an image frame. The method comprises the following steps: calculating the cosine distance between the first straight line obtained in the image frame and the preset horizontal straight line, and then representing the probability that the first straight line is a horizontal straight line by the cosine distance.

[0169] The present application can simultaneously utilize whether there is a face in the video image and fuse the face coordinate information to optimize the area or border of the video effective frame, finally determine the video effective frame coordinates of the target video, and determine the video frame border based on the video effective frame coordinates.

[0170] All the above technical solutions can be combined to form optional embodiments of the present application, which will not be described one by one here.

[0171] The embodiment of the present application obtains a video image sequence of a target video, and obtains a first image pair sequence and a horizontal mirror image pair sequence according to a first image pair composed of every two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair; determines a target difference image sequence according to the first image pair sequence and the horizontal mirror image pair sequence, each target difference image in the target difference image sequence being determined based on a first difference image of each first image pair and a second difference image of the corresponding horizontal mirror image pair; performs straight line detection on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence; determines an image effective area coordinate of the target video according to a cosine distance between each first straight line and a horizontal straight line; performs face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence; determines a video effective picture coordinate of the target video according to the face detection result corresponding to each video image in the video image sequence and the video effective area coordinate, and determines a video picture frame based on the video effective picture coordinate. The embodiment of the present application determines the coordinate position of the video effective picture by combining the straight line detection branch and the face detection branch, so as to determine the video picture frame, wherein the straight line detection branch is to combine the video image sequence of the target video into the first image pair and the corresponding horizontal mirror image pair in twos, then calculate the target difference image, and then obtain all the first straight lines of the entire video image sequence through the straight line detection, and determine the image effective area coordinate based on the cosine distance between all the first straight lines and the horizontal straight line; the face detection branch is to determine the video effective picture coordinate, that is, the video picture frame, based on the face detection result of the face detection and the image effective area coordinate, thereby improving the calculation precision of the video picture frame and improving the experience of the user watching the video.

[0172] To better implement the video processing method of the embodiment of the present application, the embodiment of the present application further provides a video processing device. Please refer to Figure 3 , Figure 3 The structure schematic diagram of the video processing device provided by the embodiment of the present application. The video processing device 200 can include:

[0173] The acquisition unit 210 is configured to acquire a video image sequence of a target video, and obtain a first image pair sequence and a horizontal mirror image pair sequence according to a first image pair composed of every two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair;

[0174] The difference calculation unit 220 is configured to calculate a target difference image sequence according to a difference between the first image pair sequence and the horizontal mirror image pair sequence, wherein each target difference image in the target difference image sequence is determined based on a first difference image of each first image pair and a second difference image of a corresponding horizontal mirror image pair;

[0175] The straight line detection unit 230 is configured to perform straight line detection on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence;

[0176] The first determination unit 240 is configured to determine an image valid region coordinate of the target video according to a cosine distance between each first straight line and a horizontal straight line;

[0177] The face detection unit 250 is configured to perform face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence;

[0178] The second determination unit 260 is configured to determine a video valid picture coordinate of the target video according to the face detection result corresponding to each video image in the video image sequence and the video valid region coordinate, and determine a video picture frame based on the video valid picture coordinate.

[0179] In some embodiments, the acquisition unit 210 is configured to:

[0180] The video images in the video image sequence are grouped into first image pairs each of which comprises two video images to obtain a first image pair sequence;

[0181] Each first image pair in the first image pair sequence is subjected to horizontal mirror image processing to obtain a horizontal mirror image pair sequence, wherein each horizontal mirror image pair in the horizontal mirror image pair sequence is in a horizontal mirror image relationship with a corresponding first image pair in the first image pair sequence.

[0182] In some embodiments, the difference calculation unit 220 is configured to:

[0183] Each first image pair in the first image pair sequence is subjected to difference calculation to obtain a first difference image of each first image pair;

[0184] Each horizontal mirror image pair in the horizontal mirror image pair sequence is subjected to difference calculation to obtain a second difference image of each horizontal mirror image pair;

[0185] summing the first difference image of each of the first image pairs with the second difference image of the corresponding horizontal mirror image pair to obtain a target difference image corresponding to each of the first image pairs, and determining a target difference image sequence according to the target difference image corresponding to each of the first image pairs.

[0186] In some embodiments, the straight line detection unit 230 is configured to:

[0187] performing straight line detection on each target difference image in the target difference image sequence based on a Hough transform algorithm to obtain a first straight line corresponding to each target difference image in the target difference image sequence.

[0188] In some embodiments, the first determination unit 240 is configured to:

[0189] calculating a cosine distance between each first straight line and a horizontal straight line;

[0190] determining an upper boundary straight line set and a lower boundary straight line set of the target video according to the cosine distance, wherein the upper boundary straight line set is used to represent a straight line set of an upper boundary of a frame of a video picture of the target video that is initially judged as a video valid frame, and the lower boundary straight line set is used to represent a straight line set of a lower boundary of the frame of the video picture of the target video that is initially judged as the video valid frame.

[0191] determining image valid region coordinates of the target video according to the upper boundary straight line set and the lower boundary straight line set.

[0192] In some embodiments, when determining the upper boundary straight line set and the lower boundary straight line set of the target video according to the cosine distance, the first determination unit 240 is configured to:

[0193] obtaining a width value and a height value of an original video frame of the target video;

[0194] obtaining a coordinate value of each first straight line, wherein the coordinate value includes a start coordinate and an end coordinate, the start coordinate includes an x-axis start coordinate value and a y-axis start coordinate value, and the end coordinate includes an x-axis end coordinate value and a y-axis end coordinate value;

[0195] classifying a first target straight line existing in the first straight line corresponding to each target difference image into the upper boundary straight line set, wherein the first target straight line is a first straight line whose half of a sum of a y-axis start coordinate value and a y-axis end coordinate value is less than or equal to half of the width value and whose corresponding cosine distance is greater than or equal to a horizontal straight line cosine distance threshold value.

[0196] corresponding to each target difference image, into the lower boundary line set, wherein the second target line is a first line whose sum of a starting coordinate value of a y-axis and a terminal coordinate value of the y-axis is greater than half of the width value, and whose corresponding cosine distance is greater than or equal to the horizontal line cosine distance threshold.

[0197] In some embodiments, the first determining unit 240, when determining the image valid region coordinate of the target video according to the upper boundary line set and the lower boundary line set, is configured to:

[0198] determine a target frame upper boundary line according to the coordinate value of each first target line in the upper boundary line set;

[0199] determine a target frame lower boundary line according to the coordinate value of each second target line in the lower boundary line set;

[0200] determine the image valid region coordinate of the target video according to the target frame upper boundary line and the target frame lower boundary line.

[0201] In some embodiments, the second determining unit 260 is configured to:

[0202] when the face detection result of the plurality of video images in the video image sequence is that a face is detected, determine the video valid picture coordinate of the target video according to the face coordinate information of the plurality of video images in which a face is detected and the video valid region coordinate.

[0203] In some embodiments, the second determining unit 260, when determining the video valid picture coordinate of the target video according to the face coordinate information of the plurality of video images in which a face is detected, is configured to:

[0204] determine a face region position of the target video according to the face coordinate information of the plurality of video images in which a face is detected;

[0205] determine the video valid picture coordinate of the target video according to the coordinate value of the face region position and the coordinate value of the video valid region coordinate.

[0206] In some embodiments, the second determining unit 260, when determining the face region position of the target video according to the face coordinate information of the plurality of video images in which a face is detected, is configured to:

[0207] obtaining face coordinate information in the plurality of video images in which the face is detected, wherein the face coordinate information comprises upper left coordinates and lower right coordinates, the upper left coordinates comprise an x-axis upper left coordinate value and a y-axis upper left coordinate value, and the lower right coordinates comprise an x-axis lower right coordinate value and a y-axis lower right coordinate value;

[0208] determining an x-axis upper left coordinate value of the face region position of the target video according to a minimum value in the x-axis upper left coordinate values of the face coordinate information corresponding to the plurality of video images;

[0209] determining a y-axis upper left coordinate value of the face region position of the target video according to a minimum value in the y-axis upper left coordinate values of the face coordinate information corresponding to the plurality of video images;

[0210] determining an x-axis lower right coordinate value of the face region position of the target video according to a maximum value in the x-axis lower right coordinate values of the face coordinate information corresponding to the plurality of video images;

[0211] determining a y-axis lower right coordinate value of the face region position of the target video according to a maximum value in the y-axis lower right coordinate values of the face coordinate information corresponding to the plurality of video images.

[0212] In some embodiments, the second determination unit 260 is configured to:

[0213] obtaining coordinate values of the video effective region coordinates, wherein the coordinate values of the video effective region coordinates comprise an x-axis upper left coordinate value, a y-axis upper left coordinate value, an x-axis lower right coordinate value, and a y-axis lower right coordinate value of the video effective region coordinates;

[0214] determining an x-axis upper left coordinate value of the video effective picture coordinates of the target video according to a minimum value in the x-axis upper left coordinate values of the face region position of the target video and the x-axis upper left coordinate values of the video effective region coordinates;

[0215] determining a y-axis upper left coordinate value of the video effective picture coordinates of the target video according to a minimum value in the y-axis upper left coordinate values of the face region position of the target video and the y-axis upper left coordinate values of the video effective region coordinates;

[0216] determining an x-axis lower right coordinate value of the video effective picture coordinates of the target video according to a maximum value in the x-axis lower right coordinate values of the face region position of the target video and the x-axis lower right coordinate values of the video effective region coordinates;

[0217] The lower right y-axis coordinate of the effective video frame is determined by the minimum value between the lower right y-axis coordinate of the face region in the target video and the lower right y-axis coordinate of the effective video frame.

[0218] In some embodiments, the second determining unit 260 is further configured to:

[0219] When the face detection result for each video image in the video image sequence is "no face detected", the coordinates of the effective video region are determined as the effective video frame coordinates of the target video.

[0220] In some embodiments, the video processing apparatus 200 further includes:

[0221] The playback unit is used to play the video frame of the target video based on the video frame border.

[0222] Each unit in the aforementioned video processing device 200 can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0223] The video processing device 200 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the video processing device 200 can be the terminal or server.

[0224] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0225] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device can be a terminal, and the terminal can be a smart vehicle terminal. For example... Figure 4 As shown, the computer device 300 may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. The communication interface 301, memory 302, and processor 303 communicate with each other via the communication bus 304. The communication interface 301 is used for data communication between the computer device 300 and external devices. The memory 302 can be used to store software programs and modules, and the processor 303 runs the software programs and modules stored in the memory 302, such as the software programs for the corresponding operations in the foregoing method embodiments.

[0226] Optionally, the processor 303 can invoke software programs and modules stored in the memory 302 to perform the following operations:

[0227] The video image sequence of the target video is acquired, and a first image pair sequence and a horizontal mirror image pair sequence are obtained according to each first image pair composed of two video images in the video image sequence and a corresponding horizontal mirror image pair of each first image pair; a target difference image sequence is determined according to the first image pair sequence and the horizontal mirror image pair sequence, each target difference image in the target difference image sequence being determined based on a first difference image of each first image pair and a second difference image of the corresponding horizontal mirror image pair; a straight line detection is performed on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence; an image effective area coordinate of the target video is determined according to a cosine distance between each first straight line and a horizontal straight line; a face detection is performed on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence; a video effective picture coordinate of the target video is determined according to the face detection result corresponding to each video image in the video image sequence and the video effective area coordinate, and a video picture frame is determined based on the video effective picture coordinate.

[0228] The application further provides a computer readable storage medium for storing a computer program. The computer readable storage medium can be applied to a computer device, and the computer program causes the computer device to perform corresponding procedures in the video processing method in the embodiments of the application. For brevity, details are not repeated here.

[0229] A schematic diagram of the storage medium provided by the embodiments of the application is shown in FIG. 4, which describes a program product 400 for implementing the above method according to the example embodiments of the application, which can adopt a portable compact disc read-only memory (CDROM) and include program codes, and can run on a computer device, such as a mobile phone. However, the program product of the application is not limited to this, and in the application, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or apparatus. Figure 5

[0230] The application further provides a computer program product, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs corresponding procedures in the video processing method in the embodiments of the application. For brevity, details are not repeated here.

[0231] ​The application further provides a computer program including computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform corresponding procedures in the video processing method of the application. For brevity, details are not repeated herein.

[0232] It should be understood that the processor of the application can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the method embodiments described above can be completed by integrated logic circuits or instructions in the form of software in the processor. The processor described above can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0233] It can be appreciated that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0234] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0235] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the apparatus and computer device described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0236] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0237] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed to a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0238] In addition, each functional unit in the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0239] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and various program code storage media.

[0240] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of video processing, the method comprising: The method comprises: obtaining a video image sequence of a target video, and obtaining a first image pair sequence and a horizontal mirror image pair sequence according to each first image pair composed of two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair; determining a target difference value image sequence according to the first image pair sequence and the horizontal mirror image pair sequence, each target difference value image in the target difference value image sequence being determined based on a first difference value image of each first image pair and a second difference value image of the corresponding horizontal mirror image pair; performing straight line detection on each target difference value image in the target difference value image sequence to obtain a first straight line corresponding to each target difference value image in the target difference value image sequence respectively; determining an image effective region coordinate of the target video according to a cosine distance between each first straight line and a horizontal straight line; performing face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence; determining a video effective picture coordinate of the target video according to the face detection result corresponding to each video image in the video image sequence and the video effective region coordinate, and determining a video picture frame based on the video effective picture coordinate; the method comprises: calculating the cosine distance between each first straight line and a horizontal straight line; determining an upper boundary straight line set and a lower boundary straight line set of the target video according to the cosine distance, wherein the upper boundary straight line set is used to represent a straight line set of an upper boundary of a frame of a video effective picture in a video picture of the target video which is initially judged to be a video effective picture, and the lower boundary straight line set is used to represent a straight line set of a lower boundary of the frame of the video effective picture in the video picture of the target video which is initially judged to be the video effective picture; determining the image effective region coordinate of the target video according to the upper boundary straight line set and the lower boundary straight line set.

2. The video processing method of claim 1, wherein, the method comprises: grouping the video images in the video image sequence into first image pairs each composed of two video images to obtain a first image pair sequence; performing horizontal mirror image processing on each first image pair in the first image pair sequence to obtain a horizontal mirror image pair sequence, wherein each horizontal mirror image pair in the horizontal mirror image pair sequence has a horizontal mirror image relationship with the corresponding first image pair in the first image pair sequence.

3. The video processing method of claim 1, wherein, the method comprises: performing difference value calculation on each first image pair in the first image pair sequence to obtain a first difference value image of each first image pair; determining a second difference image of each of the horizontal mirror image pairs; summing the first difference image of each of the first image pairs and the second difference image of the corresponding horizontal mirror image pair to obtain a target difference image corresponding to each of the first image pairs, and determining a target difference image sequence according to the target difference image corresponding to each of the first image pairs.

4. The video processing method of any of claims 1-3, wherein, the line detection on each target difference image in the target difference image sequence based on a Hough transform algorithm to obtain a first straight line corresponding to each target difference image in the target difference image sequence. the determination of the upper boundary straight line set and the lower boundary straight line set of the target video according to the cosine distance includes:

5. The video processing method of claim 1, wherein, obtaining a width value and a height value of an original video frame of the target video; obtaining a coordinate value of each first straight line, wherein the coordinate value includes a starting coordinate and a terminal coordinate, the starting coordinate includes an x-axis starting coordinate value and a y-axis starting coordinate value, and the terminal coordinate includes an x-axis terminal coordinate value and a y-axis terminal coordinate value; the first target straight line in each first straight line corresponding to each target difference image is classified into the upper boundary straight line set, wherein the first target straight line is a first straight line whose half of the sum of the y-axis starting coordinate value and the y-axis terminal coordinate value is less than or equal to half of the width value and whose corresponding cosine distance is greater than or equal to a horizontal straight line cosine distance threshold value; the second target straight line in each first straight line corresponding to each target difference image is classified into the lower boundary straight line set, wherein the second target straight line is a first straight line whose half of the sum of the y-axis starting coordinate value and the y-axis terminal coordinate value is greater than half of the width value and whose corresponding cosine distance is greater than or equal to a horizontal straight line cosine distance threshold value. the determination of the image effective area coordinate of the target video according to the upper boundary straight line set and the lower boundary straight line set includes:

6. The video processing method of claim 5, wherein, determining a target frame upper boundary straight line according to the coordinate value of each first target straight line in the upper boundary straight line set; determining a target frame lower boundary straight line according to the coordinate value of each second target straight line in the lower boundary straight line set; determining the image effective area coordinate of the target video according to the target frame upper boundary straight line and the target frame lower boundary straight line. the determination of the video effective picture coordinate of the target video according to the face detection result corresponding to each video image in the video image sequence and the video effective area coordinate includes:

7. The video processing method of claim 1, wherein, when the face detection result corresponding to multiple video images in the video image sequence is detection of a face, determining the video effective picture coordinate of the target video according to the face coordinate information in the multiple video images with detection of a face and the video effective area coordinate. ​ 8. The video processing method of claim 7, wherein, The video effective picture coordinates of the target video are determined according to the face coordinate information in the multiple video images in which the face is detected, and the video effective picture coordinates of the target video are determined according to the face coordinate information in the multiple video images in which the face is detected. The face region position of the target video is determined according to the face coordinate information in the multiple video images in which the face is detected. The face region position of the target video is determined according to the face coordinate information in the multiple video images in which the face is detected.

9. The video processing method of claim 8, wherein, The face region position of the target video is determined according to the face coordinate information in the multiple video images in which the face is detected. The face coordinate information in the multiple video images in which the face is detected is obtained, wherein the face coordinate information includes upper left coordinates and lower right coordinates, the upper left coordinates include an x-axis upper left coordinate value and a y-axis upper left coordinate value, and the lower right coordinates include an x-axis lower right coordinate value and a y-axis lower right coordinate value. The x-axis upper left coordinate value of the face region position of the target video is determined according to the minimum value of the x-axis upper left coordinate values of the face coordinate information corresponding to the multiple video images. The y-axis upper left coordinate value of the face region position of the target video is determined according to the minimum value of the y-axis upper left coordinate values of the face coordinate information corresponding to the multiple video images. The x-axis lower right coordinate value of the face region position of the target video is determined according to the maximum value of the x-axis lower right coordinate values of the face coordinate information corresponding to the multiple video images. The y-axis lower right coordinate value of the face region position of the target video is determined according to the maximum value of the y-axis lower right coordinate values of the face coordinate information corresponding to the multiple video images.

10. The video processing method of claim 9, wherein, The face region position of the target video is determined according to the face coordinate information in the multiple video images in which the face is detected. The coordinate values of the video effective region coordinates are obtained, and the coordinate values of the video effective region coordinates include x-axis upper left coordinate values, y-axis upper left coordinate values, x-axis lower right coordinate values and y-axis lower right coordinate values of the video effective region coordinates. The x-axis upper left coordinate value of the video effective picture coordinates of the target video is determined according to the minimum value of the x-axis upper left coordinate values of the face region position of the target video and the x-axis upper left coordinate values of the video effective region coordinates. The y-axis upper left coordinate value of the video effective picture coordinates of the target video is determined according to the minimum value of the y-axis upper left coordinate values of the face region position of the target video and the y-axis upper left coordinate values of the video effective region coordinates. The x-axis lower right coordinate value of the video effective picture coordinates of the target video is determined according to the maximum value of the x-axis lower right coordinate values of the face region position of the target video and the x-axis lower right coordinate values of the video effective region coordinates. The y-axis lower right coordinate value of the video effective picture coordinates of the target video is determined according to the minimum value of the y-axis lower right coordinate values of the face region position of the target video and the y-axis lower right coordinate values of the video effective region coordinates.

11. The video processing method of claim 7, wherein, The face region position of the target video is determined according to the face coordinate information in the multiple video images in which the face is detected. When the face detection result corresponding to each video image in the video image sequence is no face detected, the video effective region coordinates are determined as the video effective picture coordinates of the target video.

12. The video processing method of claim 1, wherein, The method further comprises: playing a video picture of the target video based on the video picture frame.

13. A video processing apparatus, comprising: The device comprises: an acquisition unit configured to acquire a video image sequence of a target video, and obtain a first image pair sequence and a horizontal mirror image pair sequence based on each first image pair composed of every two video images in the video image sequence and a horizontal mirror image pair corresponding to each first image pair; a difference calculation unit configured to determine a target difference image sequence based on difference calculation of the first image pair sequence and the horizontal mirror image pair sequence, wherein each target difference image in the target difference image sequence is determined based on a first difference image of each first image pair and a second difference image of the corresponding horizontal mirror image pair; a straight line detection unit configured to perform straight line detection on each target difference image in the target difference image sequence to obtain a first straight line corresponding to each target difference image in the target difference image sequence; a first determination unit configured to determine image effective region coordinates of the target video based on a cosine distance between each first straight line and a horizontal straight line; a face detection unit configured to perform face detection on the video image sequence to obtain a face detection result corresponding to each video image in the video image sequence; a second determination unit configured to determine video effective picture coordinates of the target video based on the face detection result corresponding to each video image in the video image sequence and the video effective region coordinates, and determine a video picture frame based on the video effective picture coordinates; the first determination unit is configured to: calculate the cosine distance between each first straight line and the horizontal straight line; determine an upper boundary straight line set and a lower boundary straight line set of the target video based on the cosine distance, wherein the upper boundary straight line set is used to represent a straight line set of an upper boundary of the frame of the video picture of the target video that is initially judged as a video effective picture, and the lower boundary straight line set is used to represent a straight line set of a lower boundary of the frame of the video picture of the target video that is initially judged as a video effective picture; determine the image effective region coordinates of the target video based on the upper boundary straight line set and the lower boundary straight line set.

14. The video processing device of claim 13, wherein, the acquisition unit is configured to: compose a first image pair of every two video images in the video image sequence to obtain a first image pair sequence; perform horizontal mirror image processing on each first image pair in the first image pair sequence to obtain a horizontal mirror image pair sequence, wherein each horizontal mirror image pair in the horizontal mirror image pair sequence has a horizontal mirror image relationship with the corresponding first image pair in the first image pair sequence.

15. The video processing device of claim 13, wherein, the difference calculation unit is configured to: perform difference calculation on each first image pair in the first image pair sequence to obtain a first difference image of each first image pair; determining a second difference image of each of the horizontal mirror image pairs; summing the first difference image of each of the first image pairs and the second difference image of the corresponding horizontal mirror image pair to obtain a target difference image corresponding to each of the first image pairs, and determining a target difference image sequence according to the target difference image corresponding to each of the first image pairs.

16. The video processing apparatus of any of claims 13-15, wherein, The straight line detection unit is configured to: perform straight line detection on each target difference image in the target difference image sequence based on a Hough transform algorithm to obtain a first straight line corresponding to each target difference image in the target difference image sequence.

17. The video processing device of claim 13, wherein, The first determination unit is configured to: obtain a width value and a height value of an original video frame in the target video; obtain a coordinate value of each first straight line, wherein the coordinate value includes a start coordinate and an end coordinate, the start coordinate includes an x-axis start coordinate value and a y-axis start coordinate value, and the end coordinate includes an x-axis end coordinate value and a y-axis end coordinate value; classify a first target straight line existing in the first straight line corresponding to each target difference image into the upper boundary straight line set, wherein the first target straight line is a first straight line whose half of the sum of the y-axis start coordinate value and the y-axis end coordinate value is less than or equal to half of the width value and whose corresponding cosine distance is greater than or equal to a horizontal straight line cosine distance threshold value; classify a second target straight line existing in the first straight line corresponding to each target difference image into the lower boundary straight line set, wherein the second target straight line is a first straight line whose half of the sum of the y-axis start coordinate value and the y-axis end coordinate value is greater than half of the width value and whose corresponding cosine distance is greater than or equal to the horizontal straight line cosine distance threshold value.

18. The video processing device of claim 17, wherein, The first determination unit is configured to: determine a target frame upper boundary straight line according to the coordinate value of each first target straight line in the upper boundary straight line set; determine a target frame lower boundary straight line according to the coordinate value of each second target straight line in the lower boundary straight line set; determine the image effective area coordinates of the target video according to the target frame upper boundary straight line and the target frame lower boundary straight line.

19. The video processing device of claim 13, wherein, The second determination unit is configured to: when the face detection result of the plurality of video images in the video image sequence is that a face is detected, determine the video effective picture coordinates of the target video according to the face coordinate information in the plurality of video images in which a face is detected and the video effective area coordinates.

20. The video processing device of claim 19, wherein, The second determination unit is configured to: determine a face region position of the target video according to the face coordinate information in the plurality of video images in which a face is detected; and determine the video effective picture coordinates of the target video according to the face region position of the target video. According to the coordinate value of the face region position and the coordinate value of the video effective region coordinate, the video effective picture coordinate of the target video is determined.

21. The video processing device of claim 20, wherein, When the second determining unit determines the face region position of the target video according to the face coordinate information of the multiple video images in which the face is detected, the second determining unit is configured to: obtain the face coordinate information of the multiple video images in which the face is detected, wherein the face coordinate information comprises upper left coordinates and lower right coordinates, the upper left coordinates comprise an x-axis upper left coordinate value and a y-axis upper left coordinate value, and the lower right coordinates comprise an x-axis lower right coordinate value and a y-axis lower right coordinate value; determine the x-axis upper left coordinate value of the face region position of the target video according to the minimum value of the x-axis upper left coordinate values of the face coordinate information corresponding to the multiple video images; determine the y-axis upper left coordinate value of the face region position of the target video according to the minimum value of the y-axis upper left coordinate values of the face coordinate information corresponding to the multiple video images; determine the x-axis lower right coordinate value of the face region position of the target video according to the maximum value of the x-axis lower right coordinate values of the face coordinate information corresponding to the multiple video images; and determine the y-axis lower right coordinate value of the face region position of the target video according to the maximum value of the y-axis lower right coordinate values of the face coordinate information corresponding to the multiple video images.

22. The video processing device of claim 21, wherein, When the second determining unit determines the video effective picture coordinate of the target video according to the coordinate value of the face region position and the coordinate value of the video effective region coordinate, the second determining unit is configured to: obtain the coordinate value of the video effective region coordinate, wherein the coordinate value of the video effective region coordinate comprises an x-axis upper left coordinate value, a y-axis upper left coordinate value, an x-axis lower right coordinate value, and a y-axis lower right coordinate value of the video effective region coordinate; determine the x-axis upper left coordinate value of the video effective picture coordinate of the target video according to the minimum value of the x-axis upper left coordinate value of the face region position of the target video and the x-axis upper left coordinate value of the video effective region coordinate; determine the y-axis upper left coordinate value of the video effective picture coordinate of the target video according to the minimum value of the y-axis upper left coordinate value of the face region position of the target video and the y-axis upper left coordinate value of the video effective region coordinate; determine the x-axis lower right coordinate value of the video effective picture coordinate of the target video according to the maximum value of the x-axis lower right coordinate value of the face region position of the target video and the x-axis lower right coordinate value of the video effective region coordinate; and determine the y-axis lower right coordinate value of the video effective picture coordinate of the target video according to the minimum value of the y-axis lower right coordinate value of the face region position of the target video and the y-axis lower right coordinate value of the video effective region coordinate.

23. The video processing device of claim 19, wherein, The second determining unit is further configured to: when the face detection result corresponding to each video image in the sequence of video images is that no face is detected, determine the video effective region coordinate as the video effective picture coordinate of the target video.

24. The video processing device of claim 13, wherein, The apparatus further comprises: a playing unit configured to play the video picture of the target video based on the video picture frame.

25. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is loaded by a processor to execute the video processing method in any one of claims 1-12.

26. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores a computer program, and the processor invokes the computer program stored in the memory to execute the video processing method in any one of claims 1-12.

27. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the video processing method in any one of claims 1-12.

Images