Video summary processing method, device, electronic device and storage medium

Abstract

Embodiments of the present invention disclose a video summary processing method, apparatus, electronic device, and storage medium. The video summary processing method includes: determining a location to be retrieved in a video summary based on a received retrieval instruction; wherein the video summary is generated based on at least two frames of an original video; determining a distance from the location to be retrieved to candidate motion trajectories of each moving object in the video summary; and determining a target motion trajectory based on the distance, so as to locate a video sequence in the original video based on the target moving object associated with the target motion trajectory. The embodiments of the present invention retrieve data in the video summary using a trajectory positioning method, thereby improving the efficiency and accuracy of retrieving target moving objects in the video summary.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of video processing technology, and in particular to a video summary processing method, device, electronic device, and storage medium. Background Art

[0002] With the advent of high-definition video surveillance, viewing and retrieving massive amounts of video data has become a major concern for users. Video summarization is a video compression technology that processes video data, extracts content of interest, and then rearranges and presents it. This shortens the length of video data, allowing users to quickly find the content they want to watch.

[0003] In video summary retrieval, most existing methods rely on image-based search, establishing an association between a moving object and the target to be retrieved, then matching and searching. This method is inefficient and prone to retrieval errors when there are multiple interfering objects near the moving object. Summary of the Invention

[0004] Embodiments of the present invention provide a video summary processing method, device, electronic device, and storage medium to improve the efficiency and accuracy of video summary retrieval.

[0005] In a first aspect, an embodiment of the present invention provides a video summarization processing method, including:

[0006] Determining a position to be retrieved in the video summary according to the received retrieval instruction; wherein the video summary is generated based on at least two frames of images of the original video;

[0007] Determining the distance from the to-be-retrieved position to the candidate motion trajectory of each moving object in the video summary;

[0008] A target motion trajectory is determined according to the distance, so as to locate a video sequence in the original video according to the target moving object associated with the target motion trajectory.

[0009] In a second aspect, an embodiment of the present invention further provides a video summary processing device, including:

[0010] A module for determining a position to be retrieved, configured to determine a position to be retrieved in a video summary according to a received retrieval instruction; wherein the video summary is generated based on at least two frames of an original video;

[0011] a distance determination module, configured to determine the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary;

[0012] The motion trajectory determination module is configured to determine a target motion trajectory according to the distance, so as to locate a video sequence in the original video according to a target moving object associated with the target motion trajectory.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the video summarization processing method according to any embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video summarization processing method as described in any embodiment of the present invention.

[0018] This embodiment of the present invention determines a location to be searched in a video summary based on a received search instruction, determines a target motion trajectory from all motion trajectories based on the distance between the location to be searched and all motion trajectories in the video summary, and determines the video sequence containing the target motion object in the original video based on the target motion trajectory's association with the target motion object in the video summary. This embodiment of the present invention uses a trajectory positioning method to retrieve data from the video summary, thereby improving the efficiency and accuracy of searching for target motion objects in the video summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of a video summary processing method in embodiment 1 of the present invention;

[0020] Figure 2 It is a schematic diagram of cosine angle determination;

[0021] Figure 3 It is a flowchart of determining the distance from the position to be retrieved to the candidate motion trajectory;

[0022] Figure 4 is a flowchart of a video summary processing method in a second embodiment of the present invention;

[0023] Figure 5 It is a flowchart of video summary generation;

[0024] Figure 6 is a schematic structural diagram of a video summary processing device in a third embodiment of the present invention;

[0025] Figure 7 It is a structural diagram of an electronic device in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0027] Example 1

[0028] Figure 1 This is a flowchart of the video summary processing method in the first embodiment of the present invention. This embodiment is applicable to the case where the retrieved moving object is located in the original video according to the retrieved moving object determined by the user from the video summary. The method can be executed by a video summary processing device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a background server or other device with communication and computing capabilities. Figure 1 As shown, the method specifically includes:

[0029] Step 101: Determine a position to be retrieved in a video summary according to a received retrieval instruction; wherein the video summary is generated based on at least two frames of an original video.

[0030] A search command refers to a user's instruction to mark a moving object of interest. Specifically, a search command can be issued by clicking on a moving object in a video summary. For example, while watching a video summary, a user may be interested in a moving object and then click on the location of the moving object to issue a search command. A video summary is a summary of the information in the original video, retaining the video information of the moving object of interest while removing redundant content, thereby filtering out useful information from the original video. Video summarization is a video compression technology that processes raw video data, extracts content of interest, and rearranges and presents it using a specific method. This shortens the length of the video data, allowing users to quickly find the content they want to watch. Raw video refers to raw video data directly captured by an image acquisition device. Specifically, with the advent of high-definition video surveillance, 24 / 7 surveillance cameras collect massive amounts of video data. The unprocessed video captured by the surveillance cameras is the raw video, and the compressed video obtained by summarizing the raw video is the video summary.

[0031] Specifically, a search instruction is received from a user containing information about a moving object of interest in a video summary. The moving object information of interest to the user in the search instruction is determined, and the location of the moving object in the video summary is obtained as the location to be searched. For example, when viewing the video summary of an original video, a user finds an object of interest and clicks on the object. The location of the click is obtained as the location to be searched. The object of interest to the user is not limited to a moving object, but can also be any location in the background image, indicating that the user is interested in the moving object appearing at that location in the background.

[0032] Step 102: Determine the distance between the position to be retrieved and the candidate motion trajectory of each moving object in the video summary.

[0033] Since the video summary includes the motion trajectory information of all moving objects in the original video, the probability of the position to be retrieved being on each candidate motion trajectory can be determined by determining the distance between the position to be retrieved and the candidate motion trajectory of each moving object.

[0034] Specifically, after determining the position to be searched, the distance from the position to be searched to each candidate motion trajectory is determined in sequence based on the motion trajectory data of each moving object pre-stored when generating the video summary. Exemplarily, the vertical distance from the position to be searched to each candidate motion trajectory is determined as the distance to each candidate motion trajectory.

[0035] In a feasible embodiment, step 102 includes:

[0036] Based on the DTW algorithm using the cosine angle as a constraint condition, the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary is determined.

[0037] The DTW (Dynamic Time Warping) algorithm calculates vectors of different lengths, and therefore exhibits a certain advantage in calculating the distance between multidimensional vector groups. According to the DTW algorithm principle, assume that any two trajectories S and T have lengths of M and N position points, respectively. The position coordinates of the two trajectories are:

[0038]

[0039] The distance between these two trajectories is

[0040]

[0041] The computational complexity of the DTW algorithm described above is O(M*N). In this embodiment of the present invention, the location to be searched is determined by the user marking a location of interest. Even if M = 1, the computational complexity becomes O(1*N). Therefore, by determining the location to be searched, redundant computations are avoided and computational complexity is reduced.

[0042] Furthermore, in an embodiment of the present invention, the cosine angle is used as a constraint on the termination condition of the DTW algorithm's distance calculation. The cosine angle represents the angle between the distance and the direction of the motion trajectory during the distance calculation. By using the DTW algorithm with the cosine angle as a constraint, the distance from the target location to the candidate motion trajectory of each moving object in the video summary can be determined, thereby improving the accuracy and efficiency of the DTW algorithm's distance determination.

[0043] In a feasible embodiment, the cosine angle is the angle between the target point line and the adjacent point line; wherein the target point line is the line connecting the position to be retrieved and the target point on the candidate motion trajectory, and the adjacent point line is the line connecting the target point and the adjacent points of the target point along the motion direction of the candidate motion trajectory.

[0044] like Figure 2 The diagram below shows the cosine angle determination. Figure 2 Point P is the position to be retrieved, point O is the target point selected on the candidate motion trajectory, and point Q is the position point at the next moment of point O, that is, the next position point along the motion direction of the candidate motion trajectory. Figure 2 The direction of motion of the candidate motion trajectory of points O and Q shown in is from bottom to top. The target point connection line is the line connecting point P to point O, and the adjacent point connection line is the line connecting point Q to point O. According to the law of cosines, the expression of the angle α between the target point connection line and the adjacent point connection line is: Among them, PO is the DTW distance from point P to point O, QO is the DTW distance from point Q to point O, and PQ is the DTW distance from point P to point Q.

[0045] Specifically, point P is the location of the point of interest marked by the user, that is, the location clicked with the mouse, point O is any point selected on the candidate motion trajectory as the target point, and point Q is the location point where the candidate motion trajectory appears in the next frame image of the video frame where the target point is located in the video summary.

[0046] In a feasible embodiment, based on a DTW algorithm using a cosine angle as a constraint condition, determining the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary includes:

[0047] If the cosine angle is a preset angle, the distance between the position to be retrieved and the target point is the distance from the position to be retrieved to the candidate motion trajectory.

[0048] Among them, the preset angle represents the accuracy of the target point selection. According to the determination principle of the cosine angle, when the preset angle is When , the line connecting the position to be retrieved and the target point is perpendicular to the line connecting the target point and the connected point, indicating that the distance between the position to be retrieved and the target point can represent the distance between the position to be retrieved and the candidate motion trajectory.

[0049] The specific steps of distance determination are: traverse each position point on the candidate motion trajectory, select any point as the target point, determine the constraint cosine angle, if the cosine angle The distance between the target point and the position to be retrieved is determined to be the distance from the position to be retrieved to the candidate motion trajectory where the target point is located. Then continue to select the next point on the candidate motion trajectory as the target point, and continue to determine the cosine angle until the cosine angle satisfies When , the calculation is terminated and the distance between the target point and the position to be retrieved is output, which is the distance from the position to be retrieved to the candidate motion trajectory. Figure 3 The figure shows a flow chart of determining the distance from the position to be retrieved to the candidate motion trajectory. Figure 3 As shown, the position to be searched is determined by the position of the mouse clicked by the user, which is represented as point P. According to the distance determination method, the distance from the position to be searched to each candidate motion trajectory is determined in sequence.

[0050] The cosine angle is used as a preset angle constraint to determine the termination time of the distance calculation, which improves the efficiency and accuracy of the distance calculation and avoids distance calculation errors caused by the existence of interference points on the motion trajectory.

[0051] Step 103: Determine the target motion trajectory according to the distance, so as to locate the video sequence in the original video according to the target moving object associated with the target motion trajectory.

[0052] The target motion trajectory represents the motion trajectory where the position to be retrieved is most likely located. After the target motion trajectory is determined, the target moving object can be determined, that is, the target moving object corresponding to the position to be retrieved that the user is interested in is determined.

[0053] After determining the target motion trajectory, the target moving object is determined according to the target motion trajectory to locate the video sequence where the target moving object is located in the original video, and the video sequence is provided to the user to extract the object of interest to the user.

[0054] In a feasible embodiment, determining the target motion trajectory according to the distance includes:

[0055] The candidate motion trajectory with the smallest distance is determined as the target motion trajectory.

[0056] Specifically, when performing trajectory feature retrieval and positioning, that is, determining the target motion trajectory by distance, for all candidate motion trajectories included in the video summary, the trajectory corresponding to the minimum DTW distance is taken as the target motion trajectory. For example, if the video summary includes k candidate motion trajectories, the distance from the position to be retrieved to each candidate motion trajectory is {Dist1, Dist2, ..., Dist k}, the target motion trajectory is Min Dist=Min{Dist1, Dist2,..., Dist k} corresponding to the candidate motion trajectory.

[0057] In one possible embodiment, the method further includes:

[0058] Extracting moving objects from the original video, determining original position information of each moving object in the original video, and establishing an association between each moving object and the original position information; wherein the original position information includes at least the initial time and the end time of each moving object appearing in the original video;

[0059] Accordingly, the video sequence is located in the original video according to the target moving object associated with the target motion trajectory, including:

[0060] determining an associated target motion object according to the target motion trajectory;

[0061] Determining associated original position information according to the target moving object;

[0062] The video sequence in the original video is positioned according to the initial time and the end time in the original position information, so as to locate the target moving object in the original video.

[0063] During the video summary generation process, moving objects are extracted from the original video to determine the position information of the moving objects in each frame of the original video. Specifically, moving objects are extracted and tracked through foreground-background separation, and their original position information is output. The original position information includes the time when each moving object first appears in the original video (the start time) and the time when it last appears in the original video (the end time). An association between each moving object and its original position information is established and stored, allowing subsequent determination of its associated original position information based on the moving object.

[0064] Specifically, based on the pre-established association relationship between the moving object and the original position information, the associated target moving object information can be determined according to the target motion trajectory, and the original position information can be determined according to the association relationship, that is, the first and last appearance times of the target moving object in the original video can be determined, and the video sequence between the time periods in the original video is extracted as the video segment of interest to the user, thereby achieving precise positioning of the target moving object, avoiding the waste of resources for users to search for the video segment of the target moving object in massive original videos, and improving retrieval efficiency.

[0065] The overall retrieval idea of ​​the embodiment of the present invention is: when searching for a video summary, click on any point of interest in the trajectory of the moving object as the position to be retrieved, calculate the distance between the position to be retrieved and all the surrounding candidate motion trajectories, and then output the minimum distance. Determine the trajectory with the shortest distance from the position to be retrieved as the target motion trajectory that the user wants to retrieve. Determine the video sequence of the target motion object in the original video through the original position information of the target motion trajectory to be retrieved (the starting time point information in the original video), and display the video sequence of the object of interest to the user. Position the video through the trajectory and play the video sequence to enable the user to retrieve the original video through the video summary.

[0066] This embodiment of the present invention determines a location to be searched in a video summary based on a received search instruction, determines a target motion trajectory from all motion trajectories based on the distance between the location to be searched and all motion trajectories in the video summary, and determines the video sequence containing the target motion object in the original video based on the target motion trajectory's association with the target motion object in the video summary. This embodiment of the present invention uses a trajectory positioning method to retrieve data from the video summary, thereby improving the efficiency and accuracy of searching for target motion objects in the video summary.

[0067] Example 2

[0068] Figure 4 This is a flow chart of the video summary processing method in the second embodiment of the present invention. This second embodiment is further optimized based on the first embodiment. Before determining the position to be searched in the video summary according to the received search instruction, it also includes the process of generating a video summary of the original video. Figure 4 As shown, the method includes:

[0069] Step 401: Extract moving objects from the original video to determine the original position information and motion trajectory direction information of each moving object in the original video; wherein the original position information at least includes edge position information of the moving object.

[0070] The moving object refers to an object that is in motion in the original video.

[0071] Motion object extraction can be determined based on the motion object tracking method. Motion object tracking is to find the motion object of interest in each image in a video image and determine the motion parameters such as the position, speed and acceleration of the motion object. Motion object tracking is a commonly used technical means in this field, and the specific principles will not be repeated here.

[0072] Through the foreground-background separation method, the moving objects in the original video are extracted, and the video sequence that does not contain the moving objects in the original video is removed to obtain the original video containing the moving objects. The original video containing the moving objects is then tracked. After the tracking is completed using the tracking algorithm, the position information and motion trajectory direction information of the moving objects in each video frame in each original video are output. The position and direction of each moving object are connected together to form the motion trajectory of the moving object.

[0073] The position information of the moving object refers to the position information of the moving object in the image, such as edge position information or center of mass position information. The motion trajectory direction information is used to represent the running angle of the moving object. Specifically, a moving object is tracked in the original video to obtain N position points, that is, the motion trajectory of the moving object can be expressed as {(x1, y1), ..., (x i ,y i ),...,(x N ,y N )} indicates that the position point coordinates can be determined by the center of mass coordinates of the moving object. The initial position point of the moving object in the original video is (x1, y1), and the position of the moving object at the target time is (x i ,y i ), the motion trajectory direction information of the moving object at the target moment can be determined by the angle between the position point at the target moment and the position point at the next moment, that is, The motion trajectory direction information is represented. For a motion trajectory with N position points in the original video, N-1 motion trajectory direction information is included to determine the motion direction of the moving object at any time.

[0074] Step 402: Extract video frames from the original video according to the motion trajectory direction information to obtain target video frames.

[0075] Since a video summary is a summary of the original video, if the original video contains a large number of video frames of a moving object, it is necessary to extract the video frames containing the moving object and determine the video summary based on the extracted video frames, thereby compressing the original video. However, when extracting the video frames containing the moving object from the original video, if a random extraction method is adopted, the video frames in which the moving object's direction changes may be lost, resulting in poor motion continuity of the moving object in the obtained video summary. For example, if a moving object has been walking in a straight line but suddenly changes direction to the right at a certain moment, the random extraction method may not extract the video frames in which the direction changes. In the obtained video summary, the moving object will suddenly change from the position of walking in a straight line to a certain position on the right side, resulting in a poor viewing experience for the user and the loss of important information.

[0076] In an embodiment of the present invention, video frames in the original video are extracted based on motion trajectory direction information to avoid the problem of discontinuous video summary playback caused by video frames with sudden changes in direction not appearing in the video summary. For example, the motion trajectory direction information of each moving object in the original video is determined. If the difference between the motion trajectory direction of any moving object in the video frame at the target moment and the motion trajectory direction at the previous moment exceeds a preset angle, the video frame at the target moment is extracted as the target video frame. The remaining video frames that do not exceed the preset angle can be randomly extracted or intermittently extracted based on the motion trajectory direction angle to obtain the target video frame.

[0077] When extracting target video frames from an original video containing a moving object, the motion trajectory direction information can ensure that no important information is missed in the extracted target video frames, so that the motion of the moving object in the video summary determined based on the target video frames is continuous, thereby improving the accuracy of the video summary.

[0078] Step 403: extract all moving objects in the target video frame according to their edge position information.

[0079] All moving objects are extracted based on edge position information, that is, the moving objects in each frame of the target video frame are extracted to obtain the location information of the target moving objects and the pixel value information of the location points. Specifically, the location information of the moving objects in each frame of the target video frame and the pixel value information of each location point are extracted.

[0080] Step 404: Generate a video summary of the original video based on the extracted moving objects and the preset background image.

[0081] The extracted moving object is fused with a pre-set background image, and the pixel values ​​at each position of the moving object extracted based on the edge position information are used to replace the pixel values ​​at the corresponding positions in the background to obtain a video summary. The pre-set background image in the embodiments of the present invention refers to a background image of a fixed monitoring scene. For a surveillance camera, it is fixed at a certain position, and the monitored scene is also fixed. Therefore, the background image can be pre-set based on the actual monitoring scene.

[0082] Specifically, the following formula is used to generate the video summary:

[0083]

[0084] Among them, F' t (x i ,y i ) refers to the tth video frame in the video summary at (x i ,y i ) pixel value, F t (x i ,y i ) refers to the target video frame in the t frame (x i ,y i ) pixel value of the moving object at point B t (x i ,y i ) refers to the background image (x i ,y i ) point. (x i ,y i )∈Foregroud means that the point is located in the edge position information of the moving object in the t-th frame of the target video frame, (x i ,y i )∈Backgroud means that the point is outside the edge position information of the moving object in the tth frame of the target video frame.

[0085] Specifically, in the video summary, if the target point of the target frame is in the area where the moving object is located in the corresponding target video frame, the pixel value of the point adopts the pixel value of the extracted moving object; if not, the pixel value of the point adopts the pixel value of the background image to achieve the generation of the video summary.

[0086] like Figure 5The figure shows a flowchart for video summary generation. Moving objects are detected in the original video sequence. Video sequences without moving objects are removed. Moving objects are extracted from each frame of the video sequence containing moving objects. A tracking algorithm is initialized to obtain the centroid coordinates of all moving objects in each frame. The direction of each moving object's trajectory is determined and a motion trajectory is generated. Finally, the video summary sequence is output by fusing the foreground and background images of the moving objects.

[0087] In the embodiments of the present invention, a foreground-background separation method is used to extract and track moving objects and video clips containing them. After the tracking algorithm completes tracking, it outputs the moving object's position and motion trajectory direction information. These information is then linked together to form the moving object's trajectory. The extracted moving objects are then combined in a specific temporal and spatial order to form a video summary. Incorporating motion trajectory direction information into the video summary generation process improves the accuracy of the generated video summary, enhancing the richness and accuracy of the information contained in the video summary.

[0088] Example 3

[0089] Figure 6 This is a schematic diagram of the structure of the video summary processing device in the third embodiment of the present invention. This embodiment is applicable to the case where the retrieval motion object is located in the original video according to the retrieval motion object determined by the user from the video summary. Figure 6 As shown, the device includes:

[0090] The to-be-searched position determination module 610 is configured to determine the to-be-searched position in the video summary according to the received search instruction; wherein the video summary is generated based on at least two frames of the original video;

[0091] a distance determination module 620 for determining the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary;

[0092] The motion trajectory determination module 630 is configured to determine a target motion trajectory according to the distance, so as to locate a video sequence in the original video according to the target moving object associated with the target motion trajectory.

[0093] This embodiment of the present invention determines a location to be searched in a video summary based on a received search instruction, determines a target motion trajectory from all motion trajectories based on the distance between the location to be searched and all motion trajectories in the video summary, and determines the video sequence containing the target motion object in the original video based on the target motion trajectory's association with the target motion object in the video summary. This embodiment of the present invention uses a trajectory positioning method to retrieve data from the video summary, thereby improving the efficiency and accuracy of searching for target motion objects in the video summary.

[0094] Optionally, the distance determination module includes a DTW distance determination unit, configured to:

[0095] Based on a DTW algorithm using a cosine angle as a constraint condition, a distance from the to-be-retrieved position to a candidate motion trajectory of each moving object in the video summary is determined.

[0096] Optionally, the cosine angle is the angle between the target point line and the adjacent point line; wherein, the target point line is the line between the position to be retrieved and the target point on the candidate motion trajectory, and the adjacent point line is the line between the target point and the adjacent point of the target point along the motion direction of the candidate motion trajectory.

[0097] Optionally, the DTW distance determination unit is specifically configured to:

[0098] If the cosine angle is a preset angle, the distance between the position to be retrieved and the target point is the distance from the position to be retrieved to the candidate motion trajectory.

[0099] Optionally, the motion trajectory determination module is specifically used to:

[0100] The candidate motion trajectory with the smallest distance is determined as the target motion trajectory.

[0101] Optionally, the device further includes a moving object extraction unit, configured to:

[0102] Extracting moving objects from the original video, determining original position information of each moving object in the original video, and establishing an association between each moving object and the original position information; wherein the original position information includes at least the initial time and the end time of each moving object appearing in the original video;

[0103] Accordingly, the motion trajectory determination module is specifically used to:

[0104] determining an associated target motion object according to the target motion trajectory;

[0105] Determining associated original position information according to the target moving object;

[0106] The video sequence in the original video is positioned according to the initial time and the end time in the original position information, so as to position the target moving object in the original video.

[0107] Optionally, the apparatus further comprises a video summary generation module configured to extract moving objects from the original video before determining the position to be retrieved in the video summary according to the received retrieval instruction, and determine original position information and motion trajectory direction information of each moving object in the original video; wherein the original position information includes at least edge position information of the moving object;

[0108] Extracting video frames from the original video according to the motion trajectory direction information to obtain target video frames;

[0109] Extracting all moving objects according to edge position information of all moving objects in the target video frame;

[0110] A video summary of the original video is generated according to the extracted moving objects and a preset background image.

[0111] The video summarization processing device provided in the embodiment of the present invention can execute the video summarization processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the video summarization processing method.

[0112] Example 4

[0113] Figure 7 This is a structural diagram of an electronic device provided in Example 4 of the present invention. Figure 7 A block diagram of an exemplary electronic device 12 suitable for implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0114] like Figure 7 As shown, electronic device 12 is implemented as a general-purpose computing device. Components of electronic device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components (including system memory 28 and processing unit 16).

[0115] Bus 18 represents one or more of several types of bus structures, including a storage device bus or storage device controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0116] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0117] The system storage device 28 may include computer system readable media in the form of volatile storage devices, such as random access memory devices (RAM) 30 and / or cache memory devices 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Although not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Storage device 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0118] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in storage device 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0119] The electronic device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the device 12, and / or any device that enables the device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20. Figure 7 As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0120] The processing unit 16 executes various functional applications and data processing by running programs stored in the system storage device 28, such as implementing the video summarization processing method provided in an embodiment of the present invention, including:

[0121] Determining a position to be retrieved in the video summary according to the received retrieval instruction; wherein the video summary is generated based on at least two frames of images of the original video;

[0122] Determining the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary;

[0123] A target motion trajectory is determined according to the distance, so as to locate a video sequence in the original video according to the target moving object associated with the target motion trajectory.

[0124] Example 5

[0125] The fifth embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the video summarization processing method provided in the embodiment of the present invention is implemented, including:

[0126] Determining a position to be retrieved in the video summary according to the received retrieval instruction; wherein the video summary is generated based on at least two frames of images of the original video;

[0127] Determining the distance from the to-be-retrieved position to the candidate motion trajectory of each moving object in the video summary;

[0128] A target motion trajectory is determined according to the distance, so as to locate a video sequence in the original video according to the target moving object associated with the target motion trajectory.

[0129] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by an instruction execution system, device or device or used in combination with it.

[0130] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0132] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0133] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video summarization processing method, characterized in that: include: Determining a position to be retrieved in the video summary according to the received retrieval instruction; wherein the video summary is generated based on at least two frames of the original video; the position to be retrieved includes the position of a moving object of interest to the user or any position point in the background image; Determine the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary using a DTW algorithm with a cosine angle as a constraint based on a termination condition; wherein the cosine angle is the angle between a target point connection line and a line connecting adjacent points; the target point connection line is a line connecting the position to be retrieved and a target point on the candidate motion trajectory, and the adjacent point connection line is a line connecting the target point and adjacent points of the target point along the motion direction of the candidate motion trajectory; determining a target motion trajectory according to the distance, so as to locate a video sequence in the original video according to a target moving object associated with the target motion trajectory; The method of determining the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary based on the DTW algorithm using the cosine angle as a constraint condition includes: If the cosine angle is The distance between the position to be retrieved and the target point is the distance from the position to be retrieved to the candidate motion trajectory; Before determining the position to be searched in the video summary according to the received search instruction, the method further includes: Extracting moving objects from the original video to determine original position information and motion trajectory direction information of each moving object in the original video; wherein the original position information at least includes edge position information of the moving object; If the difference between the motion trajectory direction of any moving object in the video frame at the target moment and the motion trajectory direction at the previous moment exceeds a preset angle, the video frame at the target moment is extracted as the target video frame; the remaining video frames that do not exceed the preset angle are randomly extracted or intermittently extracted according to the motion trajectory direction angle to obtain the target video frame; Extracting all moving objects according to edge position information of all moving objects in the target video frame; A video summary of the original video is generated according to the extracted moving objects and a preset background image.

2. The method according to claim 1, characterized in that Determining a target motion trajectory according to the distance includes: The candidate motion trajectory with the smallest distance is determined as the target motion trajectory.

3. The method according to claim 1, characterized in that The method further comprises: Extracting moving objects from the original video, determining original position information of each moving object in the original video, and establishing an association between each moving object and the original position information; wherein the original position information includes at least the initial time and the end time of each moving object appearing in the original video; Accordingly, locating a video sequence in the original video according to the target moving object associated with the target motion trajectory includes: determining an associated target motion object according to the target motion trajectory; Determining associated original position information according to the target moving object; The video sequence in the original video is positioned according to the initial time and the end time in the original position information, so as to position the target moving object in the original video.

4. A video summary processing device, characterized in that: include: a module for determining a position to be retrieved, configured to determine a position to be retrieved in the video summary according to a received retrieval instruction; wherein the video summary is generated based on at least two frames of an original video; and the position to be retrieved includes the position of a moving object of interest to the user or any position point in the background image; a distance determination module, configured to determine the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary using a DTW algorithm based on a computation termination condition and using a cosine angle as a constraint; wherein the cosine angle is the angle between a target point connection line and a line connecting adjacent points; the target point connection line is a line connecting the position to be retrieved and a target point on the candidate motion trajectory; and the adjacent point connection line is a line connecting the target point and adjacent points of the target point along the motion direction of the candidate motion trajectory; a motion trajectory determination module, configured to determine a target motion trajectory according to the distance, so as to locate a video sequence in the original video according to a target moving object associated with the target motion trajectory; The method of determining the distance from the position to be retrieved to the candidate motion trajectory of each moving object in the video summary based on the DTW algorithm using the cosine angle as a constraint condition includes: If the cosine angle is The distance between the position to be retrieved and the target point is the distance from the position to be retrieved to the candidate motion trajectory; The device also includes a video summary generation module, which is used to extract moving objects from the original video before determining the position to be retrieved in the video summary according to the received retrieval instruction, and determine the original position information and motion trajectory direction information of each moving object in the original video; wherein the original position information at least includes edge position information of the moving object; if the difference between the motion trajectory direction of any moving object in the video frame at the target moment and the motion trajectory direction at the previous moment exceeds a preset angle, the video frame at the target moment is extracted as the target video frame; the remaining video frames that do not exceed the preset angle are randomly extracted or intermittently extracted according to the motion trajectory direction angle to obtain the target video frame; all moving objects are extracted according to the edge position information of all moving objects in the target video frame; and a video summary of the original video is generated based on the extracted moving objects and a pre-set background image.

5. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video summarization processing method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video summarization processing method according to any one of claims 1 to 3 is implemented.

Images