Hierarchical motion representation and extraction from monocular still camera videos
Through intra-mode estimation and inter-mode matching, combined with binary graphs and motion vector clustering, the problems of high motion detection cost and poor accuracy in monocular still camera video sequences are solved, and the calculation efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202080048996.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-05-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-05-18
AI Technical Summary
In the prior art, when capturing video sequences using a monocular still camera, the calculation cost of motion detection is high and the accuracy is limited, making it difficult to effectively segment the moving objects and infer the hierarchical order.
Intra-mode estimation and inter-mode matching methods are used to generate binary graph mark label changes, non-stationary image segments are detected, and the moving segments are represented as seven-parameter vectors for clustering, so as to realize motion detection and hierarchical order inference.
It reduces the computational cost, improves the accuracy of motion detection, can effectively segment the motion data in the video and infer the hierarchical order, and is suitable for video sequences captured by monocular still cameras.
Smart Images

Figure CN114072839B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to motion detection in images, and more particularly to segmenting motion and inferring layering order in video sequences captured using a monocular still camera. Background Art
[0002] Moving object detection is a technique used in computer vision and image processing. Multiple consecutive frames in a video are compared using various methods to determine if any moving objects are detected. Moving object detection has been widely used in video surveillance, human activity analysis, road condition monitoring, airport security, and coastal protection monitoring. In object detection systems, sometimes the objects are not predefined. After object segmentation, measuring the motion of objects is also useful for video scene understanding.
[0003] In an image frame, the layer order defines what is on top of what. It is a model for defining the depth order (3D) of objects. In one example, in a typical painting, the first layer is the sky, which is the farthest layer. The next layer may contain buildings and airplanes. The next layer below may be cars in front of the buildings, etc. The car layer may partially hide the buildings. In front of the car may be a person, which hides part of the car. The person is the last layer because there is nothing covering the person, so the person remains unobstructed. Typical motion detection systems need to understand the local depth order to perform motion detection and action recognition.
[0004] However, known state-of-the-art motion estimation and segmentation methods are computationally expensive because they use expensive optical flow features, dynamic cameras, and treat all pixels equally. Existing methods can be implemented using graphics processing units (GPUs), which are electronic circuits specialized for parallel image processing and machine learning, and are computationally expensive. If frames are captured using a monocular still camera, the computational complexity can be reduced, but the accuracy of motion detection may be affected.
[0005] In view of the above, there is a need for an object detection system that is computationally inexpensive and can accurately segment motion and infer hierarchical order in video sequences captured with a monocular stationary camera. Summary of the invention
[0006] According to an embodiment of the present disclosure, a method for segmenting motion data in a video and inferring a hierarchical order is provided. The method may include: using intra-frame pattern estimation to segment a first image frame of a video, and using inter-frame pattern matching to segment one or more other image frames of the video. The method may also include: generating a label change map for the current segmented image frame by generating a binary map of the current segmented image frame, and then marking one or more pixels whose labels have changed relative to the previously segmented image frame. The method may also include: detecting one or more non-static image segments of the current segmented image frame based on the label change map, and detecting one or more motion image segments in the one or more non-static image segments. The method may also include: estimating the motion of one or more motion image segments, representing each motion segment as a vector with seven parameters, and clustering one or more similar motion segment vectors based on the corresponding one or more parameters.
[0007] According to another embodiment of the present disclosure, a system for segmenting motion data in a video and inferring a hierarchical order is provided. The system includes: a memory for storing one or more instructions, and a processor communicatively coupled to the memory. The processor may include: a video segmentation module for segmenting a first image frame of a video using intra-frame pattern estimation, and segmenting one or more other image frames of the video using inter-frame pattern matching. The system may also include: a label frame difference generation module for generating a label change map for a current segmented image frame by generating a binary map of the current segmented image frame, and then marking one or more pixels whose labels have changed relative to the previous segmented image frame. The system may also include: a non-static segment detection module for detecting one or more non-static image segments of the current segmented image frame based on the label change map. The system may also include: a motion segment detection module for detecting one or more motion image segments in one or more non-static image segments. The system may also include: a motion estimation module for estimating the motion of one or more motion image segments; and a motion segmentation module for representing each motion segment as a vector with seven parameters, and clustering one or more similar motion segment vectors based on the corresponding one or more parameters.
[0008] According to another embodiment of the present disclosure, a computer programmable product for segmenting motion data in a video and inferring a hierarchical order is provided. The computer programmable product includes a set of instructions, which, when executed by a processor, causes the processor to: segment a first image frame of a video using intra-frame pattern estimation; segment one or more other image frames of a video using inter-frame pattern matching; generate a label change map for a current segmented image frame by generating a binary map of the current segmented image frame; then mark one or more pixels whose labels have changed relative to a previously segmented image frame; detect one or more non-static image segments of the current segmented image frame based on the label change map; detect one or more motion image segments in one or more non-static image segments; estimate the motion of one or more motion image segments; represent each motion segment as a vector with seven parameters; and cluster one or more similar motion segment vectors based on the corresponding one or more parameters.
[0009] According to another embodiment of the present disclosure, a monocular still camera surveillance system is provided, which includes a monocular still camera for capturing video, a memory for storing one or more instructions, and a processor, which is coupled to the memory and configured to execute the one or more instructions stored in the memory to: segment a first image frame of the video using intra-frame pattern estimation; segment one or more other image frames of the video using inter-frame pattern matching; generate a label change map for the current segmented image frame by generating a binary map of the current segmented image frame; then mark one or more pixels whose labels have changed relative to the previous segmented image frame; detect one or more non-still image segments of the current segmented image frame based on the label change map; detect one or more motion image segments in the one or more non-still image segments; estimate the motion of the one or more motion image segments; represent each motion segment as a vector with seven parameters; cluster one or more similar motion segment vectors based on the corresponding one or more parameters; and perform motion detection and activity recognition based on one or more clusters.
[0010] It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] When read in conjunction with the accompanying drawings, the above overview and the following detailed description of illustrative embodiments will be better understood. In order to illustrate the present disclosure, exemplary configurations of the present disclosure are shown in the accompanying drawings. However, the present disclosure is not limited to the specific methods and means disclosed herein. In addition, it will be appreciated by those skilled in the art that the accompanying drawings are not drawn to scale. Where possible, similar elements are represented by the same numerals.
[0012] Figure 1illustrates an environment in which various embodiments of the present disclosure may be practiced;
[0013] Figure 2 A motion detection system for motion detection and segmentation of a video sequence captured by a camera according to an embodiment of the present disclosure is illustrated;
[0014] Figure 3A and 3B 1 to 6 consecutive video frames and their corresponding mean-shift segmented images according to an embodiment of the present disclosure are illustrated;
[0015] Figure 4 illustrates an exemplary video sequence depicting still and non-still image segments according to an embodiment of the present disclosure;
[0016] Figure 5A illustrates exemplary video sequences depicting various types of non-still image segments according to embodiments of the present disclosure;
[0017] Figure 5B shows a ternary segmentation map generated by a motion segment detection module for a video sequence according to an embodiment of the present disclosure; and
[0018] Figure 6 is a flow chart illustrating a method for performing motion detection and segmentation on a video sequence captured by a camera according to an embodiment of the present disclosure.
[0019] In the accompanying drawings, underlined numbers are used to represent the item in which the underlined number is located or the item to which the underlined number is adjacent. Non-underlined numbers are related to items as described below, that is, the item is identified by a line connecting the non-underlined number and the item. When a number is not underlined and is accompanied by an associated arrow, the non-underlined number is used to identify the general item to which the arrow points. DETAILED DESCRIPTION
[0020] The detailed description of the accompanying drawings is intended as a description of the currently preferred embodiments of the present disclosure, and is not intended to represent the only form in which the present disclosure can be implemented. It should be understood that the same or equivalent functions can be implemented by different embodiments intended to be included in the spirit and scope of the present disclosure.
[0021] As used in the specification and claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. For example, the term "an item" may include a plurality of items unless the context clearly dictates otherwise.
[0022] It will be appreciated by those skilled in the art that the elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the size of some elements in the drawings may be exaggerated relative to other elements to improve understanding of the present disclosure.
[0023] There may be additional components in the aforementioned application that are not depicted on one of the described drawings. If such components are described but not depicted in the drawings, the absence of such drawings should not be considered as an omission of such design from the specification.
[0024] As required, detailed embodiments of the present disclosure are disclosed herein; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be embodied in various forms. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but only as a basis for the claims and as a representative basis for teaching those skilled in the art to apply the present disclosure in various ways in virtually any appropriate detailed structure. In addition, the terms and phrases used herein are not intended to be limiting, but to provide an understandable description of the present disclosure.
[0025] The following detailed description illustrates embodiments of the present disclosure and the manner in which they can be implemented. Although the best mode for carrying out the present disclosure has been disclosed, those skilled in the art will recognize that other embodiments for carrying out or practicing the present disclosure are also possible.
[0026] Figure 1 An environment 100 is shown in which various embodiments of the present disclosure may be practiced. The environment 100 includes a camera 101 , a motion detection system 102 , and a computing device 103 .
[0027] The camera 101 is configured to capture and store video of one or more objects in real time. Examples of the camera 101 include, but are not limited to, a monocular still camera primarily used for surveillance purposes.
[0028] The motion detection system 102 is configured to perform motion detection and segmentation on the video sequence captured by the camera 101 .
[0029] The motion detection system 102 includes a central processing unit (CPU) 106, an operation panel 108, and a memory 110. The CPU 106 is a processor, computer, microcontroller, or other circuit that controls the operation of various components such as the operation panel 108 and the memory 110. The CPU 106 can execute software, firmware, and / or other instructions stored, for example, on a volatile or non-volatile memory (e.g., the memory 110) or otherwise provided to the CPU 106. The CPU 106 can be connected to the operation panel 108 and the memory 110 via a wired or wireless connection, such as one or more system buses, cables, or other interfaces. In an embodiment of the present disclosure, the CPU 106 can include custom graphics processing unit (GPU) server software to provide real-time object detection and prediction for all cameras on a local network.
[0030] The operation panel 108 may be a user interface of the image forming apparatus 100 and may be in the form of a physical keyboard or a touch screen. The operation panel 108 may receive input related to selected functions, preferences and / or authentication from one or more users and may provide and / or receive input visually and / or audibly.
[0031] In addition to storing instructions and / or data for use by CPU 106 in managing the operation of image forming device 100, memory 110 may also include user information associated with one or more users of image forming device 100. For example, the user information may include authentication information (e.g., a username / password pair), user preferences, and other user-specific information. CPU 106 may access this data to assist in providing control functions (e.g., sending and / or receiving one or more control signals) related to the operation of operating panel 108 and memory 110.
[0032] The camera 101 and the motion detection system 102 may be controlled / operated by a computing device 103. Examples of computing devices 103 include smart phones, personal computers, laptops, etc. The computing device 103 enables a user / operator to view and save videos captured by the camera 101 and access videos / images processed by the motion detection system 102. The computing device 103 may execute a mobile application of the motion detection system 102, thereby enabling a user to access and process the video stream captured by the camera 101.
[0033] The camera 101, the motion detection system 102, and the computing device 103 may be connected to each other via a wired connection or via a wireless communication network 104. The communication network 104 may be any suitable wired network, a wireless network, a combination of these networks, or any other conventional network without limiting the scope of the present disclosure. A few examples may include a local area network (LAN), a wireless LAN connection, an Internet connection, a point-to-point connection, or other network connections and combinations thereof.
[0034] In an embodiment, camera 101, motion detection system 102, and computing device 103 may be integrated into a single device, where the single device is a portable smartphone with a built-in camera and display.
[0035] Figure 2 A motion detection system 102 for motion detection and segmentation of a video sequence captured by a camera 101 according to an embodiment of the present disclosure is illustrated.
[0036] The motion detection system 102 includes a CPU 106 , which includes a segmentation module 201 , a label frame difference generation module 202 , a non-static segment detection module 203 , a motion segment detection module 204 , a first motion estimation module 205 , a second motion estimation module 206 and a motion segmentation module 207 .
[0037] In an embodiment of the present disclosure, each of the seven components 201-207 can exist on a single physical computer, or exist on more than one computer and communicate through a message passing protocol. If these seven components are running on one computer, they can run in different processes that communicate through a communication protocol. If they are running in a single process, they may run as threads. A thread is the smallest processing unit that can be executed in an operating system. In most modern operating systems, threads exist in processes, that is, a single process may contain multiple threads. If running in a single process, they can also use reactive design to enable these components to share one or more threads. Components 201-207 can be implemented in a programming language.
[0038] The segmentation module 201 is configured to perform video segmentation on the video captured by the camera 101 using a joint space-time-range adaptive mean shift. Video segmentation aims to segment (moving) objects and backgrounds along the time direction. The motion area in the image is associated with the motion of certain foreground / background objects / areas (such as foreground people, background trees, or roads). In an embodiment of the present disclosure, video segmentation based on mean shift is based on video segmentation, where the segmentation of moving / static objects / background is obtained by inter-frame pattern matching and motion vector pattern estimation in consecutive frames. The idea of video segmentation by pattern matching is that the pattern of pixels in the current frame should be consistent with the pattern of a subset of data (or regions) representing the same object or region.
[0039] Objects / regions that newly appear in the current frame due to new foreground objects or uncovered background areas are segmented by intra-frame pattern estimation. For fast processing, the segmentation module 201 uses a rectangular 6-dimensional radially symmetric kernel (a hypersphere of radius h in the X, Y, T, R, G, B spatiotemporal multi-channel space) to perform mean-shift based video segmentation. The data space should be properly normalized to account for the different physical meanings and ranges of the variables.
[0040] Figure 3A and 3B The first to sixth consecutive video frames 302a to 312a and their corresponding mean-shifted segmented images 302b to 312b are illustrated. Each segment in the segmented image is color-coded by the mean-shift convergence mode color (a weighted average), and the segment boundaries are highlighted.
[0041] In an embodiment of the present disclosure, the first frame 302a of the sequence is segmented by intra-frame pattern estimation. Intra-frame pattern estimation is mainly used to handle new areas introduced in the current frame, i.e., new foreground objects or revealing previously occluded backgrounds. Each pixel in the corresponding segmented image 302b is represented by its convergence pattern. Pixels that converge to the same pattern are linked and assigned the same label using fast propagation, and a label map is also generated for the first frame 302a.
[0042] Starting from the second frame 304a, segmentation is performed using inter-frame pattern matching. Inter-frame pattern matching is used to find similar patterns between pixels in the current frame and pixels in the previous frame. This can be used to track corresponding motion / static areas through image frames. In an embodiment of the present disclosure, each pixel in the second segmented frame 304b is assigned to the mean shift pattern it searches for in the segmented previous frame 302b. This subtle detail of the proposed segmentation algorithm is crucial to this application because it leads to very stable results with smooth spatiotemporal label tubes. The better stability is because the current frame 304a is segmented by density pattern lookup in the segmented previous frame 302b.
[0043] In an embodiment of the present disclosure, the mean shift segmentation method also includes propagating segmentation labels. Pixels in the current frame 304a are assigned the most frequent label found within the kernel distance in the previously segmented frame 302b. As a faster version of the algorithm, it is checked whether pixels with the same spatial coordinates in the previous frame 302b are found within the kernel radius distance in 6D space, and labels are assigned to them when the test is true. When a pixel within the kernel distance is not found in the previous frame 302b, the pixel is assigned a new label unless there is a labeled pixel within the kernel radius distance in the current frame 304a. Therefore, the region correspondence is resolved and new labels are assigned to new regions, generated by uncovered background or objects entering the scene. Except for segments that touch the image support boundary, the motion of the corresponding segments is set to zero.
[0044] Back to Figure 2 , the label frame difference generation module 202 is configured to generate a label change map for each current segmented image frame. In an embodiment of the present disclosure, the label change map is formed by generating a binary map of the current segmented image frame, and then marking pixels whose labels have changed from the previous segmented image frame. Pixels with labels that have changed from the previous segmented image frame may belong to one of the following categories: occluded or unoccluded parts of old objects or parts of new objects entering the scene.
[0045] The non-static segment detection module 203 is configured to detect and discard each static image segment from the current segmented image frame. The static image segment has zero motion and does not need further processing, while the non-static segments need to be further classified as static or motion segments. The motion vectors of static segments are zero because they correspond to static objects that are covered or uncovered.
[0046] Typically, images captured with a static camera contain large areas of no motion. Therefore, a segment generated in a previous image frame belonging to a no motion region does not change its shape or content in the corresponding current image frame. However, the presence of noise and illumination variations (which should be low between consecutive frames) may cause minor changes. For example, it can be seen that the actual boundaries of the segments on the table (see Figure 3A and 3B , the first segmented image 302b to the sixth segmented image 312b) are artificially created by mean shift clustering and do not correspond to object boundaries. Slight shadows and highlights, as well as lighting gradients on the table may contribute to the formation of segments on the table. This can be solved by checking the gradients on the segment boundaries. However, this may cause problems when segmenting objects with diffuse edges (such as clouds) or objects with low contrast relative to their background.
[0047] In an embodiment of the present disclosure, the non-static segment detection module 203 detects and discards each static image segment of the current image frame by checking the cardinality of the intersection on the union set calculated based on the segment and its counterpart in the previous frame against a threshold.
[0048] Figure 4 An exemplary video sequence 400 depicting still and non-still image segments is illustrated in accordance with an embodiment of the present disclosure.
[0049] The exemplary video sequence 400 includes a first segmented image frame 402 including three segments, namely a first square segment 404 and a second square segment 406 on a black background segment 408. A second segmented image frame 410 is the next segmented image frame of the video sequence 400. In the second segmented image frame 410, the first square segment 404 has moved toward the second square segment 406, while the second square segment 406 is stationary. A third image 412 highlights white pixels that have changed during the transition from the first segmented image frame 402 to the second segmented image frame 404. Therefore, the second square segment 406 can be detected as a stationary segment and discarded therefrom.
[0050] Return to reference Figure 2, the motion segment detection module 204 is configured to identify static segments and moving non-static segments in the current segmented image frame. The motion segment detection module 204 first identifies segments that do not change shape, that is, they do not change cardinality. These segments represent the top local layer, and their motion can be accurately estimated by robust regression and an affine model. The remaining segments can be static or dynamic. Static segments change shape when they are covered or uncovered by segments belonging to objects moving in higher layers. Dynamic segments may change their shape for various reasons, such as being occluded by another moving segment or due to changes in perspective. Static segments have a large proportion of boundary pixels that are fixed. Moving non-static segments can be of two types, namely, unoccluded moving segments and occluded moving segments.
[0051] Figure 5A Illustrated is an exemplary video sequence 500 depicting various types of non-still image segments in accordance with an embodiment of the present disclosure.
[0052] The video sequence 500 includes a first image frame 502 including four segments, namely a first square segment 504, a second square segment 506, a third square segment 508, and a black background segment 510. The black background segment 510 is static and occluded by all the other three segments 504, 506, and 508.
[0053] As seen in the second image frame 512, the first square segment 504 moves one pixel down and three pixels to the right and is unoccluded. As the first square segment 504 moves, it partially occludes the second square segment 506. The second square segment 506 also moves one pixel up and one pixel to the right, but it is occluded by the first square segment 504. The third square segment 508 is static and unoccluded.
[0054] The third image 514 highlights pixels that have changed during the transition from the first frame 502 to the second frame 512 .
[0055] Figure 5B A ternary segmentation map 516 generated by the motion segment detection module 204 for the video sequence 500 according to an embodiment of the present disclosure is shown. The ternary segmentation map 516 includes motion non-stationary segments 518 and static non-stationary segments 520. The third square segment 508 is static and is therefore part of the static non-stationary segment 520. The static non-stationary segment 520 has zero motion and does not require further processing. The motion segment 518 can be formed by unoccluded motion segments and occluded motion segments.
[0056] Return to reference Figure 2The first motion estimation module 205 is configured to estimate the motion of unoccluded motion segments such as the first square segment 504 (see Figure 5A The first square segment 504 retains its size and shape, and is therefore not occluded, and its position in the first image frame 502 and the second image frame 512 is known. In an embodiment of the present disclosure, the motion of the first square segment 504 can be accurately represented by a general motion model (e.g., a 2D affine model). The model parameters can be estimated by a robust registration method.
[0057] Return to reference Figure 2 The second motion estimation module 206 is configured to estimate the motion of the occluded motion segment such as the second square segment 506 (see Figure 5A second image frame 512). For occluded motion segments, it is expected that the motion estimation accuracy will decrease. If there is only one occluded segment and the occluded segment itself is not occluded, the motion of the occluded segment can be resolved by estimating the motion if the boundary pixels of the occluded segment are not shared with the occluding segments. Since most segments are small, this assumption can be accepted as true. Similarly, to estimate the motion of selected boundary pixels, robust regression using an affine model is a tool because it can tolerate a large amount of outlier data. Robust regression is a form of regression analysis that aims to overcome some of the limitations of traditional parametric and non-parametric methods, where regression analysis seeks to find a relationship between one or more independent variables and a dependent variable.
[0058] The affine model can be defined as:
[0059] x'=xs cosα–y sinα+t x
[0060] y'=x sinα+ys cosα+t y
[0061] It is defined by four parameters: s, α, t x and t y These parameters are estimated from pairs of segments in adjacent frames using robust registration.
[0062] Return to reference Figure 2 , the motion segmentation module 207 is configured to first represent each segment as a vector with seven parameters. In the embodiment of the present disclosure, the seven parameters include two spatial coordinates of the centroid of the corresponding segment, four affine parameters including translation and rotation parameters corresponding to the similarity transformation, and a segment weight defined by the total number of pixels of the segment. In one example, a typical segment vector v is in the form of v = [s, α, t x,t y ,x c ,y c ] and weight n p , where s, α, t x and t y are four affine parameters, x c and c are the spatial coordinates of the center of mass of the corresponding segment.
[0063] Once the segment vector for each segment is determined, the motion segmentation module 207 is configured to form one or more clusters of one or more segment vectors based on the corresponding first to sixth parameters. Clustering is a method of grouping similar entities. For two vectors, the similarity may be a distance function. The similarity measure required in clustering uses the first six parameters of the segment vector, except for the segment weight. It acts like copying the vector of any segment multiplied by its number of pixels (its cardinality). In an example, if only two segments are grouped in a cluster, the cluster center can be updated to the weighted average of the segment vectors: v c =(n p1 *v1+n p2 *v 2) / (n p1 +n p2 ), where n p1 and n p2 are the individual weights of the two fragments.
[0064] The clustering process helps to group the segments to find moving objects or parts of articulated objects, such as a human body (like a leg, forearm, or head) with similar motions. All parts of a moving object are assumed to have similar (not identical) motions. The similarity defines the granularity of the representation. A person may walk at an average speed of 5 km / h, but the legs and arms can move independently. In activity recognition applications, the motion of each limb must be characterized.
[0065] In an embodiment of the present disclosure, parts of an object with different colors or textures may have similar motions between several frames, even though the object may be three-dimensional and may be rotated about any axis. In order to group segments with similar motions, mean shift clustering in the 6D space - translation - rotation - scale space may be used again. In order to take into account the size of the segments, the weighted sum in the algorithm may be calculated as if the centroid of the segment had been copied for each of its pixels. The mean shift algorithm does not require the number of clusters to be known in advance, but rather requires the scale matrix of the density estimation kernel to be specified. Each cluster has different motions, and after clustering, the motion segmentation module 207 is configured to perform motion segmentation on the image, detect and track motion therein, and perform action recognition.
[0066] Embodiments of the present disclosure provide a computationally efficient solution to the hierarchical motion segmentation problem, which is built within a cascaded architecture consisting of multiple processing stages. Each processing stage safely discards a portion of the image from further processing. The initial stage processes more data but has lower complexity, while the later stages tend to have higher complexity but process less data. For accurate motion estimation, a robust estimation method is used and the advantages of a static camera scene are taken advantage of. Typically, for a static camera, most pixels are static (e.g., typically 75%) and are detected earlier by the label frame difference generation module 202 of the motion detection system 102. Only 25% of the total pixels need to be further processed by the remaining components 203-207. Among the 25% of the total pixels, some objects are unoccluded objects and are easy to process. By adopting a video segmentation algorithm that is both stable and fast, the motion detection system 102 helps to provide only local depth order information that is relevant to most human motion modeling applications.
[0067] Figure 6 6 is a flow chart showing a method 600 for motion detection and segmentation of a video sequence captured by a camera according to an embodiment of the present disclosure. Some steps may be Figure 2 The system shown is discussed.
[0068] In step 602, the first image frame of the video is segmented using intra-frame pattern estimation, and one or more other image frames of the video are segmented using inter-frame pattern matching. Intra-frame pattern estimation is mainly used to process new areas introduced in the current frame, that is, new foreground objects or to find previously occluded backgrounds, and inter-frame pattern matching is used to find similar patterns between pixels in the current frame and pixels in the previous frame. In an embodiment of the present disclosure, the video is captured by a monocular still camera. In another embodiment of the present disclosure, one or more image frames of the video are segmented based on a joint time-space-range adaptive mean shift algorithm, which uses a hypersphere with a radius of h in the X, Y, T, R, G, B spatiotemporal multi-channel space. In addition, using inter-frame pattern matching to segment one or more other image frames of the video includes segmenting the current image frame based on a density pattern search in a segmented previous image frame.
[0069] In step 604, a label change map is generated for the current segmented image frame by generating a binary map of the current segmented image frame, and then marking one or more pixels whose labels have changed relative to the previous segmented image frame. Pixels with labels that have changed from the previous segmented image frame may belong to one of the following categories: occluded or unoccluded parts of old objects or parts of new objects entering the scene.
[0070] At step 606, one or more non-static image segments of the current segmented image frame are detected based on the label change map. In an embodiment of the present disclosure, the static image segments have zero motion and do not need further processing, while the non-static segments need to be further classified as static or motion segments. The motion vectors of the static segments are zero because they correspond to static objects that are covered or uncovered.
[0071] In step 608, one or more motion image segments in the one or more non-still image segments are detected. In an embodiment of the present disclosure, the one or more motion segments include one or more unobstructed motion segments and one or more obstructed motion segments.
[0072] In step 610, the motion of one or more motion image segments is estimated. In an embodiment of the present disclosure, a two-dimensional affine model is used to estimate the motion of one or more motion segments. In an embodiment of the present disclosure, the motion of the occluded first square segment can be accurately represented by a general motion model (e.g., a 2D affine model). The model parameters can be estimated by a robust registration method. In another embodiment of the present disclosure, the motion of the unoccluded motion segment is estimated using a robust regression of the affine model.
[0073] At step 612, each motion segment is represented as a vector having seven parameters. In an embodiment of the present disclosure, the seven parameters include two spatial coordinates of the centroid of the corresponding segment, four affine parameters including translation and rotation parameters corresponding to the similarity transformation, and a weight defined by the total number of corresponding pixels.
[0074] In step 614, one or more similar motion segment vectors are clustered based on the corresponding one or more parameters. In an embodiment of the present disclosure, once the segment vector of each segment is determined, one or more clusters of the one or more segment vectors are formed based on the corresponding first to sixth parameters. Clustering is a method of grouping similar entities. For two vectors, the similarity may be a distance function. The similarity measure required in clustering uses the first six parameters of the segment vector except the segment weight. The clustering process helps to group the segments to find parts of a moving object or a joint object, such as a human body (such as a leg, forearm, or head) with similar movements.
[0075] The clustering process helps to group the segments to find moving objects or parts of articulated objects, such as a human body (like a leg, forearm, or head) with similar motions. All parts of a moving object are assumed to have similar (not identical) motions. The similarity defines the granularity of the representation. A person may walk at an average speed of 5 km / h, but the legs and arms can move independently. In activity recognition applications, the motion of each limb must be characterized.
[0076] In an embodiment of the present disclosure, one or more motion recognition results may be sent to a user computing device. In some embodiments, the user computing device may be configured to store or display such recognition results.
[0077] The embodiments of the present disclosure described above may be modified without departing from the scope of the present disclosure as defined by the appended claims. Expressions such as "comprises," "includes," "contains," "consisting of," "having," "is," and the like used to describe and claim the present disclosure are intended to be interpreted in a non-exclusive manner, i.e., allowing for the presence of items, components, or elements that are not explicitly described. References to the singular should also be interpreted as relating to the plural.
Claims
1. A method for segmenting motion data in a video and inferring a layered order, the method comprising: Segmenting a first image frame of the video and one or more other image frames of the video based on a joint time-space-range adaptive mean shift algorithm, wherein the joint time-space-range adaptive mean shift algorithm uses a hypersphere with a radius of h in an X, Y, T, R, G, B spatiotemporal multi-channel space; Generate a label change map for the current segmented image frame by generating a binary map of the current segmented image frame, wherein the label change map is used to indicate labels of pixels in the current segmented image frame; marking one or more pixels whose labels have changed relative to a previously segmented image frame according to the label change map; Detecting and discarding each still image segment from the current segmented image frame to obtain one or more non-still image segments of the current segmented image frame, wherein the non-still image segments include still image segments and moving image segments; detecting one or more moving image segments from among the one or more non-still image segments; estimating motion of the one or more motion picture segments; Representing each motion image segment as a vector having seven parameters, wherein the seven parameters include two spatial coordinates of the centroid of the corresponding segment, four affine parameters including translation and rotation parameters corresponding to a similarity transformation, and a weight defined by the total number of corresponding pixels; and One or more similar motion image segment vectors are clustered based on the two spatial coordinates of the centroid of the corresponding segment and the four affine parameters including translation and rotation parameters corresponding to the similarity transformation.
2. The method of claim 1, wherein the video is captured by a monocular still camera. 3 . The method according to claim 1 , wherein the one or more motion image segments include one or more unobstructed motion image segments and one or more obstructed motion image segments. The method of claim 1 , wherein the motion of the one or more motion image segments is estimated using a two-dimensional affine model.
5. A system for segmenting motion data in a video and inferring a layered order, the system comprising: A memory for storing one or more instructions; and a processor, communicatively coupled to the memory and configured to execute the one or more instructions stored in the memory, the processor comprising: a video segmentation module, configured to segment a first image frame of the video and one or more other image frames of the video based on a joint time-space-range adaptive mean shift algorithm, wherein the joint time-space-range adaptive mean shift algorithm uses a hypersphere with a radius of h in an X, Y, T, R, G, B spatiotemporal multi-channel space; a label frame difference generation module, configured to generate a label change map for the current segmented image frame by generating a binary map of the current segmented image frame, and then mark one or more pixels whose labels have changed relative to the previous segmented image frame according to the label change map, wherein the label change map is used to indicate the labels of the pixels in the current segmented image frame; a non-still segment detection module, configured to detect and discard each still image segment from the current segmented image frame, to obtain one or more non-still image segments of the current segmented image frame, wherein the non-still image segments include static image segments and moving image segments; a motion segment detection module, configured to detect one or more motion image segments in the one or more non-static image segments; a motion estimation module for estimating motion of the one or more motion picture segments; and A motion segmentation module is used to represent each motion image segment as a vector with seven parameters, wherein the seven parameters include two spatial coordinates of the centroid of the corresponding segment, four affine parameters including translation and rotation parameters corresponding to the similarity transformation, and a weight defined by the total number of corresponding pixels; and cluster one or more similar motion image segment vectors based on the two spatial coordinates of the centroid of the corresponding segment and the four affine parameters including translation and rotation parameters corresponding to the similarity transformation.
6. The system of claim 5, wherein the video is captured by a monocular still camera. 7 . The system according to claim 5 , wherein the one or more motion image segments include one or more unobstructed motion image segments and one or more obstructed motion image segments.
8. The system of claim 5, wherein the motion of the one or more motion image segments is estimated using a two-dimensional affine model.
9. A computer programmable product for segmenting motion data in a video and inferring a layering order, the computer programmable product comprising a set of instructions which, when executed by a processor, cause the processor to: The first image frame of the video and one or more other image frames of the video are segmented based on a joint time-space-range adaptive mean shift algorithm, wherein: The joint time-space-range adaptive mean shift algorithm uses a hypersphere with a radius of h in the X, Y, T, R, G, B spatiotemporal multi-channel space; Generate a label change map for the current segmented image frame by generating a binary map of the current segmented image frame, wherein the label change map is used to indicate labels of pixels in the current segmented image frame; marking one or more pixels whose labels have changed relative to a previously segmented image frame according to the label change map; Detecting and discarding each still image segment from the current segmented image frame to obtain one or more non-still image segments of the current segmented image frame, wherein the non-still image segments include still image segments and moving image segments; detecting one or more moving image segments from among the one or more non-still image segments; estimating motion of the one or more motion picture segments; Representing each motion image segment as a vector having seven parameters, wherein the seven parameters include two spatial coordinates of the centroid of the corresponding segment, four affine parameters including translation and rotation parameters corresponding to a similarity transformation, and a weight defined by the total number of corresponding pixels; and One or more similar motion image segment vectors are clustered based on the two spatial coordinates of the centroid of the corresponding segment and the four affine parameters including translation and rotation parameters corresponding to the similarity transformation.
10. The computer programmable product of claim 9, wherein the video is captured by a monocular still camera.
11. The computer programmable product of claim 9, wherein the one or more motion image segments include one or more unobstructed motion image segments and one or more obstructed motion image segments.
12. The computer programmable product of claim 9, wherein the motion of the one or more motion image segments is estimated using a two-dimensional affine model.
13. A monocular still camera surveillance system comprising: A monocular still camera for capturing video; A memory for storing one or more instructions; and a processor coupled to the memory and configured to execute the one or more instructions stored in the memory to: Segmenting a first image frame of the video and one or more other image frames of the video based on a joint time-space-range adaptive mean shift algorithm, wherein the joint time-space-range adaptive mean shift algorithm uses a hypersphere with a radius of h in an X, Y, T, R, G, B spatiotemporal multi-channel space; Generating a label change map for the current segmented image frame by generating a binary map of the current segmented image frame, wherein the label change map is used to indicate labels of pixels in the current segmented image frame; marking one or more pixels whose labels have changed relative to a previously segmented image frame according to the label change map; Detecting and discarding each still image segment from the current segmented image frame to obtain one or more non-still image segments of the current segmented image frame, wherein the non-still image segments include still image segments and moving image segments; detecting one or more moving image segments from among the one or more non-still image segments; estimating motion of the one or more motion picture segments; Representing each motion image segment as a vector having seven parameters, wherein the seven parameters include two spatial coordinates of the centroid of the corresponding segment, four affine parameters including translation and rotation parameters corresponding to a similarity transformation, and a weight defined by the total number of corresponding pixels; clustering one or more similar motion image segment vectors based on the two spatial coordinates of the centroid of the corresponding segment and the four affine parameters including translation and rotation parameters corresponding to the similarity transformation; and Motion detection and activity recognition are performed based on the clustering.
Citation Information
Patent Citations
A method and apparatus for generating depth maps for stereoscopic rendering of planar video.
CN102263979A
Method and system for encoding video frames using a plurality of processors
CN103081466A
