Detection of anomalous behavior

Abstract

To provide a method for detecting anomalous behavior.SOLUTION: Operations include extracting a depiction of a person and associated movement of the person from a first video clip of a first training video included in a first domain dataset. The operations further include superimposing the depiction of the person and corresponding movement into a second video clip of a second training video included in a second domain dataset to generate a third video clip. The operations also include annotating the third video clip to indicate that the movement of the person corresponds to a particular type of behavior, the annotating being based on the first video clip also being annotated to indicate that the movement of the person corresponds to the particular type of behavior. Moreover, the operations include training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip included therewith.SELECTED DRAWING: Figure 1B

Description

[Technical field]

[0001] The embodiments discussed in this disclosure relate to detecting anomalous behavior. [Background technology]

[0002] Machine learning models are becoming more prevalent in a variety of settings and applications. In general, a machine learning model may learn or be trained to make decisions based on a training dataset, which may include real-world examples of an environment associated with the machine learning model. For example, a machine learning model may be trained to detect or identify anomalous behavior of an individual or object, which may be captured via video (e.g., recorded video, live video feeds, etc.).

[0003] The subject matter of the claims in this disclosure is not limited to embodiments that solve any disadvantages or that operate only in such environments. Rather, this background is provided only to illustrate one example technology where some embodiments described in this disclosure may be practiced. Summary of the Invention

[0004] The operations may include extracting from a first video clip of a first training video included in the first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip. Extracting the depiction of the person may include extracting an image mask of the person from the first video clip, and may be responsive to the first video clip being annotated to indicate that the movements of the person depicted in the first video clip correspond to a particular type of behavior. The operations may further include overlaying the depiction of the person and the corresponding movements on a second video clip of a second training video included in the second domain dataset to generate a third video clip. The overlay may be based on the image mask of the person. Additionally, the operations include annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to a particular type of behavior. The annotating may be based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to a particular type of behavior. Additionally, the operations include training the machine learning model to identify the particular type of behavior using a second training video having an annotated third video clip.

[0005] The object and advantages of the embodiments will be realized and attained at least by the elements, features, and combinations particularly pointed out in the appended claims.

[0006] Both the foregoing general description and the following detailed description are presented by way of example and are explanatory only and are not restrictive of the invention, as claimed. [Brief description of the drawings]

[0007] Example embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0008] [Figure 1A] FIG. 1 depicts an exemplary environment for generating a composite video clip.

[0009] [Figure 1B] 1B shows an example of generating composite frames for the composite video clip of FIG. 1A.

[0010] [Diagram 2] 1 shows a block diagram of an exemplary computing system that may be used to generate a composite video clip.

[0011] [Diagram 3] 1 is a flow chart of an exemplary method for generating a composite video clip. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Video anomaly detection (VAD) is a computer vision task that involves identifying events that are not expected to occur, such as fights, shoplifting, etc. VAD has practical applications in a wide range of fields, including security and surveillance.

[0013] In some embodiments, a machine learning (ML) model may be trained to perform VAD. In some examples, such training and testing may be performed using same-domain videos. Additionally or alternatively, such training may be performed using cross-domain videos.

[0014] Same-domain video anomaly detection may primarily focus on analyzing videos within a particular domain or contextual framework. This may require training an ML model on a dataset that includes normal activities and anomalies specific to the same domain. For example, for anomaly detection in surveillance videos corresponding to a particular location, an ML model may be trained utilizing a training dataset of surveillance footage corresponding to the particular location along with corresponding potential anomalous events.

[0015] Conversely, cross-domain video anomaly detection may involve training an ML model to identify anomalies in videos across different domains or contexts. This may involve training an ML model on a source domain that may exhibit potential substantial dissimilarity with the target domain for which anomaly detection is desired. By way of illustration, an ML model may be trained on a training dataset of normal videos originating from one domain, such as surveillance videos at a first location (e.g., an airport, a train station, a store, an intersection, etc.), and then used to detect anomalies in a second location that is different from the first location. Additionally or alternatively, cross-domain anomaly detection may be with respect to the same location or region, but between different camera angles and / or viewpoints of the location or region.

[0016] In this disclosure, an ML model trained using a same-domain video may be referred to as a "same-domain system." Additionally, the term "same-domain VAD" may generally refer to a VAD performed based on a same-domain system and / or same-domain training. Furthermore, in this disclosure, an ML model trained using a cross-domain video may be referred to as a "cross-domain system." Additionally, the term "cross-domain VAD" may generally refer to a VAD performed based on a cross-domain system and / or cross-domain training.

[0017] In real-world settings, cross-domain VAD can be much more useful than same-domain VAD because the target domain may differ from the source domain in terms of context, location, camera angle, anomaly category, etc. However, despite the advantages of cross-domain VAD compared to same-domain VAD, most existing approaches focus on same-domain VAD and rely on weakly-supervised learning (WSL) or unsupervised learning (USL) due to the difficulty of collecting frame-level annotations for anomaly videos in each target domain.

[0018] According to one or more embodiments of the present disclosure, ML models may be trained for cross-domain VAD in a supervised manner using “synthetic” image / video generation. In this disclosure, supervised training may include using training data from a source domain dataset, where the source domain dataset is labeled for frames that include “abnormal” or “normal” behavior. In these and other embodiments, the source domain dataset may be obtained from a single source domain. Furthermore, behaviors depicted in the source domain dataset may be extracted to generate “synthetic” video clips that correspond to one or more target domains that may differ from the source domain. Furthermore, in some embodiments, the synthetic video clips may be used to train ML models to identify and / or classify specific types of behaviors that have been added to the synthetic video clips. The term “synthetic” with respect to a video clip may refer to a video clip that has been modified to include depictions of objects, people, behaviors, etc. that may not have been previously included in or captured by the video clip.

[0019] In this disclosure, general references to people, objects, behaviors, movements, etc. may refer to depictions of people, objects, behaviors, movements, etc. in images, videos, video clips, etc. Thus, references to images, videos, or video clips "including" objects, people, behaviors, movements, etc. may refer to images, videos, or video clips that include depictions of such objects, people, behaviors, etc. Similarly, references to the insertion or extraction of objects, people, behaviors, movements, etc. with respect to images, videos, or video clips may accordingly refer to the insertion or extraction of the respective depictions of such objects, people, behaviors, etc.

[0020] Training an ML model in the manner described in this disclosure may enable improved cross-domain VAD by being able to utilize a relatively small number of labeled source domain datasets (e.g., one source domain dataset) to generate many labeled training videos that may correspond to multiple different domains, such that the benefits of supervised training may be obtained for a much broader range of datasets and domains than those to which the original labeled video clips correspond.

[0021] Embodiments of the present disclosure will now be described with reference to the accompanying drawings.

[0022] FIG. 1A is a diagram depicting an example environment 100 related to generating an annotated composite video clip 108 ("composite clip 108") based on an annotated source domain video clip 104 ("source clip 104") and an unannotated target domain video clip 106 ("target clip 106"), in accordance with one or more embodiments of the present disclosure. In these or other embodiments, the environment 100 may be related to training a machine learning model 110 ("ML model 110") using the composite video clip 108. The description of FIG. 1A is provided with respect to generating the composite clip 108 by inserting abnormal behavior of a person into the target clip 106. However, the principles and concepts described herein may also be applied to generating a composite clip by inserting normal behavior of a person into the target clip 106. Additionally or alternatively, the principles and concepts described herein may be applicable to generating a composite clip to include one or more types of behavior of a person, an object, an animal, etc.

[0023] The source clips 104 may be obtained from a source domain dataset that may include one or more video clips depicting one or more people and corresponding behaviors of the one or more people. In some embodiments, the source domain dataset may correspond to a particular domain that may be associated with a VAD. For example, the source domain dataset may include one or more surveillance videos. In these or other embodiments, the surveillance videos of the source domain dataset may be of a particular location at a particular camera angle. In these and other embodiments, the source clips 104 may be annotated to indicate that one or more movements associated with each of the one or more people captured in the source clips 104 correspond to a particular type of behavior (e.g., the annotations may indicate whether the movements correspond to "normal" behavior or to "abnormal" or "non-normal" behavior). The terms "normal," "abnormal," and "non-normal" are not defined in this disclosure in that many different applications may have many different methodologies and / or criteria used to determine whether a particular behavior is "normal," "abnormal," or "non-normal" when determining how to determine the corresponding annotation. Thus, what is considered in this disclosure is simply what the annotations indicate, not how those annotations came about.

[0024] The target clip 106 may be obtained from a target domain dataset that may include one or more video clips. The target domain dataset may correspond to a different domain than a source domain from which the source clip 104 may be obtained. For example, the target domain may include one or more videos corresponding to (e.g., depicting) a first location, and the source domain may include one or more videos corresponding to a second location different from the first location. The locations may differ in settings, circumstances, etc. Additionally or alternatively, the target domain and the source domain may correspond to the same or similar locations, but with respect to different camera angles, viewpoints, fields of view, etc.

[0025] In these and other embodiments, the target clip 106 may not be annotated with respect to the behavior of one or more people. For example, the target clip 106 may include one or more people and corresponding movements depicted therein, but the target clip 106 may not be annotated to indicate a particular type of behavior corresponding to the movements (e.g., the target clip 106 may not be annotated to indicate whether the movements correspond to abnormal or normal behavior). Additionally or alternatively, in some embodiments, the target clip 106 may not include any people depicted therein or corresponding movements or behaviors.

[0026] The environment 100 may include a video module 102 configured to generate a composite clip 108 based on a source clip 104 and a target clip 106. The video module 102 may include code and routines configured to enable a computing system to perform one or more operations. In some embodiments, the video module 102 may be implemented using one or more processors, central processing units (CPUs), graphics processing units (GPUs), data processing units (DPUs), parallel processing units (PPUs), microprocessors (e.g., for performing one or more operations or controlling the execution of operations), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), accelerators (e.g., deep learning accelerators (DLAs)), and / or other processor types. In some other examples, the video module 102 may be implemented using a combination of hardware and software. In these and other embodiments, video module 102 may be implemented by one or more computing devices, such as those described in more detail with respect to FIG.

[0027] In some embodiments, the video module 102 may be configured to obtain a source clip 104 that may be selected from the source domain dataset. In these and other embodiments, the source clip 104 may be selected from the source domain dataset in response to the source clip 104 being annotated to indicate that movements of a person depicted in the source clip 104 correspond to a particular type of behavior. In the example of FIG. 1, the annotations may indicate that the movements correspond to anomalous behavior, and the source clip 104 may be obtained by the video module 102 based on such annotations.

[0028] The video module 102 may also be configured to acquire a target clip 106. The target clip 106 may be acquired in response to a target domain associated with the target clip 106 being a target domain for the VAD.

[0029] In these and other embodiments, the video module 102 may be configured to extract depictions of people and movements corresponding to anomalous behavior from the source clip 104. In some embodiments, the depictions of people and corresponding movements may be extracted in response to the person's movements being indicated by the annotation as corresponding to anomalous behavior. In these or other embodiments, the extraction of the depictions of people and corresponding movements may be based on a frame-by-frame basis of the source clip 104.

[0030] In these or other embodiments, the extracted depictions of people and corresponding movements may be combined with the target clip 106 to obtain a composite clip 108. In these or other embodiments, the composite clip 106 may be annotated to indicate that the depictions of people's movements extracted from the source clip 104 and included in the composite clip 108 correspond to a particular type of behavior (e.g., anomalous behavior). Such annotations may be based on the source clip 104 that includes the same annotations for the corresponding movements.

[0031] 1B illustrates an example of generating a composite frame 118 of a composite clip 106 based on a source frame 114 of a source clip 104 and a target frame 116 of a target clip 108. In some embodiments, generating the composite clip 108 may include generating a plurality of composite frames as described herein and compiling the composite frames to form the composite clip 108.

[0032] As shown in FIG. 1B, in some embodiments, the video module 102 may obtain a source frame 114 from a source clip 104. The source frame 114 may be an image including a representation 120 of a person. The source frame 114 may also be annotated to indicate that the representation 120 of the person corresponds to anomalous behavior. Based on such annotations, the video module 102 may be configured to perform an image mask extraction operation 124 ("mask extraction 124"). The mask extraction 124 may include extracting an image mask frame 122 ("mask 122") corresponding to the representation 120 of the person. The mask 122 may include an outline 121 of the representation 120 of the person depicted in the source frame 114.

[0033] In some embodiments, mask extraction 124 may include detecting and identifying a region in source frame 114 that corresponds to the depiction of person 120. For example, identifying the region may include identifying pixels in source frame 114 that correspond to the depiction of person 120. Additionally, mask extraction 124 may include identifying a region in source frame 114 that corresponds to the depiction of person 120 (e.g., identifying the location of pixels that include the depiction of person 120). In these and other embodiments, the shape of the identified region (which may indicate the outline 121 of the representation 120 of the person in the source frame 114) may be extracted from the source frame 114 to obtain the mask 122. Additionally or alternatively, other information and representations from the source frame 114 may be omitted from addition to the mask 122, such that the mask 122 may include only the outline 121 of the representation 120 of the person.

[0034] In these and other embodiments, the depiction 120 of the person may be extracted from the source frame 114 as an extracted person frame 128 based on the mask 122 and the source frame 114. For example, in some embodiments, the extracted person frame 128 may be obtained from the source frame 114 by performing a multiplication 126 (e.g., a pixel-wise multiplication) between the mask 122 and the source frame 114.

[0035] Additionally or alternatively, a target frame 116 of the target clip 106 may be obtained. The target frame 116 may be a background image that may include a background 130 into which it may be desirable to insert a depiction 120 of a person to obtain a composite frame 118.

[0036] In some embodiments, the composite frame 118 may be generated using one or more frame compositing operations 134 ("frame composition 134"). The frame compositing operations 134 may include overlaying the person and corresponding motion onto the target frame 116 to generate the composite frame 118.

[0037] The frame composition operation 143 may be based on the target frame 116, the mask 122, and the extracted person frame 128. For example, in some embodiments, a mask inverse frame 131 of the mask 122 ("mask inverse 131") may be obtained. The mask inverse 131 may be a color inverse of the mask 122 and may include an inverted outline 132 of the outline 121 of the mask 122.

[0038] In some cases, the position of the person representation 120 in the source frame 114 may correspond to a target position of the person representation 120 in the composite frame 118. Thus, in these and other cases, the position of the inverted outline 132 in the mask inversion 131 may be the same as the position of the outline 121 in the mask 122. In other cases, the position of the person representation 120 in the source frame 114 may not correspond to a target position of the person representation 120 in the composite frame 118. Thus, in these and other cases, the position of the inverted outline in the mask inversion 131 may move according to the target position such that the position of the inverted outline 132 in the mask inversion 131 corresponds to the target position of the person representation 120 in the composite frame 118.

[0039] In these and other embodiments, the frame composition 134 may include a multiplication 136 (e.g., pixel-by-pixel multiplication) between the target frame 116 and the mask inversion 131. The multiplication 136 may insert an outline of the person representation 120 into the target frame 116 to obtain an intermediate frame (not explicitly shown in FIG. 1B ). Additionally or alternatively, the frame composition 134 may include an addition 138 (e.g., pixel-by-pixel addition) of the intermediate frame and the extracted person frame 128 to obtain the composite frame 118. The addition 138 may insert pixels from the extracted person frame 128 corresponding to the person representation 120 into the outline of the person included in the intermediate frame such that the person representation 120 is included in the composite frame 118. In this way, the generation of the composite frame 118 may allow the insertion of the person representation 120 while also allowing other background elements of the source frame 114 to be omitted so that the person representation 120 is more naturally included in the composite frame 118.

[0040] In some embodiments, frame composition 134 may include one or more other operations related to generating composite frame 118. For example, in some embodiments, a location of a representation 120 of a person within composite frame 118 may be determined. In some embodiments, the determined location may be based on an area depicted in target frame 116 where the person may be located.

[0041] For example, the target frame 116 may include a sidewalk, storefront, alley, passageway, walkway, etc., where a person might typically be located. Thus, placing the person representation 120 in such an area may aid the composite frame 118 and corresponding composite clip 108 in depicting a possible or realistic scenario. Conversely, the target frame 116 may depict an area where a person is not, or is unlikely to be, located, such as in thick vegetation, on the side of a building, on top of a pole, on top of a tree, etc. Thus, placing the person representation 120 in one of such areas in the target frame 116 may result in an unlikely or impossible scenario depiction in the composite frame 118 and corresponding composite clip 108.

[0042] In some embodiments, a location area may be identified within the target frame 116. A location area may include any area within the target frame 116 that depicts a physical area in which a person may be located. As indicated above, a location area may include depictions of aisles, storefronts, pathways, footpaths, sidewalks, etc. A location area may be identified using any suitable technique.

[0043] For example, in some embodiments, semantic segmentation may be performed on the target frame 116 and / or the target clip 106. Semantic segmentation may identify and label different types of regions depicted in the target frame 116 and / or the target clip 106, including regions where a person may be located. For example, semantic segmentation may be used to identify and / or label aisles, storefronts, pathways, footpaths, walkways, etc. depicted in the target frame 116. In these or other embodiments, semantic segmentation may be used to identify and / or label the ground plane depicted in the target frame 116. Additionally or alternatively, semantic segmentation may be used to identify locations on trees, vegetation, walls, buildings, pillars, sky, ground surfaces depicted in the target frame 116 that may correspond to locations where a person is unlikely to be located.

[0044] In these and other embodiments, the location region may include portions of the target frame 116 that are indicated by semantic segmentation as corresponding to locations where a person may be located. For example, portions of the target frame 116 that are identified as depicting an aisle, a storefront, an alley, a footprint, a sidewalk, etc. may be identified as a location region.

[0045] In some embodiments, the placement of the representation 120 of the person in the target frame 116 for generating the composite frame 118 may be constrained by a placement region. For example, the constraint may indicate that the representation 120 of the person may only be placed within an identified placement region. In these or other embodiments, the target location of the representation 120 of the person in the composite frame 118 may accordingly be within and constrained to the placement region.

[0046] 1B for additional frames of the source clip 104 to obtain additional composite frames. The movement (and corresponding behavior accordingly) of the person representation 120 in the source clip 104 may be propagated to the composite clip 108 based on the position of the person representation 120 changing in different source frames of the source clip 104 due to the movement of the person representation 120, where the changed position may be propagated in the position of the outline of the person representation 120 in subsequently generated mask frames, mask-inverted frames, and extracted person frames.

[0047] In these or other embodiments, the movement of the person's depiction 120 in the composite clip 108 may be the same as the movement of the person's depiction 120 in the source clip 104. Furthermore, in instances where the movement of the person's depiction 120 is the same, the initial position of the person's depiction 120 in the composite clip 108 may be the same as the initial position of the person's depiction 120 in the source clip 104.

[0048] Additionally or alternatively, the initial position of the representation 120 of the person in the compositing clip 108 may be different from the initial position of the representation 120 of the person in the source clip 104 while the motion is maintained (e.g., to ensure that the person 120 is located within the placement area in the compositing clip 108). In these or other embodiments, the transformation of the position of the representation 120 in generating each corresponding composite frame 118 may be the same.

[0049] In some embodiments, the movement of the person's representation 120 may be modified for the composite clip 108 compared to the source clip 104. Such an approach may allow multiple different composite clips 108, each depicting a different inserted movement, to be generated based on the same source clip 104 and the same movement depicted in the source clip 104. In some embodiments, the movement modification may include modifying the speed of the person's representation 120, the direction of the person's representation 120, and / or the altitude of the person's representation 120.

[0050] In these and other embodiments, motion correction may be achieved by correcting one or more changes between the position of the person's representation 120 between successive composite frames 118 compared to the change between the position of the person's representation 120 between successive source frames 114. For example, the position of the person's representation 120 changes by (Δx1, Δy1) between two successive source frames 114, and the motion may be corrected by changing the position of the person's representation 120 by (Δx2, Δy2) between two corresponding composite frames 118.

[0051] In some embodiments, a synthetic video clip 108 having newly inserted movements that have been annotated as corresponding to a particular type of behavior (e.g., abnormal behavior or normal behavior) may be provided to an ML model 110. The ML model 110 may include any suitable algorithm, computer model, neural network, deep learning model, or other system and / or model that may be trained to perform one or more computer vision tasks, such as a VAD. Although referred to as a "model," the ML model 110 may include any number of models that may operate individually or together.

[0052] In these or other embodiments, the synthetic clips 108 and corresponding annotations may be used as training data for the ML model 110. For example, the synthetic video clips 108 may include depictions of additional movements annotated as corresponding to anomalous behaviors, and the ML model 110 may be trained to better identify anomalous behaviors based on such input. In some embodiments, the ML model 110 may be trained based on many synthetic clips 108 generated in the manner described herein. As previously described, the synthetic clips 108 may correspond to a domain different from the source domain from which the source clips 104 may be obtained. Furthermore, as shown, multiple different synthetic clips 108 depicting multiple different movements may be generated based on the same movement contained in the same source clip 104. Thus, embodiments of the present disclosure may help improve the ability to train the ML model 110 by allowing for the provision of a large amount of training data for multiple different domains based on a limited amount of already annotated source data from a limited number of source domain datasets.

[0053] Modifications, additions, or omissions may be made to FIGS. 1A and 1B without departing from the scope of the present disclosure. For example, environment 100 may include more or fewer elements than those shown and described in this disclosure. For example, in some embodiments, video module 102 and the operations described in connection with video module 102 may be performed by a single device or may be distributed across different systems. Additionally or alternatively, as indicated above, the described operations may be applied to generate a composite clip 108 in which "normal" behaviors are added and annotated accordingly, as opposed to "abnormal" behaviors. Furthermore, multiple different target clips 106 may be used to generate corresponding composite clips 108 in which the target clips 106 may correspond to two or more different domains. Furthermore, any number of different source clips 104 may be used, which may correspond to any number of different source domains and source domain data sets.

[0054] 2 illustrates a block diagram of an example computing system 202 that may be used to generate a composite video clip in accordance with at least one embodiment of the present disclosure. In some embodiments, the computing system 202 may be configured to implement or direct one or more operations associated with the video module 102 of FIG. 1A. Additionally or alternatively, the computing system 202 may be used to implement the ML model 110 of FIG. 1A. The computing system 202 may include a processor 250, a memory 252, and a data storage 254. The processor 250, the memory 252, and the data storage 254 may be communicatively coupled.

[0055] In general, the processor 250 may include any suitable special purpose or general purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored on any applicable computer-readable storage medium. For example, the processor 250 may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Although shown as a single processor in FIG. 2, the processor 250 may include any number of processors that are individually or collectively configured to perform or direct the execution of any number of operations described in this disclosure. Additionally, one or more processors may reside in one or more different electronic devices, e.g., different servers.

[0056] In some embodiments, processor 250 may be configured to interpret and / or execute program instructions and / or process data stored in memory 252, data storage 254, or memory 252 and data storage 254. In some embodiments, processor 250 may fetch program instructions from data storage 254 and load the program instructions into memory 252. After the program instructions are loaded into memory 252, processor 250 may execute the program instructions.

[0057] For example, in some embodiments, a video module (e.g., the video module of FIG. 1) may be included as program instructions in data storage 254. Processor 250 may fetch the program instructions for the video module from data storage 254 and load the program instructions for the video module into memory 252. After the program instructions for the video module are loaded into memory 252, processor 250 may execute the program instructions such that computing system 202 may implement (e.g., perform and / or cause the performance of) operations associated with the video module as directed by the instructions.

[0058] Memory 252 and data storage 254 may include computer-readable storage media for carrying or storing computer-executable instructions or data structures. Such computer-readable storage media may include any available media that can be accessed by a general-purpose or special-purpose computer, such as processor 250. By way of example, and not limitation, such computer-readable storage media may include tangible or non-transitory computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory devices (e.g., solid-state memory devices), or any other storage medium that can be used to store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 250 to perform a certain operation or group of operations.

[0059] Modifications, additions, or omissions may be made to computing system 202 without departing from the scope of the disclosure. For example, in some embodiments, computing system 202 may include any number of other components that may not be explicitly shown or described.

[0060] 3 is a flowchart of an example method 300 of generating a composite video clip according to at least one embodiment described in the present disclosure. Method 300 may be performed by any suitable system, apparatus, or device. By way of example, video module 102 of FIG. 1A or computing system 202 of FIG. 2 (e.g., as directed by video module 102 in some embodiments) may perform one or more of the operations associated with method 300. Although shown with discrete blocks, steps and operations associated with one or more of the blocks of method 300 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0061] At block 302, depictions of people and corresponding movements may be extracted from the annotated source domain video clip. In these and other embodiments, extraction may be performed frame-by-frame of the source domain video clip. In some embodiments, such extraction may include one or more of the acts described with respect to FIGS. 1A and 1B.

[0062] A target domain video clip may be selected for inserting depictions of people and corresponding movements depicted in the source domain video clip at block 304. In some embodiments, such selection may include one or more of the acts described with respect to FIGS.

[0063] In block 306, in some embodiments, a placement region within the target domain video clip may be identified. The placement region may be an area identified as suitable for placement of the extracted depiction of the person and corresponding movement. In some embodiments, such identification may include one or more of the acts described with respect to FIGS. 1A and 1B.

[0064] At block 308, in some embodiments, a movement may be selected. In some embodiments, the movement selection may include selecting a movement depicted in the source domain video clip. Additionally or alternatively, the movement selection may include selecting to modify one or more of a speed, direction, or altitude of the depiction of the person as compared to the movement of the person depicted in the source domain video clip.

[0065] In these or other embodiments, selecting the movement may include selecting a placement location for the depiction of the person within each frame of the target domain video clip that corresponds to the movement. Such placement locations may include an initial location, one or more intermediate locations, and a final location. These locations may change from frame to frame according to the selected movement such that the movement is depicted by the composite video clip generated according to method 300. In some embodiments, selecting one or more locations (e.g., an initial location) may include one or more of the acts described with respect to FIGS. 1A and 1B.

[0066] At block 310, it may be determined whether the depiction of the selected movement is within the identified placement region of the target domain video clip. For example, it may be determined whether any of the placement locations of the depiction of the person are located outside the placement region. In response to at least one of the placement locations being outside the placement region, the method 300 may proceed from block 310 to block 312.

[0067] At block 312, one or more placement locations may be modified. For example, in some embodiments, the initial placement location may be modified such that all other subsequent locations may also be modified to modify the location of the overall movement. Additionally or alternatively, the final placement location may be modified such that the final location and all other preceding locations may also be modified to modify the location of the overall movement. In these or other embodiments, one or more locations outside the placement region may be individually modified such that the movement itself is also modified. Following block 312, the method 300 may return to block 310.

[0068] Returning to block 310, in response to all placement locations being outside the placement area, the method 300 may proceed from block 310 to block 314. At block 314, a composite video clip may be generated. In some embodiments, generating the composite video clip may include one or more of the operations described with respect to Figures 1A and 1B.

[0069] Those skilled in the art will appreciate that the functions and / or operations performed for this and other processes, operations, and methods disclosed herein may be implemented in a different order. Additionally, the outlined functions and operations are provided only as examples, and some of the functions and operations may optionally be combined into fewer functions and operations or expanded into additional functions and operations without detracting from the essence of the disclosed embodiments. In some embodiments, method 300 may include additional or fewer blocks.

[0070] For example, in some embodiments, the determination of whether a person is included in the placement area of ​​the entire composite video clip may be performed after generation of the composite video clip, and the composite video clip may be modified accordingly in response to movement falling outside the placement area at any time during the composite video clip. Additionally or alternatively, the determination of whether movement is within the placement area may be performed on a frame-by-frame basis during generation of the composite video clip frames. In these and other embodiments, applicable modifications may be made with respect to that particular composite frame prior to compiling all frames into the composite video clip.

[0071] Further, method 300 may include one or more other operations, such as training a machine learning model using the synthetic video clips. For example, the annotations of the synthetic video clips may be used as ground truth data in training the machine learning model. Additionally, method 300 may be performed to generate any number of synthetic video clips from any number of source domain video clips and / or target domain video clips.

[0072] As indicated above, the embodiments described in this disclosure may involve the use of a special purpose or general purpose computer (e.g., processor 250 of FIG. 2) that includes various computer hardware or software modules, as discussed in more detail below. Additionally, as indicated above, the embodiments described in this disclosure may be implemented using a computer-readable medium (e.g., memory 252 or data storage 254 of FIG. 2) that carries or has computer-executable instructions or data structures stored thereon.

[0073] As used in this disclosure, the terms "module" and "component" may refer to a specific hardware implementation configured to perform the actions of the module or component and / or a software object or software routine that may be stored and / or executed on general-purpose hardware of a computing system (e.g., computer-readable medium, processing device, etc.). In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). Although some of the systems and methods described in this disclosure are generally described as being implemented in software (stored on and / or executed by general-purpose hardware), specific hardware implementations or combinations of software and specific hardware implementations are also possible and contemplated. In this description, a "computing entity" may be any computing system defined above in this disclosure, or any module or combination of modules running on a computing system.

[0074] The terms used in this disclosure, particularly in the appended claims (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including, but not limited to," etc.).

[0075] Additionally, where a specific number of introduced claim provisions are intended, such intent is expressly set forth in the claim, and where no such provision is present, no such intent exists. For example, as an aid to understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim provisions. However, the use of such phrases should not be construed to suggest that the introduction of a claim provision with the indefinite article "a" or "an" limits any particular claim containing such introduced claim provision to an embodiment containing only one such provision, even when the same claim contains the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be construed to mean "at least one" or "one or more"), and the same is true in the case of the use of an indefinite article used to introduce a claim provision.

[0076] Additionally, even if a particular number of provisions in an introduced claim is explicitly recited, a person skilled in the art would recognize that such provisions should be interpreted to mean at least the recited number (e.g., the mere provision of "two provisions", without other modifiers, means at least two provisions, or more than two provisions). Furthermore, when a convention similar to "at least one of A, B, and C" or "one or more of A, B, and C, etc." is used, it is generally intended that such a structure includes A alone, B alone, C alone, A and B, A and C, B and C, or A, B, and C, etc.

[0077] Moreover, any disjunctive word or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to contemplate the possibility of including one of the terms, either of the terms, or both of the terms. For example, the phrase "A or B" should be understood to include the possibilities of "A," "B," or "A and B." This interpretation of the phrase "A" or "B" remains applicable even though the term "A and / or B" may sometimes be used to include the possibilities of "A" or "B," or "A" and "B."

[0078] All examples and conditional language set forth in this disclosure are intended for educational purposes to aid the reader in understanding the disclosure and the concepts the inventors have contributed to furthering the art, and should not be construed as being limited to the examples and conditions so specifically set forth. Although embodiments of the present disclosure have been described in detail, various modifications, substitutions, and alterations may be made thereto without departing from the spirit and scope of the present disclosure.

[0079] The present disclosure also discloses the following notes:

[0080] (Appendix 1) 1. A method comprising: extracting from a first video clip of a first training video included in a first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip, wherein extracting the depiction of the person includes extracting an image mask of the person from the first video clip and is responsive to the first video clip being annotated to indicate that the movements of the person depicted in the first video clip correspond to a particular type of behavior; superimposing a depiction and corresponding motion of the person onto a second video clip of a second training video included in a second domain dataset based on the image mask of the person to generate a third video clip; annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior, based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior; and training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip. (Appendix 2) 2. The method of claim 1, wherein the annotation indicates that the particular type of behavior is anomalous behavior. (Appendix 3) 2. The method of claim 1, wherein the movements of the person depicted in the third video clip are modified compared to the movements of the person depicted in the first video clip. (Appendix 4) 4. The method of claim 3, wherein the movement is modified with respect to one or more of a direction of the movement, a speed of the movement, or an altitude of the movement. (Appendix 5) modifying the motion of the person depicted in the first video clip to generate a modified motion depiction; generating a fourth video clip by overlaying the representation of the person and the modified motion representation onto at least the second video clip; annotating the fourth video clip to indicate that the modified motion depiction contained in the fourth video clip corresponds to the particular type of behavior; and 2. The method of claim 1, further comprising: training the machine learning model to identify the particular type of behavior using the second training video having the annotated fourth video clip. (Appendix 6) 2. The method of claim 1, wherein the depiction of the person and corresponding movement is superimposed on a depiction of a particular area in the second video clip in response to the particular area being identified as a placement area. (Appendix 7) 7. The method of claim 6, wherein the particular region is identified as the placement region based on a semantic segmentation corresponding to the second video clip. (Appendix 8) 1. A system comprising: one or more processors; and one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed by the one or more processors, cause the system to perform operations, including: extracting from a first video clip of a first training video included in a first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip, wherein extracting the depiction of the person includes extracting an image mask of the person from the first video clip and is responsive to the first video clip being annotated to indicate that the movements of the person depicted in the first video clip correspond to a particular type of behavior; superimposing a depiction and corresponding motion of the person onto a second video clip of a second training video included in a second domain dataset based on the image mask of the person to generate a third video clip; annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior, based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior; and training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip. (Appendix 9) 9. The system of claim 8, wherein the annotation indicates that the particular type of behavior is anomalous behavior. (Appendix 10) 9. The system of claim 8, wherein movements of the person depicted in the third video clip are modified compared to movements of the person depicted in the first video clip. (Appendix 11) 11. The system of claim 10, wherein the movement is modified with respect to one or more of a direction of the movement, a speed of the movement, or an altitude of the movement. (Appendix 12) The operation includes: modifying the motion of the person depicted in the first video clip to generate a modified motion depiction; generating a fourth video clip by overlaying the representation of the person and the modified motion representation onto at least the second video clip; annotating the fourth video clip to indicate that the modified motion depiction contained in the fourth video clip corresponds to the particular type of behavior; and 9. The system of claim 8, further comprising: training the machine learning model to identify the particular type of behavior using the second training video having the annotated fourth video clip. (Appendix 13) 9. The system of claim 8, wherein the depiction of the person and corresponding movement is superimposed on a depiction of a particular area in the second video clip in response to the particular area being identified as a placement area. (Appendix 14) 14. The system of claim 13, wherein the particular region is identified as the placement region based on a semantic segmentation corresponding to the second video clip. (Appendix 15) One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform an operation, the operation including: extracting from a first video clip of a first training video included in a first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip, wherein extracting the depiction of the person comprises extracting an image mask of the person from the first video clip and responsive to the first video clip being annotated to indicate that the movements of the person depicted in the first video clip correspond to a particular type of behavior; superimposing a depiction and corresponding motion of the person onto a second video clip of a second training video included in a second domain dataset based on the image mask of the person to generate a third video clip; annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior, based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior; and training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip. (Appendix 16) 16. The one or more non-transitory computer-readable storage media of claim 15, wherein the annotation indicates that the particular type of behavior is anomalous behavior. (Appendix 17) 16. The one or more non-transitory computer-readable storage media of claim 15, wherein movements of the person depicted in the third video clip are modified compared to movements of the person depicted in the first video clip. (Appendix 18) The operation includes: modifying the motion of the person depicted in the first video clip to generate a modified motion depiction; generating a fourth video clip by overlaying the representation of the person and the modified motion representation onto at least the second video clip; annotating the fourth video clip to indicate that the modified motion depiction contained in the fourth video clip corresponds to the particular type of behavior; and and training the machine learning model to identify the particular type of behavior using the second training video having the annotated fourth video clip. (Appendix 19) 16. The one or more non-transitory computer-readable storage media of claim 15, wherein the depiction of the person and corresponding movement is superimposed on a depiction of a particular area in the second video clip in response to the particular area being identified as a placement area. (Appendix 20) 20. The one or more non-transitory computer-readable storage media of claim 19, wherein the particular region is identified as the placement region based on a semantic segmentation corresponding to the second video clip.

Claims

1. 1. A method comprising: extracting from a first video clip of a first training video included in a first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip, wherein extracting the depiction of the person includes extracting an image mask of the person from the first video clip and is responsive to the first video clip being annotated to indicate that movements of the person depicted in the first video clip correspond to a particular type of behavior; superimposing a depiction and corresponding motion of the person onto a second video clip of a second training video included in a second domain dataset based on the image mask of the person to generate a third video clip; annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior, based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior; and training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip.

2. The method of claim 1 , wherein the annotation indicates that the particular type of behavior is anomalous behavior.

3. The method of claim 1 , wherein movements of the person depicted in the third video clip are modified relative to movements of the person depicted in the first video clip.

4. The method of claim 3 , wherein the movement is modified with respect to one or more of the direction of the movement, the speed of the movement, or the altitude of the movement.

5. modifying the motion of the person depicted in the first video clip to generate a modified motion depiction; generating a fourth video clip by overlaying the representation of the person and the modified motion representation onto at least the second video clip; annotating the fourth video clip to indicate that the modified motion depiction contained in the fourth video clip corresponds to the particular type of behavior; 2. The method of claim 1 , further comprising: training the machine learning model to identify the particular type of behavior using the second training video having the annotated fourth video clip.

6. The method of claim 1 , wherein the representation of the person and corresponding movements are superimposed on a representation of a particular area in the second video clip in response to the particular area being identified as a placement area.

7. The method of claim 6 , wherein the particular region is identified as the placement region based on a semantic segmentation corresponding to the second video clip.

8. 1. A system comprising: one or more processors; and one or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed by the one or more processors, cause the system to perform operations, including: extracting from a first video clip of a first training video included in a first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip, wherein extracting the depiction of the person includes extracting an image mask of the person from the first video clip and is responsive to the first video clip being annotated to indicate that movements of the person depicted in the first video clip correspond to a particular type of behavior; superimposing a depiction and corresponding motion of the person onto a second video clip of a second training video included in a second domain dataset based on the image mask of the person to generate a third video clip; annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior, based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior; and training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip.

9. One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause the system to perform an operation, the operation including: extracting from a first video clip of a first training video included in a first domain dataset a depiction of a person and associated movements of the person depicted in the first video clip, wherein extracting the depiction of the person comprises extracting an image mask of the person from the first video clip and responsive to the first video clip being annotated to indicate that movements of the person depicted in the first video clip correspond to a particular type of behavior; superimposing a depiction and corresponding motion of the person onto a second video clip of a second training video included in a second domain dataset based on the image mask of the person to generate a third video clip; annotating the third video clip to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior, based on the first video clip from which the depiction of the person and associated movements were extracted also being annotated to indicate that the movements of the person depicted in the third video clip correspond to the particular type of behavior; and training a machine learning model to identify the particular type of behavior using the second training video having the annotated third video clip.

Images