COMPUTER IMPLEMENTED METHOD, COMPUTER PROGRAM AND APPARATUS FOR VIDEO PROCESSING AND GENERATION OF THUMBNAILS FROM VIDEO SEQUENCES, AND VIDEO SURVEILLANCE SYSTEM COMPRISING SUCH APPARATUS - Patent application

Abstract

A computer-implemented method, a computer program product, and an apparatus for video processing and generating thumbnails from video sequences, as well as a video surveillance system including such an apparatus, are provided. The computer-implemented method for video processing includes acquiring a first video sequence of an area of ​​interest that includes a first predetermined object or activity of interest, and acquiring a second video sequence of an area of ​​interest that includes a second predetermined object or activity of interest. The method further includes determining whether a recording period of the first video sequence and a recording period of the second video sequence overlap for a period of time, and, if the recording periods of the first and second video sequences overlap for a period of time, defining at least one first video clip using frames of the first and / or second video sequences from at least the period of overlap. Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a computer-implemented method, a computer program and an apparatus for video processing, as well as a video surveillance system comprising such an apparatus. The present disclosure also relates to a computer-implemented method, a computer program and an apparatus for generating thumbnails from video sequences. In particular, the present disclosure relates to the field of video surveillance, where video sequences showing different objects or activities of interest need to be processed and thumbnails thereof need to be generated. [Background technology]

[0002] A video surveillance system typically includes a number of video cameras that record video data upon detection of an object and / or activity within the field of view of one or more of the video cameras. These video cameras can be triggered upon detection of an object and / or activity by one or more sensors that operate at all times and cover an area larger than the area of ​​the video cameras. Thus, recording by the video cameras can advantageously be activated when an object and / or activity appears in the coverage area of ​​the video cameras and stopped for the rest of the time. Such an operating method makes it possible to limit the recording of video sequences to potentially important events and to limit the storage capacity required to operate the video surveillance system.

[0003] The recorded video sequences are typically stored in one or more recording servers and may be accessed by an operator or user wishing to retrieve video sequences of an object or activity of interest. The operator typically relies on the metadata generated by the video analysis software and queries the metadata using one or more search plug-ins to find relevant video sequences. The one or more search plug-ins may query metadata in different databases, which may advantageously be external and / or independent of each other (or have no natural way of sharing information). Such databases may advantageously be created by different parties (e.g., partners or third parties). These different databases may advantageously contain metadata related to the same video sequence, for example, one database may contain metadata related to the color of the vehicle and another database may contain metadata related to the make and / or model of the vehicle detected by one or more video cameras. One database may also contain metadata related to the weather in the video sequence to which it is associated.

[0004] The metadata may also be accessed via on-demand queries via a web resource, or in any other suitable manner. The metadata may also be advantageously provided in the form of a metadata stream within a virtual (or virtualization) management software (VMS).

[0005] Thus, it is typically possible to attach (or combine or associate) metadata from different sources to video sequences originating from a single video camera or originating from different video cameras, in other words, in some cases, it is not necessary to have a one-to-one relationship between a piece of metadata and a video camera.

[0006] Many video analytics software modules are available that can analyze video surveillance data and detect specific objects or activities. These will typically attach metadata to the video stream indicating the time and location within the frame that the object or activity was detected, along with data describing the nature of the detected object or activity. Examples of types of objects that can be detected are people, cars, license plates, etc. Some object recognition algorithms are sophisticated enough to classify people by age and gender, and the metadata can indicate characteristics of the detected object, such as color, type of vehicle, or the person's age or the person's gender. Some video analytics modules can also detect activities such as patterns of behavior, such as loitering, which can also be indicated by metadata included in the video surveillance data and associated with the video data. When metadata is included in the video surveillance data, it can be used to perform searches for specific objects, activities, or events. For example, searches can be performed for black SUVs, or people of a particular age, gender, and hair color, or vehicles with a particular license plate.

[0007] The video analysis software that generates the metadata can run in a variety of locations: the analysis software can run on the video camera, on an analysis server, on a recording server, or on a client device, In a cloud-based system, the video analysis may be performed in the cloud.

[0008] When an operator or user wishes to retrieve video sequences of a region of interest that match two or more search criteria during a particular time period, several problems arise.

[0009] First, as a result of a user's search, the user may be presented with video sequences from each search plugin used in the search, i.e., each search plugin will provide search results for whatever criteria are searched for, even if these search results overlap with search results from another search plugin. For example, a first search plugin may provide video sequences containing men with a particular hair color, a second search plugin may provide video sequences containing men of a particular age group, and a third search plugin may provide video sequences containing cars of a particular color. If a user specifies that they are looking for these criteria, it is not necessarily useful for the user to be presented with three video sequences showing the same scene at the same time. This problem is exacerbated when one or more search plugins provide multiple search results. Furthermore, the user cannot combine, view, and process all of these video sequences provided by the associated metadata to create a coherent understanding of how these video sequences match the search criteria. In summary, the user is presented with an excessive number of search results with time-consuming analysis of all the video sequences.

[0010] Second, the user cannot specify how well the search results should match the search criteria. For example, the user cannot narrow down the number of search results by adding more search plugins (e.g., criteria), since each plugin would provide additional search results. Thus, in an environment where many video cameras generate many recorded video sequences (e.g., in a shopping mall), there is a trade-off for the user between adding more search criteria to generate relevant search results and limiting the number of search results to a reasonable number by limiting the number of search criteria. Furthermore, the user cannot choose to present search results that match only some of the search criteria, for example, when the user is unsure about one criterion (e.g., whether the car he is looking for is blue or black). In summary, the content of the video sequences presented as search results may not adequately match the user's search request. Thus, there is a general need to better address such user's search requests by improving the relevance of the search results.

[0011] Additionally, there is a need to improve how search results are presented (or displayed) to a user by defining more relevant thumbnails that show two objects or activities of interest that were searched for by the user.

[0012] More specifically, if the video sequence to be presented as a result of a search performed by a user is a sequence (or clip) created from two video sequences recorded separately upon detection of a particular object and / or activity, a thumbnail for this clip needs to be defined.

[0013] The technical solutions presented in this disclosure and the appended claims aim to address at least one of the above problems. Summary of the Invention

[0014] The present invention provides a computer-implemented method of video processing comprising: acquiring a first video sequence of an area of ​​interest including a first predefined object or activity of interest; acquiring a second video sequence of an area of ​​interest including a second predefined object or activity of interest; determining whether a recording period of the first video sequence and a recording period of the second video sequence overlap for a period of time; and, if the recording periods of the first and second video sequences overlap for a period of time, defining at least one first video clip using frames of the first and / or second video sequences from at least the period of overlap.

[0015] The invention also provides a method of generating a thumbnail from a video sequence, comprising identifying a first frame of the video sequence containing a first object or activity of interest, identifying a second frame of the video sequence containing a second object or activity of interest, and generating a thumbnail using at least a portion of the first frame and at least a portion of the second frame of the video sequence, the thumbnail containing first and second predetermined objects or activities, the video sequence being advantageously the first video clip as defined above, and the first and second objects or activities being respectively the first and second predetermined object or activity of interest as defined above.

[0016] Preferred features of the invention are set out in claims 1 to 19.

[0017] The invention also provides a computer program for image processing as claimed in claim 20.

[0018] The present invention also provides an image processing device as set forth in claim 21 or 22.

[0019] The present invention also provides a video surveillance system as claimed in any one of claims 23 to 25. [Brief description of the drawings]

[0020] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0021] [Figure 1] 1 illustrates generally a video surveillance system in which the present invention may be implemented; [Diagram 2] Diagram showing the graphical user interfaces of two different search plugins [Diagram 3] 3 is a flow chart of a computer-implemented method of video processing in accordance with the present invention. [Figure 4] , [Diagram 5] , [Figure 6] , [Figure 7] , [Figure 8] , [Figure 9] , [Figure 10] , [Figure 11] , [Figure 12] , [Figure 13] , [Figure 14] , [Figure 15] , and [Figure 16] 3A-3C are schematic diagrams illustrating different search results that may be returned by the computer-implemented method of video processing according to the present invention; [Figure 17] 3 is a flow chart of a computer-implemented method for generating thumbnails from a video sequence in accordance with the present invention. [Figure 18] , [Figure 19] , [Figure 20] , [Figure 21] , [Figure 22(a)] , and [Figure 22(b)]4A-4D illustrate different thumbnails that may be returned by a computer-implemented method for generating thumbnails from a video sequence in accordance with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] FIG. 1 shows an example of a video surveillance system 100 in which an embodiment of the present invention can be implemented. The system 100 comprises a client device 120, a management server 130, a video processing device 140, a thumbnail generating device 150, an indexing server 160, and a recording server 170 connected to a number of video cameras 110a, 110b, 110c. It should be noted that the present invention is not limited to this configuration. One or more of the aforementioned servers or devices may be combined into a single device or server. For example, the functions of the client device 120, the video processing device 140, and the thumbnail generating device 150 may be performed by a single device, e.g., the client device 120. Similarly, the functions of the management server 130, the indexing server 160, and the recording server 170 may be performed by a single server, e.g., the recording server 170. The present invention is not limited to this configuration. Other servers or devices may be present in the system 100. For example, an archive server (not shown) may be provided to archive old data stored in the record server 170 that does not need to be immediately accessible from the record server 170, but that the user does not want to delete permanently. Note that a fail-over record server (not shown) may be provided in case the main record server fails. The analysis server may also run analysis software for image analysis such as motion or object detection, face recognition, event detection, etc. The functionality of some of these devices or servers may also be distributed across different devices or servers. For example, the functionality of the thumbnail generator 150 may be distributed across the client devices 120 and the record server 170.

[0023] A number of video surveillance cameras 110a, 110b, 110c transmit video data to a recording server 170. The client device 120 is, for example, a fixed terminal that provides an interface through which an operator can view live video data from the video cameras 110a, 110b, 110c or recorded video data from the recording server 170.

[0024] Video cameras 110a, 110b, 110c capture image data and transmit it to recording server 170 as multiple video data streams.

[0025] Recording server 170 stores the video data streams captured by video cameras 110a, 110b, 110c. The video data is streamed from recording server 170 to client device 120 depending on which live or recorded stream is selected by the operator for viewing.

[0026] The client device 120 is configured to communicate with the management server 130, the video processing device 140, the thumbnail generator 150, the indexing server 160, and the recording server 170 via a first network / bus 121. The recording server 170 communicates with the cameras 110a, 110b, 110c via a second network / bus 122. It should be noted that the present invention is not limited to these particular networks / buses 121, 122, and the above-mentioned devices and servers may be connected in different ways. For example, the client device 120, the video processing device 140, the thumbnail generator 150, the indexing server 160, and the recording server 170 may be directly connected to the management server 130 and serve as an interface between them. Alternatively, the management server 130, the video processing device 140, the thumbnail generator 150, the indexing server 160, and the recording server 170 may be directly connected to the client device 120 and serve as an interface between them.

[0027] The management server 130 includes management software that manages information about the configuration of the surveillance / monitoring system 100, such as alarm conditions, details of attached peripherals (hardware), which data streams are being recorded to which recording servers, etc. The management server 130 also manages user information, such as operator privileges. When a client device 120 is connected to the system or a user logs in, the management server 130 determines whether the user is authorized to view the video data. The management server 130 also initiates an initialization or setup procedure in which the management server 130 sends configuration data to the client device 120. This configuration data defines the video cameras in the system and the recording servers to which each camera is connected (if there are multiple recording servers). The client device 120 then caches the configuration data. This configuration data includes information necessary for the client device 120 to identify the video cameras and to retrieve data from the video cameras and / or recording servers. It should be noted that the present invention is not limited to this particular example, and the configuration data may be stored on other devices or servers of the video surveillance system.

[0028] Object detection / recognition can be applied to the video data by object recognition software. This object recognition software generates metadata (or content metadata) that is associated with the video stream and defines where objects have been detected within the frame. This content metadata may also define what type of object or activity was detected, e.g., person, car, dog, bike, alarm, motion. Other types of video analytics software can also generate content metadata, such as behavior detection (e.g., loitering detection), license plate recognition, or facial recognition.

[0029] The object detection / recognition software or any video analysis software that generates the metadata may run on the client device 120 or on an analysis server, or on the recording server 170 or on the video camera itself. It is not essential to the invention that the metadata be generated. The metadata may be stored on the recording server along with the video surveillance data and streamed to the client device with or without having the video data associated with it.

[0030] The video surveillance system of Figure 1 is one example of a system in which the present invention may be implemented. However, other configurations are possible. For example, while the system of Figure 1 is an "on-premise" system, the present invention may also be implemented in a cloud-based system. In a cloud-based system, video cameras stream data to the cloud and at least the recording server 170 resides in the cloud. Furthermore, video analytics may be performed in the cloud. Client devices request video data from the cloud for a user to view.

[0031] Client device 120 may obtain the metadata, for example, from indexing server 160 or recording server 170. Alternatively, the video data obtained by client device 120 may include metadata generated by video analysis performed elsewhere (e.g., on an analysis server, a recording server, or a video camera).

[0032] There are further alternative ways in which the present invention can be implemented. For example, searching for video sequences can be performed at the indexing server 160 or the recording server 170 and the search query is sent from the client device 120.

[0033] In this example, the recording server 170 runs the analysis software without the need for a dedicated analysis server, and the generated metadata is indexed in an organized manner on the indexing server 160 that can be queried by a user via the client device 120, for example, via a graphical user interface (GUI) such as that shown in FIG. 2.

[0034] The GUI of Fig. 2 includes two search plug-ins (SP) 200, 210 that can search for specific types of content in video sequences stored in the recording server 170 and return specific video sequences as search results that match different search queries. In this example, the first search plug-in 200 can search for vehicles of a specific color and type, and the second search plug-in 210 can search for people in a specific pose or performing a specific movement. However, the invention is not limited to this example and can include more than two search plug-ins. The invention can also be implemented using one or more search plug-ins configured to search metadata generated by the analysis software and indexed in the indexing server 160, for example.

[0035] A user may search for different objects or activities of interest of the same type, or combinations thereof, provided the user defines at least two different search criteria. For example, a user may search for blue cars and red cars. Alternatively, a user may search for different types of objects or activities of interest, such as a blue car and an alarm whose sound is recorded in a video sequence. A user may search for a combination of two objects of interest, two activities of interest, or one or more objects and one or more activities of interest. Examples of objects include, but are not limited to, vehicles, license plates, people, animals, signs (such as arrows), or geometric formations. Examples of activities of interest include, but are not limited to, access by an alarm, sound, person, vehicle, or animal to a given area (e.g., an area around or behind a fence), or access to a location or resource controlled or monitored by an access control system (e.g., a hallway behind a door controlled by a badge system).

[0036] Advantageously, content metadata associated with a video sequence and related to objects or activities of interest within the video sequence may be augmented with additional metadata related to access control (e.g., data related to deactivating alarms in a building comprising an area of ​​interest, etc.). One or more of the aforementioned plug-ins may then be configured to retrieve this additional metadata. This additional metadata may be generated by any suitable software known to those skilled in the art.

[0037] Fig. 3 shows a flow chart of a computer-implemented method of video processing according to the invention, which essentially comprises four steps. In a first step S300, the computer will acquire a first video sequence of an area of ​​interest that includes a first predefined object or activity of interest, and in a second step S310, the computer will acquire a second video sequence of an area of ​​interest that includes a second predefined object or activity of interest. It is noted that steps S300 and S310 can be performed simultaneously. Steps S300 and S310 can be performed by the user entering one or more search criteria into two search plug-ins, as described above. In this way, the object and / or activity of interest is predefined by the user. In a next step S320, the computer will determine whether the recording periods of the first and second video sequences overlap in time. By "overlapping in time" it is meant that the video sequences were recorded at least partially simultaneously. For example, a first video sequence was recorded on October 1, 2020, between 4:00 and 4:15, when a blue car (first predetermined object) in the target area was detected and disappeared, and a second video sequence was recorded on the same day, between 4:12 and 4:17, when a pedestrian (second predetermined object) in the same target area was detected. Since the first and second video sequences overlap, it is possible to define a first clip using frames of the first and / or second video sequences from the overlap period showing at least two objects simultaneously. In the above example, a clip may be defined using frames from the first and / or second video sequences from the period of overlap, 4:12-4:15 PM. It is also possible to define a longer video clip that includes more frames before and / or after the overlap period. In the above example, it is possible to define a longer video clip that includes all frames from the first video sequence and all frames from the second video sequence to define a clip corresponding to the recording period between 4:00 PM and 4:17 PM.Alternatively, it is possible to define one or more intermediate clips, for example a clip including all frames from the first and second video sequences from the overlap period with the second video sequence and all other frames from the first video sequence without any other frames from the second video sequence, i.e. a clip corresponding to a recording period between 4:00 and 4:15 PM, or a clip including all frames from the first and second video sequences from the overlap period and all other frames from the second video sequence without any other frames from the first video sequence, i.e. a clip corresponding to a recording period between 4:12 and 4:17 PM.

[0038] Preferably, this clip should include all frames from the first and second video sequences that overlap in time, i.e., all frames of the first and second video sequences from the overlap period between the first and second video sequences and all frames of the first and second video sequences that are outside the overlap period (hereinafter referred to as "OR" scenario), or all frames from the first and / or second video sequences from the overlap period, which correspond to sequences that match all the user's search criteria for the longest possible period (hereinafter referred to as "AND" scenario). Clips corresponding to this "OR" and "AND" scenario will be the most meaningful to the user, since they allow the user to see clips that show the two objects and / or activities of interest at all times that they appeared in the area of ​​interest, whether alone or together ("OR" scenario or "OR" result), or clips that show only the two objects and / or activities of interest at all times that they were together in the same area of ​​interest ("AND" scenario or "AND" result), respectively.

[0039] A clip corresponding to an "AND" scenario, for example, would allow an operator to quickly check whether the runner attempted to carjack the blue car, while a clip corresponding to an "OR" scenario, for example, would allow an operator to more easily identify the runner after the blue car has departed.

[0040] Clearly, clips corresponding to "OR" and "AND" scenarios can be defined on an automated basis based on the user's search criteria.

[0041] It should be noted that the system may be configured to present the user with only clips that correspond to either the "OR" or "AND" result.

[0042] Furthermore, if it is possible to define clips according to an "OR" scenario, it is possible to define clips according to an "AND" scenario, and vice versa. Clips according to the "OR" and "AND" scenarios may be the same even if the recording periods of overlapping search results from different search plug-ins are the same. However, as will be explained in more detail below, it may be advantageous not to display clips according to the "AND" scenario when such clips do not satisfy all of the user's search criteria, for example, when a user searches for three predefined objects and / or activities of interest (or uses three different search plug-ins) and the search results obtained from only two search plug-ins overlap in time. In this way, the user is aware that there may not be a clip that simultaneously shows all of the user's search criteria.

[0043] The first and second video sequences are recorded so as to cover the same region of interest. In this way, frames from the first and second video sequences recorded during the overlap period are identical and it is possible to define the first clip using frames from either of the two video sequences from at least the overlap period, or frames from both video sequences. In the context of the present invention, frames of different video sequences are considered to be identical if they are copies of each other, without taking into account the metadata attached to them. This objective is even easier to achieve if the video sequences originate from the same video camera and correspond to a single field of view of the video camera. This video camera may provide a field of view of more than 180 degrees, for example 360 ​​degrees. However, the region of interest may also correspond to a region observed by several video cameras, for example corresponding to a panoramic field of view obtained by combining the fields of view of different video cameras.

[0044] The method may also include acquiring any additional video sequences of the region of interest that are at least partially recorded during the overlap period with the first and / or second video sequences and that comprise another object and / or activity of interest. In this way, it is possible to define at least one second clip that corresponds to the overlap period between the first and additional video sequences or between the second and additional video sequences. If there are two or more such additional video sequences, the second clip may correspond to the overlap period between at least two of the acquired video sequences (of the first, second and additional video sequences), e.g. between two additional video sequences. It is noted that the reference to a "first" or "second" clip does not preclude the existence of multiple first or second video clips according to the aforementioned "AND" and "OR" scenarios.

[0045] According to the present invention, each of the first and second video clips is derived from at least two video sequences according to the above-mentioned "AND" and "OR" scenarios. It should be noted that a clip may be derived from more than two video sequences if they overlap during a period of time.

[0046] The above method will now be explained using examples in Figs. 4 to 17 showing combinations of search results according to "AND" and "OR" scenarios.

[0047] In the present disclosure, the term "search result" or "individual search result" shall be understood to mean a video sequence (or individual video sequence), such as the first, second or additional video sequence mentioned above, obtained from a "search plug-in", unless otherwise specified. The term "search plug-in" designates means for searching any such video sequence based on at least one search criterion, such as, for example, a search engine having a scroll-down menu with a list of searchable predefined objects or activities of interest, or a search field in which an operator can enter a sentence corresponding to said at least one search criterion. The invention is not limited to the case where search plug-ins are independent of each other from a software or coding point of view.

[0048] In these figures the following keys are used: -SPx-Ry: Input search result "y" searched with search plugin "x". -OR: Output result for "OR" scenarios. -AND: Output result of "AND" scenario. -{z}: A comma-separated list of input search results that form an "OR" or "AND" result. -[------|------]: A span of search results on the "Timeline" defined by opening and closing brackets and a trigger time. "-" indicates a frame in a video sequence. "|" indicates a trigger time. "!" indicates a generated trigger time, as explained further below.

[0049] The metadata associated with a video sequence may advantageously comprise one or more of the following elements: "BeginTime": The beginning or start time of the search results. "EndTime": The end time of the search results. "TriggerTime": A timestamp within the BeginTime and EndTime timespan at which an object and / or activity in a video sequence is visible or recognizable.

[0050] It should be noted that the present invention is not limited to the specific names given in the above definitions. Advantageously, the trigger time corresponds to the time in the video sequence where a given object and / or activity in the video sequence is most obvious or recognizable. In other words, the trigger time preferably corresponds to the "most important" frame in the video sequence, for example, when the video sequence includes a suspicious person walking past, then the trigger time can refer to the frame where the person's face is most recognizable.

[0051] FIG. 4 shows an example in which a first search plug-in (SP1) and a second search plug-in (SP2) each return a first (SP1-R01) and a second (SP2-R01) video sequence, each corresponding to one of the operator's two search queries (or conditions), as described above. Here, it can be seen that the second video sequence was recorded completely while the first video sequence was being recorded. Thus, the recording periods of the first and second video sequences overlap in time, and it is possible to define a first clip corresponding to a combination of both the first and second video sequences ("OR" scenario {SP1-R01, SP2-R01}) and / or a first clip including only frames from the time period of overlap between the first and second video sequences ("AND" scenario {SP1-R01, SP2-R01}). As described above, the operator may choose to view clips from both the "AND" and "OR" scenarios, or from only one of them. Alternatively, the computer may be configured to display results from both scenarios, or from only one scenario. It can also be seen that in this example, the trigger time of the first video sequence (SP1-R01) does not fall within the overlap time between the first and second video sequences. However, the trigger time of the second video sequence (SP2-R01) is within the overlap period. It is therefore advantageous to use a frame from the first and / or second video sequence corresponding to this trigger time to generate a thumbnail including a first predefined object or activity of interest from the first video sequence, together with a second object or activity of interest from the second video sequence. This thumbnail thus represents both the objects or activities in the "AND" and "OR" results.

[0052] FIG. 5 shows an example where the recording periods of the first (SP1-R01) and second (SP1-R02) video sequences do not overlap, i.e. the first and second video sequences were not recorded at the same time. Moreover, search results originating from the same search plug-in are not used to define clips, since they contain the same predefined object or activity of interest and do not contain the second or additional predefined object or activity of interest for which the user is searching. Therefore, for these two reasons, it is not possible to define clips according to "AND" and "OR" scenarios, much less a common thumbnail to represent such clips. In this case, the search results can be presented to the operator, for example, as four individual search results ("OR" results: {SP1-R01} and {SP1-R02}, and "AND" results: {SP1-R01} and {SP1-R02}), or not at all.

[0053] FIG. 6 shows an example where the recording periods of a first (SP1-R01) and a second (SP1-R02) video sequence overlap, but the video sequences originate from the same search plug-in (SP1) (as in the example of FIG. 5). For example, the first ({SP1-R01}) and the second ({SP1-R02}) video sequence each contain a blue car, i.e., the same predefined object of interest as the user's search request. For this reason, again, it is not possible to define any clips according to the "AND" and "OR" scenarios, much less a common thumbnail to represent such any clips. In this case, the search results can be presented to the operator as four individual search results ("OR" results: {SP1-R01} and {SP1-R02}, and "AND" results: {SP1-R01} and {SP1-R02}) or not at all.

[0054] Fig. 7 shows a similar example to Fig. 5, but the first (SP1-R01) and second (SP2-R01) video sequences originate from two different search plug-ins (SP1 and SP2). In cases where the search results of the different search plug-ins do not overlap in time, it may be advantageous not to present any search results to the operator, due to the "AND" scenario. In this way, the operator realizes that it is not possible to define a clip containing two predefined objects of interest and / or activities that correspond to the operator's search query, since these two predefined objects of interest and / or activities never appear simultaneously in the region of interest.

[0055] FIG. 8 shows an example where search results from two different search plug-ins overlap in time, but neither the first (SP1-R01) nor the second video sequence (SP2-R01) was recorded completely during the recording period of the other one. In this case, it is possible to define a clip that includes all frames of the first and second video sequences ("OR" scenario {SP1-R01, SP2-R01}) and / or a clip that includes all frames from the first and second video sequences from the entire period of overlap ("AND" scenario {SP1-R01, SP2-R01}). It can be seen that in this example, neither the trigger time of the first and second video sequences falls within the period of overlap. In such a case where only two search results from different search plug-ins overlap each other, it is proposed to use a frame from the first or second video sequence with the earliest trigger time to generate a thumbnail representing the clip corresponding to the "OR" scenario, and to use the first frame of the clip corresponding to the "OR" scenario to generate a thumbnail representing this clip. The first frame of the clip corresponding to the "AND" scenario used to generate the thumbnail can be defined as having a "made-up trigger time" because no trigger time corresponding to this frame was present in the individual search results.

[0056] According to a not shown example, if the search results from two different search plug-ins overlap in time, but neither the first nor the second video sequence was recorded completely during the recording period of the other of the first and second video sequences, it is also possible to use the frames corresponding to the trigger times made for the "AND" scenario to generate thumbnails representing clips corresponding to the "OR" scenario. In this case, the thumbnails for the "OR" scenario will show both the first and the second predefined objects or activities of interest, in contrast to the example of Fig. 8, and only the first predefined object or activity of interest will be present in the thumbnails representing clips corresponding to the "OR" scenario.

[0057] It should be noted that when frames showing different objects and / or activities of interest are selected as thumbnails for a video sequence or clip, it may be advantageous to surround said objects and / or activities of interest within a bounding box to facilitate identification by the operator.

[0058] FIG. 9 shows another example where a first search plug-in (SP1) returns a single search result (SP1-R01) and a second search plug-in (SP2) returns two search results (SP2-R01 and SP2-R02), where only the first results from the two search plug-ins overlap in time (SP1-R01 and SP2-R01). In this case, it is possible to define a first clip that includes all frames of the temporally overlapping video sequences ("OR" scenario {SP1-R01, SP2-R01}) and / or a first clip that includes only frames from the overlap period between the temporally overlapping video sequences ("AND" scenario {SP1-R01, SP2-R01}). As mentioned above, in such cases where only two search results from different search plug-ins overlap each other, the earliest trigger time of the trigger times of the temporally overlapping video sequences can be used to define the frame used to generate a thumbnail representing any such clip corresponding to the "OR" scenario. Video sequences that do not overlap in time with any other video sequences may be presented as individual search results in an "OR" scenario (such as sequence {SP2-R02} in this example), or may not be presented at all.

[0059] FIG. 10 shows an example including three or more search plug-ins, more specifically, three search plug-ins, where a first search plug-in (SP1) returns a single search result (SP1-R01), a second search plug-in (SP2) returns a single search result (SP2-R01), and a third search plug-in (SP3) returns two search results (SP3-R01 and SP3-R02). The two search results from the third plug-in can be considered as "additional video sequences" as described above. In this example, the first search result (SP3-R01) from the third search plug-in (SP3) overlaps with the first search result (SP1-R01) from the first search plug-in (SP1), and the second search result (SP3-R02) from the third search plug-in (SP3) overlaps with the first search result (SP2-R02) from the second search plug-in (SP2). In this case, it is possible to define two first clips ("OR" scenario {SP1-R01, SP3-R01} and {SP2-R01, SP3-R02}) each containing all frames of video sequences from different plugins that overlap in time. As mentioned above, in such a case where only two search results from different search plugins overlap each other, the earliest trigger time of the trigger times of the video sequences that overlap in time can be used to define the frame that is used to generate the thumbnail representing any such clip corresponding to the "OR" scenario. However, for the "AND" scenario, it may be advantageous not to present any search results to the operator, since there are no video sequences from all the different search plugins that overlap in time at the same time. Therefore, the clips {SP1-R01, SP3-R01} and {SP2-R01, SP3-R02} that correspond to the "AND" scenario are preferably not presented to the user or are removed / deleted. The word "NONE" can for example be displayed in the list of "AND" results. In this manner, the operator realizes that it is not possible to define a clip that includes three predetermined objects of interest and / or activities that correspond to the operator's search query because these three predetermined objects of interest and / or activities will never appear simultaneously in the region of interest.Alternatively, it may not be possible to indicate to the user that any such clips could not be defined.

[0060] FIG. 11 illustrates an example including two search plug-ins, where a first search plug-in (SP1) returns three search results (SP1-R01, SP1-R02, SP1-R03) and a second search plug-in (SP2) returns two search results (SP2-R01, SP2-R02). In this example, the first search result (SP1-R01) of the first search plug-in (SP1) overlaps with the first search result (SP2-R01) and the second search result (SP2-R02) of the second search plug-in (SP2). Also, the second search result (SP1-R02) of the first search plug-in (SP1) overlaps with the second search result (SP2-R02) of the second search plug-in (SP2). As previously discussed (e.g., in the example of FIG. 6), search results from the same search plug-in are not used to define a clip because those search results include the same predefined object or activity of interest and do not include the second or additional predefined object or activity of interest for which the user is searching. In this example, it is possible to define three clips that contain all frames of video sequences from different search plugins that overlap in time ("OR" scenario {SP1-R01, SP2-R01}, {SP1-R01, SP2-R02} and {SP1-R02, SP2-R02}), two clips that contain only frames of the overlapping time between video sequences from different search plugins ("AND" scenario {SP1-R01, SP2-R01} and {SP1-R02, SP2-R02}), and / or one clip that contains a single frame of the overlapping time between video sequences from different search plugins that overlap by one frame ("AND" scenario {SP1-R01, SP2-R02}). Individual search results that do not overlap with other video sequences can be presented as such in the "OR" scenario because these results fulfill at least one of the search criteria of the user's search request.Similar to the example of FIG. 10, in cases where only two search results from different search plugins overlap each other, the earliest trigger time of the trigger times of the video sequences from the different search plugins that overlap in time can be used to define the frame used to generate a thumbnail representing any such clips corresponding to the "OR" scenario. The earliest trigger time that falls within the overlap period of the video sequences from the different search plugins that overlap in time can be used to define the frame used to generate a thumbnail representing any such clips corresponding to the "AND" scenario (such as the "AND" result {SP1-R01, SP2-R01}). Similar to the example of FIG. 8, in cases where the trigger time does not fall within the overlap period of the video sequences from the different search plugins that overlap each other in time, the first frame of the corresponding clip defined in the "AND" scenario can be used to generate a thumbnail representing this clip. The first frame of each clip corresponding to the "AND" scenario used to generate a thumbnail can be defined as having a made-up trigger time as previously described with reference to FIG. 8 (see the "AND" scenarios {SP1-R01, SP2-R02} and {SP1-R02, SP2-R02} in this example).

[0061] FIG. 12 shows an example including three search plugins, where three video sequences from different search plugins overlap in time (search results SP1-R01, SP2-R01, SP3-R02). It is therefore possible to define a clip that uses all frames of all video sequences from different plugins that overlap in time (an "OR" scenario ({SP1-R01, SP2-R01, SP3-R02}) and / or a clip that uses only frames of the overlapping time of all video sequences from different plugins that overlap in time (an "AND" scenario {SP1-R01, SP2-R01, SP3-R02}). In such a case where three or more search results from different search plugins overlap each other, the earliest trigger time that overlaps with the earliest search result among the trigger times of the video sequences from the different search plugins that overlap in time can be used to define the frames that are used to generate thumbnails representing any such clips corresponding to the "OR" scenario. Furthermore, similar to the example of FIG. 10, the earliest trigger time that falls within the overlapping period of the video sequences from the different search plugins that overlap in time can be used to define the frames that are used to generate thumbnails representing any such clips corresponding to the "OR" scenario. (where the trigger time of SP1-R01 is the only trigger time that falls within the overlap period of search results SP1-R01, SP2-R01, and SP3-R02, and is therefore used to define the frames used to generate thumbnails representing clips corresponding to the "AND" scenario {SP1-R01, SP2-R01, SP3-R02}.) In this example, clips from video sequences SP2-R01 and SP3-R01, individual search result SP3-R03, and their thumbnails (in the "OR" scenario) are obtained as described in the previous example.

[0062] 13-16 show additional examples in which at least one of the individual search results includes a preliminary sequence at the beginning of the video sequence and at least one closing sequence at the end of the video sequence. The preliminary and closing sequences are individually referred to as padding sequences, i.e. sequences in which a given object or activity of the video sequence does not necessarily appear and / or the video quality is lower than that of the rest of the video sequence. Such padding sequences may exist because the conditions of the acquisition of the video sequence change before normalization, for example, the target area is a dark room and the light is turned on, thus creating a padding sequence in which the light has not yet reached its full brightness and the quality of the recording of the video sequence is consequently lower. This padding sequence may also exist because the video camera starts recording when triggered by a sensor covering an area larger than the target area, and the given object or activity of interest falls into the target area after the start of the recording of the video sequence.

[0063] Whenever such padding sequences are present, they are preferably not taken into account for purposes of determining whether recording periods of video sequences overlap in time with one another, however, it may be advantageous to include any such padding sequences in clips defined according to the "AND" and "OR" scenarios.

[0064] 13 to 16, any padding sequences are represented by thin lines and any non-padding sequences are represented by thick lines. Individual frames of the video sequences are no longer shown for simplicity.

[0065] The metadata associated with a video sequence may advantageously comprise one or more of the following elements: "BeginTime": The beginning or start time of the search results, not including padding. "BeginTimeWithPadding": The beginning or start time of the search results including padding. "EndTime": The end time of the search results, not including padding. "EndTimeWithPadding": The end time of the search results including padding. "TriggerTime": A timestamp within the BeginTime and EndTime timespan at which an object and / or activity in a video sequence is visible or recognizable.

[0066] It should be noted that the present invention is not limited to the specific names given in the above definitions.

[0067] As previously mentioned, the trigger time corresponds to a time within a video sequence at which a given object and / or activity within the video sequence is most apparent or recognizable.

[0068] FIG. 13 shows an example where the first search result (SP1-R01) of the first search plug-in (SP1) contains two padding sequences, namely a preliminary and a closing sequence, while the first search result of the second search plug-in does not contain a padding sequence. In this example, it can be seen that the search results from these two plug-ins only overlap each other, since the closing sequence of the first search result (SP1-R01) of the first search plug-in (SP1) overlaps with the first search result (SP2-R01) of the second search plug-in (SP2). Since this padding sequence is not taken into account for determining whether the video sequences overlap, the two search results are considered not to overlap, and therefore it is not possible here to define a clip using frames of the first and / or second video sequences according to the "AND" and "OR" scenarios. However, the individual search results can still be presented to the operator as two individual search results ({SP1-R01} and {SP2-R01}) under the "OR" scenario. Additionally, as discussed with reference to FIG. 10, the word "NONE" may, for example, appear in the list of "AND" results since there are no clips that correspond to the "AND" scenario.

[0069] FIG. 14 shows an example similar to the one in FIG. 13, but where a first search result (SP1-R01) of a first search plug-in (SP1) overlaps with a first search result (SP2-R01) of a second search plug-in (SP2) without taking into account the padding sequence of the first search result (SP1-R01) of the first search plug-in (SP1). In this case, it is possible to define a clip that includes all frames of the first and second video sequences ("OR" scenario {SP1-R01, SP2-R01}) and / or a clip that includes all frames from the first and second video sequences from the entire period of overlap ("AND" scenario {SP1-R01, SP2-R01}). Although this padding sequence is not taken into account for determining whether the video sequences overlap, frames of padding sequences from video sequences from different search plug-ins that overlap in time may still be included in the clip defined under the "OR" scenario. However, frames of a padding sequence from a video sequence that overlaps in time with another video sequence from another search plug-in will only be included in a clip defined under the "AND" scenario to the extent that the padding sequence also overlaps with a padding or non-padding sequence of said other video sequence. Preferably, frames of any padding sequence that overlaps with a non-padding sequence from a different search plug-in will be presented as a frame from a non-padding sequence in the "OR" scenario (e.g., the closing sequence of the search result SP1-R01 presented as a non-padding sequence in the "OR" scenario {SP1-R01, SP2-R01}). On the other hand, frames of any padding sequence that overlaps with a non-padding sequence from a different search plug-in will preferably be presented as a frame from a padding sequence in the "AND" scenario (e.g., the closing sequence of the search result SP1-R01 presented as a padding sequence in the "AND" scenario {SP1-R01, SP2-R01}). Thumbnails for clips defined according to the "AND" and "OR" scenarios are defined similarly to the previous examples.

[0070] Figure 15 shows an example similar to that of figure 14, with the difference that the first search result (SP2-R01) of the second search plug-in (SP2) is recorded completely while the first result (SP1-R01) of the first search plug-in (SP1) is recorded, and with the difference that the closing sequence of the first search result of the first search plug-in does not overlap with the first search result of the second search plug-in. According to the rules mentioned with reference to figure 14, the frames of the padding sequence of the first search result of the first search plug-in are preferably presented as frames from the padding sequence, since they are included in a clip containing all frames of the overlapping video sequences ("OR" scenario {SP1-R01, SP2-R01}) and do not overlap with the first search result of the second search plug-in. Also, frames of the padding sequence of the first search result of the first search plug-in are not included in the clip that includes all frames from video sequences from all periods of temporal overlap ("AND" scenario {SP1-R01, SP2-R01}) because the padding sequence of the first search result of the first search plug-in does not overlap in time with the first search result of the second search plug-in. Thumbnails for clips defined according to the "AND" and "OR" scenarios are again defined similarly to the previous examples.

[0071] Fig. 16 shows a final example of a method for video processing based on the example shown in Fig. 15, but with the difference that the first search result (SP2-R01) from the second search plug-in (SP2) also includes a spare and a closed padding sequence. In this case, it can be seen that the spare sequence of the first search result of the second search plug-in partially overlaps with the spare sequence of the first search result of the first search plug-in and partially overlaps with the non-padding part of the first search result of the first search plug-in. As explained above with respect to Fig. 14, frames of any padding sequence that overlap with a non-padding sequence from a different search plug-in will preferably be presented as frames from a non-padding sequence in an "OR" scenario, and frames of any padding sequence that overlap with another padding sequence from a different search plug-in will preferably be presented as frames from a padding sequence in an "OR" scenario. On the other hand, frames of any padding sequence that overlap with a non-padding sequence from a different search plug-in will preferably be presented as frames from a padding sequence in an "AND" scenario, regardless of whether the frames of the padding sequence overlap with a padding or non-padding sequence.

[0072] Fig. 17 shows a flow chart of a computer-implemented method of generating thumbnails from video sequences according to the invention, which essentially comprises three steps. In a first step S400, the computer will identify a first frame in the video sequence that contains a first object or activity of interest, and in a second step S410, the computer will identify a second frame in the video sequence that contains a second object or activity of interest. Alternatively, steps S400 and S410 can be performed simultaneously. The first and second frames are identified in clips resulting from the two video sequences, preferably according to the computer-implemented method of video processing described above. In a next step S420, the computer will generate a thumbnail using at least a portion of the first frame and at least a portion of the second frame of the video sequences or clips, the thumbnail will include the first and second objects or activities.

[0073] The method may, for example, include a step in which a first and a second frame of a video sequence are extracted or copied from the video sequence for generating the thumbnail, in other words, the frames may be copied from the video sequence or clip before being used to generate the thumbnail.

[0074] FIG. 18 shows thumbnails that would typically be presented to an operator searching for runners and cars. In this example, each thumbnail presented in the grid view of thumbnails 500 corresponds to an individual search result that corresponds to at least one of the operator's search criteria. Car results 1, 6, 8, and 10 come from a first search plugin, and runner results 2, 3, 4, 5, 7, 9, 11, and 12 come from a second search plugin. Objects and / or activities of interest searched for by the user are preferably displayed within a bounding box on each thumbnail to facilitate their identification by the operator. In this example and the following examples, the thumbnails are clickable to open the video sequence to which the thumbnail corresponds. It should be noted that the present invention is not limited to the grid view display described above. The numbers above the thumbnails are also not an essential feature of the present invention.

[0075] However, as is clear from this example, simply showing all results from two search plugins or criteria individually can make them very difficult for an operator to process. Often there will be many results, and an operator cannot easily determine whether the cars and runners shown in the result thumbnails are reasonably close in time to each other. The thumbnails displayed know nothing about the other search plugins, and are based on the trigger time at which the search plugin independently selected the search results.

[0076] The aforementioned computer-implemented method of video processing allows an operator to define at least one clip that includes both first and second objects or activities of interest for which he is searching, and the aforementioned computer-implemented method of generating thumbnails from a video sequence allows an operator to further define a thumbnail that includes both the first and second objects or activities of interest.

[0077] Figure 19 shows an example where both methods of video processing and thumbnail generation are used. In this example, the results 5+6, 7+8, and 10+11 can be observed to be merged into a combined video sequence or clip, and the operator then knows that the car and the runner appeared at the same time in these clips.

[0078] An operator may choose to remove individual search results that are not clips that match all of the user's search criteria, for example, when a user searches for three given objects and / or activities of interest (or uses three different search plug-ins), but the search results from only two of the search plug-ins overlap in time. In this way, the user is aware that no clip may exist that simultaneously exhibits all of the user's search criteria.

[0079] Applying this to the example of Figure 18, the operator then only needs to look at the three combined video sequences in which the car and runner appeared together, giving the thumbnails 610a, 620a, 630a shown in Figure 20. While this gives a much better overview of the relevant video data, the operator still needs to examine the three clips to identify which ones contain the exact car and person being searched for.

[0080] FIG. 21 shows three additional examples of thumbnails 610b, 620b, 630b that can be obtained according to the method of generating thumbnails according to the present invention. According to the example of thumbnail 610b corresponding to search result 5+6, the computer takes a portion of the frames that correspond to the trigger times for the two individual video sequences and generates a single combined thumbnail by simply stitching the individual thumbnails together. The "stitching boundary" is selected by the computer based on the location of the objects and / or activities of interest associated with the individual search results. According to the example of thumbnail 620b corresponding to search result 7+8, the computer proceeds as in the first example, but does not add any boundary to separate the first and second frames or portions thereof. According to the example of thumbnail 630b corresponding to search result 10+11, the computer selects a frame that corresponds to the trigger time for one of the individual video sequences and uses it as the base thumbnail. From the frame that corresponds to the trigger time of the other individual video sequence, the computer extracts the relevant objects or activities of interest and overlays it with the base thumbnail to represent the combined video sequence or clip.

[0081] Figures 22(a) and 22(b) show further examples of thumbnails that can be obtained according to the method of generating thumbnails according to the invention. According to the example of thumbnail 610c corresponding to search result 5+6, from the frames that coincide with the trigger times of the two individual search results, the computer extracts the relevant objects and / or activities of interest, enlarges them and generates a thumbnail that represents the combined video sequence or clip. This thumbnail preferably includes a border as described above with reference to Figure 21. According to the example of thumbnail 620c corresponding to search result 7+8, the computer uses a predefined background (such as a neutral reference frame) as a base thumbnail. The neutral frame may advantageously be a picture of the target area imaged without the first and second objects or activities of interest or without any objects or activities therein.

[0082] From the frames that match the trigger times of the two individual search results, the computer extracts the relevant objects and overlays them with a predefined background that represents the combined video sequence or clip. According to the example of thumbnails 630c, 630d corresponding to search result 10+11, the computer retrieves the frames that match the trigger times for the two individual video sequences and shows them one by one as thumbnails in the "pagination control". This switching between thumbnails can be done in various ways, for example, by manually clicking a button that allows the operator to switch between the two frames of the thumbnail. Alternatively, the computer may display the frames that match the trigger times for the two individual video sequences and show them one by one as thumbnails over a predefined time period.

[0083] Although the examples shown in Figures 19-22(b) relate to clips containing only two objects or activities of interest, the invention also covers the case where more than two objects are searched for (similar to the previous example relating to "additional video sequences") and therefore the corresponding generated thumbnail will contain more than two objects or activities.

[0084] Although the present invention has been described with reference to various embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, and that the present invention can be embodied in various forms without departing from the main characteristics of the invention as defined in the claims.

[0085] The following numbered statements provide additional information regarding the present invention. The claims are provided after this numbered statement.

[0086] [Description 1] 1. A computer-implemented method for video processing, comprising: acquiring a first video sequence of an area of ​​interest including a first predetermined object or activity of interest; acquiring a second video sequence of the region of interest including a second predetermined object or activity of interest; determining whether a recording period of the first video sequence and a recording period of the second video sequence overlap for a period of time; if the recording periods of the first and second video sequences overlap for a period of time, defining at least one first video clip using frames of the first and / or second video sequences from at least the period of overlap; A method comprising:

[0087] [Statement 2] acquiring at least one additional video sequence of the region of interest, each additional video sequence including a predetermined object or activity of interest within the region of interest and having a recording duration; determining whether the recording period of each additional video sequence overlaps over a period with the recording period of any other additional video sequence and / or the recording period of the first and / or second video sequence; if a recording period of any other of said first, second and other additional video sequences overlaps with a recording period of said additional video sequence for a period of time, defining at least a second video clip using frames of said additional and / or other video sequences from said period of overlap; The method as described in statement 1, further comprising:

[0088] [Statement 3] A method as described in description 1 or 2, wherein all the video sequences are obtained by retrieving video data stored in a recording server, the recording server storing video data from multiple video cameras stored therein.

[0089] [Statement 4] A method as recited in any one of statements 1 to 3, further comprising generating metadata about all the given objects or activities of interest.

[0090] [Statement 5] 5. A method as described in any one of statements 1 to 4, wherein all the video sequences originate from a single video camera.

[0091] [Statement 6] 6. A method as described in any one of statements 1 to 5, wherein the region of interest corresponds to the field of view of a single video camera.

[0092] [Statement 7] 7. A method as recited in any one of statements 1 to 6, wherein all the predefined objects are objects of a type selected from a group of objects consisting of vehicles, license plates, humans, and animals.

[0093] [Statement 8] 8. A method as recited in claim 7, wherein all of the given objects correspond to different types of objects within the group of objects.

[0094] [Statement 9] The aforementioned method in combination with description 3, wherein searching for video data is performed by a user inputting one or more search criteria into at least two search plug-ins.

[0095] [Statement 10] The above method in combination with description 7 or 8, wherein each search plug-in is a search engine configured to identify an object or an activity within the group of objects of interest in all of the video sequences.

[0096] [Statement 11] A method as described in any one of statements 1 to 10, wherein all of the video sequences are obtained as search results of at least two queries from the user, each query relating to a different predefined object or activity of interest among the predefined objects or activities of interest.

[0097] [Statement 12] 12. A method as recited in any one of claims 1 to 11, wherein each said video sequence includes content metadata relating to a predefined object or activity of interest in said video sequence.

[0098] [Statement 13] 4. The method as recited in claim 3, wherein the content metadata is generated by an analytics plug-in on the recording server.

[0099] [Statement 14] 14. The method as recited in any one of statements 1 to 13, wherein obtaining all the video sequences is done by searching content metadata of different video sequences.

[0100] [Statement 15] 15. A method as described in any one of statements 1 to 14, wherein at least one video sequence of all the video sequences includes at least one preliminary sequence at the beginning of the video sequence and / or at least one closing sequence at the end of the video sequence, the preliminary and / or closing video sequences, respectively, referred to as padding sequences, in which the predetermined object or activity of interest of the video sequence does not appear and / or the video quality is lower than that of the remainder of the video sequence.

[0101] [Statement 16] 16. The method as recited in claim 15, wherein at least one of the padding sequences is not considered in determining whether all of the video sequences overlap each other for the period of time.

[0102] [Statement 17] 17. The method as recited in any one of statements 1-16, wherein at least the first video clip includes all frames of the first or second video sequences from the period of overlap.

[0103] [Statement 18] 7. The method as recited in claim 6, wherein at least the first video clip further includes all frames of the first and / or second video sequences from before and after the period of overlap.

[0104] [Statement 19] The above method in combination with statements 2 and 9 or 2 and 10, further comprising canceling the definition of the first and / or second video clip when three or more search plugins are used when querying the at least one database and when the first and / or second video clip does not include results from all of the three or more search plugins used.

[0105] [Statement 20] A method as described in any one of claims 1 to 19, further comprising displaying the first video clip or a thumbnail of the first video clip to a user without displaying the first and second video sequences in which the first video clip is defined.

[0106] [Statement 21] identifying a first frame of the first video clip that includes the first predetermined object or activity of interest; identifying a second frame of the first video clip that includes the second predetermined object or activity of interest; generating a thumbnail using at least a portion of the first frame and at least a portion of the second frame of the first video clip, the thumbnail including the first and second predetermined objects or activities; 20. The method as recited in any one of statements 1 to 19, further comprising:

[0107] [Statement 22] 22. The method as recited in any one of statements 1-21, comprising extracting the first frame and the second frame from the first video clip to generate the thumbnail.

[0108] [Statement 23] extracting the first or second predetermined object or activity from one of the two frames; overlaying the extracted first or second predetermined object or activity with the other of the two frames; 23. The method as recited in claim 21 or 22, further comprising generating the thumbnail by:

[0109] [Statement 24] extracting the first and second predetermined objects or activities from the two frames; overlaying the extracted first and second predetermined objects or activities with a predetermined background; 23. The method as recited in claim 21 or 22, further comprising generating the thumbnail by:

[0110] [Statement 25] 25. A method as recited in any one of statements 1-24, wherein the first and second predetermined objects or activities are magnified on the predetermined background.

[0111] [Statement 26] 23. A method as recited in claim 21 or 22, further comprising displaying each of the two frames, one after the other, as the thumbnail.

[0112] [Statement 27] 27. A method as recited in any one of statements 1 to 26, wherein each frame of the thumbnail is displayed in sequence for a predetermined period of time.

[0113] [Statement 28] 27. The method as recited in claim 26, wherein the thumbnail comprises at least one clickable button that allows a user to switch between the two frames of the thumbnail.

[0114] [Statement 29] 29. A method as recited in any one of statements 21 to 28, wherein the first and second predetermined objects or activities are each contained within a bounding box on the thumbnail.

[0115] [Statement 30] 30. A method as recited in any one of statements 21 to 29, wherein the first and / or second predetermined object is an object of a type selected from the group of objects consisting of vehicles, license plates, humans and animals.

[0116] [Statement 31] A method as recited in any one of statements 1-30, wherein the first and / or second predetermined objects correspond to different types of objects within the group of objects.

[0117] [Statement 32] 32. A method as recited in any one of Descriptions 21-31, wherein the first video clip includes metadata defining a start time and an end time of the first video clip.

[0118] [Statement 33] 33. The method as recited in any one of statements 21-32, further comprising displaying one of the thumbnails.

[0119] [Statement 34] 34. A method as described in any one of claims 21 to 33, comprising searching the first video clip based on object or facial recognition, motion, or bounding boxes present in the first video clip, or searching content metadata associated with the first video clip, to identify the first and second predetermined objects or activities of interest before identifying the first and second frames.

[0120] [Statement 35] A method as described in any one of claims 1 to 34, wherein searching content metadata includes sending a search request to an indexing server that stores the content metadata, and receiving search results from the indexing server indicating video frames that match search criteria.

[0121] [Statement 36] 36. A method as recited in any one of claims 1 to 35, wherein the content metadata includes trigger times corresponding to the frames that include the first and second predetermined objects or activities.

[0122] [Statement 37] A method as described in any one of statements 1 to 36, wherein the trigger time corresponds to a frame of the first video clip and the first and second predetermined objects or activities are most obvious or recognizable in the first video clip.

[0123] [Statement 38] A method as described in any one of statements 21 to 37, wherein the thumbnail includes an internal border separating one of the two frames or a portion thereof from the other of the two frames, the border being positioned so as not to obscure the first and second predetermined objects or activities.

[0124] [Statement 39] A computer program which, when executed on a computer, causes the computer to carry out a method as defined in any one of statements 1 to 38.

[0125] [Statement 40] receiving from a user a first query related to a first predetermined object or activity of interest and a second query related to a second predetermined object or activity of interest; obtaining, from at least one database, a first video sequence of an area of ​​interest including a first predetermined object or activity of interest and a second video sequence of an area of ​​interest including a second predetermined object or activity of interest; determining whether a recording period of the first video sequence and a recording period of the second video sequence overlap for a period of time; and defining at least one first video clip using frames of the first and / or second video sequences from a period of overlap when the recording periods of the first and second video sequences overlap for a period of time. 16. A video processing device comprising: at least one processor configured to:

[0126] [Statement 41] The at least one processor: identifying a first frame of the first video clip that includes the first predetermined object or activity of interest; identifying a second frame of the first video clip that includes the second predetermined object or activity of interest; generating a thumbnail including the first and second predetermined objects or activities using at least a portion of the first frame and at least a portion of the second frame of the first video clip; 41. The video processing device as recited in claim 40, further configured to:

[0127] [Statement 42] A video surveillance system comprising a video processing device as described in description 41 or 42 and a client device having a display, the client device comprising at least one processor configured to display at least the first video clip or a thumbnail of the first video clip.

[0128] [Statement 43] 43. A video surveillance system as described in claim 42, wherein the client device is further configured to display the first video clip or a thumbnail of the first video clip without displaying the first and second video sequences in which the first video clip is defined.

[0129] [Statement 44] A video surveillance system as described in claim 41 or 42, further comprising a video camera and a recording server, the recording server comprising: Recording the video sequence from a video camera in a database in a storage medium; receiving a search query from the client device; Extracting video sequences from the database in accordance with the search query; Defining and transmitting one of the first video clips to the client device A video surveillance system comprising a processor configured to:

Claims

1. A method performed by a video processing device, comprising: obtaining, by a first search plug-in, a first video sequence of an area of ​​interest that includes a first predetermined object or activity of interest; obtaining, by a second search plug-in, a second video sequence of the region of interest that includes a second predetermined object or activity of interest; determining whether a recording period of the first video sequence and a recording period of the second video sequence overlap; if there is an overlap period during which recording periods of the first and second video sequences overlap, defining at least one first video clip for presentation to a user using frames of the first and / or second video sequences from at least the overlap period; A method comprising:

2. obtaining at least one additional video sequence of the subject area from a third search plug-in, each additional video sequence including a predefined object of interest or activity within the subject area and having a recording duration; determining whether the recording period of each additional video sequence overlaps over a period with the recording period of any other additional video sequence and / or with the recording period of the first and / or second video sequences; if a recording period of any other of said first, second and other additional video sequences overlaps with a recording period of said additional video sequence for a period of time, defining at least a second video clip using frames of said additional and / or said other video sequences from said overlapping period; The method of claim 1 further comprising:

3. The method of claim 1 , wherein all the video sequences originate from a single video camera.

4. The method of claim 1 , wherein the region of interest corresponds to the field of view of a single video camera.

5. A method according to any one of claims 1 to 4, wherein the first and / or second predetermined objects of interest are objects of a type selected from a group of objects including vehicles, license plates, humans and animals.

6. The method of claim 1 , wherein at least the first video clip comprises all frames of the first or second video sequences from the overlapping period.

7. The method of claim 6 , wherein at least the first video clip further comprises all frames of the first and / or second video sequences from before and after the overlap period.

8. identifying a first frame of the first video clip that includes the first predetermined object or activity of interest; identifying a second frame of the first video clip that includes the second predetermined object or activity of interest; generating a thumbnail using at least a portion of the first frames and at least a portion of the second frames of the first video clip, the thumbnail including the first and second predetermined objects or activities of interest; The method of claim 1 , further comprising:

9. extracting the first or second predetermined object or activity of interest from one of the two frames; overlaying the extracted first or second predetermined object of interest or activity with the other of the two frames; The method of claim 8 , further comprising generating the thumbnail by:

10. extracting the first and second predetermined objects or activities of interest from two frames; overlaying the extracted first and second predetermined objects of interest or activities with a predetermined background; The method of claim 8 , further comprising generating the thumbnail by:

11. The method of claim 10 , wherein the first and second predetermined objects or activities of interest are magnified on the predetermined background.

12. The method of claim 8 , further comprising displaying each of two frames, one after the other, as the thumbnail.

13. 13. A method according to any one of claims 8 to 12, wherein the first and second predetermined objects or activities of interest are each contained within a bounding box on the thumbnail.

14. 14. A method according to any one of claims 8 to 13, wherein the first and / or second predetermined objects of interest are objects of a type selected from a group of objects comprising vehicles, license plates, humans and animals.

15. 15. A method according to any one of claims 8 to 14, wherein the first video clip includes metadata defining a start time and an end time of the first video clip.

16. 16. The method of claim 8, further comprising: searching the first video clip based on object or face recognition, motion, or bounding boxes present in the first video clip, or searching content metadata associated with the first video clip, to identify the first and second predetermined objects or activities of interest before identifying the first and second frames.

17. 17. The method of claim 16, wherein searching the content metadata includes sending a search request to an indexing server that stores the content metadata, and receiving search results from the indexing server that indicate video frames that match search criteria.

18. The method of claim 17 , wherein the content metadata includes trigger times corresponding to the frames that include the first and second predetermined objects or activities of interest.

19. 19. The method of claim 8, wherein the thumbnail includes an internal border separating one of the two frames or a portion thereof from the other of the two frames, the border being positioned so as not to obscure the first and second predetermined objects or activities of interest.

20. A computer program which, when executed on a computer, causes the computer to carry out the method according to any one of claims 1 to 19.

21. A first acquisition means for acquiring, by a first plug-in, a first video sequence of a region of interest that includes a first predetermined object or activity of interest; second acquisition means for acquiring, by a second plug-in, a second video sequence of an area of ​​interest that includes a second predetermined object or activity of interest; a determination means for determining whether a recording period of the first video sequence overlaps with a recording period of the second video sequence; means for defining, when an overlap period exists during which recording periods of the first and second video sequences overlap, at least one first video clip using frames of the first and / or second video sequences from the overlap period for providing to a user; A video processing device comprising:

22. 22. A video surveillance system comprising: the video processing device according to claim 21; and a client device, wherein the client device displays at least the first video clip or a thumbnail of the first video clip on a display.

23. 23. The video surveillance system of claim 22, wherein the client device is further configured to display the first video clip or a thumbnail of the first video clip without displaying the first and second video sequences in which the first video clip is defined.

Images