Object analysis method and system based on video processing

Abstract

The present invention provides an object analysis method and system based on video processing, which relate to the field of data processing technology. In the present invention, a monitoring video frame sequence of an object to be processed and a monitoring audio frame sequence of an object to be processed are separated; a key information mining operation is performed on the monitoring video frame sequence of the object to be processed to output an original video feature representation; a multi-stage feature aggregation operation is performed on the original supplementary feature representation and the original video feature representation corresponding to the monitoring audio frame sequence of the object to be processed to obtain multiple aggregated video feature representations; a feature merging operation is performed on the multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, an important object monitoring video frame subsequence is analyzed; and the important object monitoring video frame subsequence is used as a representative video of the object to be processed. Based on the above method, the reliability of object analysis can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an object analysis method and system based on video processing. Background Art

[0002] Video surveillance is used in a variety of application scenarios. After collecting the object surveillance video, the corresponding surveillance object can be analyzed and processed based on the object surveillance video. In the process of analysis, extracting important local videos from the object surveillance video is an important link, which can improve the accuracy and efficiency of subsequent processing. However, in the prior art, there is a problem of poor reliability in the process of extracting important local videos. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an object analysis method and system based on video processing to improve the reliability of object analysis.

[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0005] An object analysis method based on video processing, comprising:

[0006] For a surveillance video of an object to be processed, separating a surveillance video frame sequence of the object to be processed and a surveillance audio frame sequence of the object to be processed in the surveillance video of the object to be processed, wherein the surveillance video of the object to be processed is formed based on video surveillance of the object to be processed;

[0007] Performing a key information mining operation on the surveillance video frame sequence of the object to be processed to output a corresponding original video feature representation;

[0008] Performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations;

[0009] Performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed based on the original video feature representation and the merged video feature representation;

[0010] The important object monitoring video frame subsequence is used as the representative video of the object to be processed.

[0011] In some preferred embodiments, in the above-mentioned object analysis method based on video processing, the step of performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation to obtain the aggregated video feature representations formed by the multi-stage feature aggregation operations respectively includes:

[0012] The feature aggregation operations in the first stage include:

[0013] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0014] Adding the original supplementary feature representation to the original video feature representation to form an intermediate feature representation corresponding to the original video feature representation;

[0015] Performing an aggregation operation on the intermediate feature representation and the original supplementary feature representation, and performing a key information mining operation on the feature representation formed by the aggregation, so as to form an aggregated video feature representation corresponding to the feature aggregation operation of the first stage;

[0016] For each stage of feature aggregation operation except the first stage, the feature aggregation operation includes:

[0017] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0018] Adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation;

[0019] An aggregation operation is performed on the intermediate feature representation and the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, and a key information mining operation is performed on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation in the current stage.

[0020] In some preferred embodiments, in the above-mentioned object analysis method based on video processing, the step of performing a key information mining operation on the monitoring audio frame sequence of the object to be processed to output the original supplementary feature representation corresponding to the monitoring audio frame sequence of the object to be processed includes:

[0021] Performing a feature space mapping operation on the monitored audio frame sequence of the object to be processed to form a corresponding feature space mapping feature representation, and performing a row and column distribution adjustment operation of feature representation parameters on the feature space mapping feature representation to form a first adjusted feature representation;

[0022] performing a filtering operation on the first adjusted feature representation to form a filtered feature representation;

[0023] Performing a row-column distribution adjustment operation of feature representation parameters on the filtered feature representation to form a corresponding second adjusted feature representation;

[0024] The second adjusted feature representation is subjected to a filtering operation to form an original supplementary feature table corresponding to the monitoring audio frame sequence of the object to be processed.

[0025] In some preferred embodiments, in the above-mentioned object analysis method based on video processing, the step of adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form the corresponding intermediate feature representation includes:

[0026] Performing a row-column distribution adjustment operation on the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, so as to form an adjusted aggregated video feature representation;

[0027] Performing an aggregation operation on the adjusted aggregated video feature representation and the original supplementary feature representation to form a corresponding optimized supplementary feature representation;

[0028] An aggregation operation is performed on the adjusted aggregated video feature representation and the optimized supplementary feature representation to form a corresponding intermediate feature representation.

[0029] In some preferred embodiments, in the above-mentioned object analysis method based on video processing, the step of performing an aggregation operation on the adjusted aggregated video feature representation and the original supplementary feature representation to form a corresponding optimized supplementary feature representation includes:

[0030] Performing a splicing operation on the adjusted aggregated video feature representation and the original supplementary feature representation, and performing a linear integration operation on the feature representation formed by the splicing operation to form a corresponding linearly integrated first feature representation;

[0031] Performing parameter distribution adjustment and filtering operations on the linearly integrated first feature representation to form a corresponding optimized linearly integrated first feature representation;

[0032] A superposition operation is performed on the original supplementary feature representation and the optimized linearly integrated first feature representation, and a filtering operation is performed on the feature representation formed by the superposition operation to form a corresponding optimized supplementary feature representation.

[0033] In some preferred embodiments, in the above-mentioned object analysis method based on video processing, the step of performing an aggregation operation on the adjusted aggregated video feature representation and the optimized supplementary feature representation to form a corresponding intermediate feature representation includes:

[0034] Performing a splicing operation on the adjusted aggregated video feature representation and the optimized supplemental feature representation, and performing a linear integration operation on the feature representation formed by the splicing operation to form a corresponding linearly integrated second feature representation;

[0035] Performing parameter distribution adjustment and filtering operations on the linearly integrated second feature representation to form a corresponding optimized linearly integrated second feature representation;

[0036] A superposition operation is performed on the optimized supplementary feature representation and the optimized linearly integrated second feature representation, and a filtering operation is performed on the feature representation formed by the superposition operation to form a corresponding intermediate feature representation.

[0037] In some preferred embodiments, in the above-mentioned object analysis method based on video processing, the step of performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed, comprises:

[0038] For each aggregated video feature representation in the plurality of aggregated video feature representations:

[0039] Extracting an overall distribution environment feature representation of an aggregated video feature representation, and determining a corresponding overall video feature representation based on the overall distribution environment feature representation and the aggregated video feature representation;

[0040] Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation;

[0041] Performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation;

[0042] Aggregating the hole-filtered video feature representation and the original video feature representation to form a corresponding deep aggregated video feature representation, and filtering and interpolating the deep aggregated video feature representation to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed;

[0043] The step of performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation includes:

[0044] For each of the configured multiple hole parameters, performing a hole filtering operation on the merged video feature representation based on the hole parameter to form a local hole filtering video feature representation corresponding to the hole parameter;

[0045] An aggregation operation is performed on a plurality of local hole filter video feature representations corresponding to the plurality of hole parameters to form a hole filter video feature representation corresponding to the merged video feature representation.

[0046] The embodiment of the present invention further provides an object analysis system based on video processing, comprising:

[0047] A monitoring video separation module, used for separating a monitoring video of an object to be processed into a monitoring video frame sequence of the object to be processed and a monitoring audio frame sequence of the object to be processed in the monitoring video of the object to be processed, wherein the monitoring video of the object to be processed is formed based on video monitoring of the object to be processed;

[0048] A key information mining module is used to perform a key information mining operation on the monitoring video frame sequence of the object to be processed to output a corresponding original video feature representation;

[0049] A feature aggregation module, used for performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation, so as to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations;

[0050] An important video frame extraction module is used to perform a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyze the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed;

[0051] The object representative video determination module is used to use the important object monitoring video frame subsequence as the representative video of the object to be processed.

[0052] In some preferred embodiments, in the above-mentioned object analysis system based on video processing, the feature aggregation module is specifically used for:

[0053] The feature aggregation operations in the first stage include:

[0054] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0055] Adding the original supplementary feature representation to the original video feature representation to form an intermediate feature representation corresponding to the original video feature representation;

[0056] Performing an aggregation operation on the intermediate feature representation and the original supplementary feature representation, and performing a key information mining operation on the feature representation formed by the aggregation, so as to form an aggregated video feature representation corresponding to the feature aggregation operation of the first stage;

[0057] For each stage of feature aggregation operation except the first stage, the feature aggregation operation includes:

[0058] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0059] Adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation;

[0060] An aggregation operation is performed on the intermediate feature representation and the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, and a key information mining operation is performed on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation in the current stage.

[0061] In some preferred embodiments, in the above-mentioned object analysis system based on video processing, the important video frame extraction module is specifically used for:

[0062] For each aggregated video feature representation in the plurality of aggregated video feature representations:

[0063] Extracting an overall distribution environment feature representation of an aggregated video feature representation, and determining a corresponding overall video feature representation based on the overall distribution environment feature representation and the aggregated video feature representation;

[0064] Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation;

[0065] Performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation;

[0066] The hole filtered video feature representation and the original video feature representation are aggregated to form a corresponding deep aggregated video feature representation, and the deep aggregated video feature representation is filtered and interpolated to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed.

[0067] The object analysis method and system based on video processing provided by the embodiment of the present invention can first separate the monitoring video frame sequence of the object to be processed and the monitoring audio frame sequence of the object to be processed; perform key information mining operation on the monitoring video frame sequence of the object to be processed to output the original video feature representation; perform multi-stage feature aggregation operation on the original supplementary feature representation and the original video feature representation corresponding to the monitoring audio frame sequence of the object to be processed to obtain multiple aggregated video feature representations; perform feature merging operation on multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyze the important object monitoring video frame subsequence; and use the important object monitoring video frame subsequence as the representative video of the object to be processed. Based on the above content, before determining the important object monitoring video frame subsequence, the video frame and the audio frame in the video will be separated first, so that the corresponding feature representation can be mined respectively, and then through the multi-stage feature aggregation operation, the obtained merged video feature representation can be made more reliable and richer in information, so that the determination accuracy of the important object monitoring video frame subsequence can be improved, thereby improving the reliability of object analysis and improving the deficiencies in the prior art.

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A structural block diagram of an object analysis platform based on video processing provided in an embodiment of the present invention.

[0070] Figure 2 A schematic flow chart of the steps of the object analysis method based on video processing provided in an embodiment of the present invention.

[0071] Figure 3 A schematic diagram of various modules included in the video processing-based object analysis system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0073] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] like Figure 1 As shown, an embodiment of the present invention provides an object analysis platform based on video processing, wherein the object analysis platform based on video processing may include a memory and a processor.

[0075] In detail, the memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, they can be electrically connected to each other through one or more communication buses or signal lines. The memory can store at least one software function module (computer program) that can exist in the form of software or firmware. The processor can be used to execute the executable computer program stored in the memory, thereby realizing the object analysis method based on video processing provided by the embodiment of the present invention (as described later).

[0076] Optionally, in some embodiments, the memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0077] Optionally, in some embodiments, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0078] and, Figure 1 The structure shown is for illustration only. The object analysis platform based on video processing may also include Figure 1 More or fewer components as shown, or with Figure 1 The various configurations shown may, for example, include a communication unit for exchanging information with other devices.

[0079] Optionally, in some implementations, the video processing-based object analysis platform may be a server with data processing capabilities.

[0080] Combination Figure 2 The embodiment of the present invention further provides a method for object analysis based on video processing, which can be applied to the above-mentioned object analysis platform based on video processing. The method steps defined in the process related to the method for object analysis based on video processing can be implemented by the object analysis platform based on video processing. Figure 2 The specific process shown is explained in detail.

[0081] Step S110 : separating the object monitoring video to be processed and the object monitoring audio frame sequence to be processed from the object monitoring video to be processed.

[0082] In an embodiment of the present invention, the object analysis platform based on video processing can separate the monitoring video of the object to be processed into a monitoring video frame sequence of the object to be processed and a monitoring audio frame sequence of the object to be processed in the monitoring video of the object to be processed. The monitoring video of the object to be processed is formed based on video monitoring of the object to be processed.

[0083] Step S120, performing a key information mining operation on the surveillance video frame sequence of the object to be processed to output a corresponding original video feature representation.

[0084] In an embodiment of the present invention, the object analysis platform based on video processing can perform key information mining operations on the monitoring video frame sequence of the object to be processed to output the corresponding original video feature representation, that is, to realize feature mining of the monitoring video frame sequence of the object to be processed. For example, it can be mapped to the feature space first, and then a series of processing such as filtering, linear mapping, etc. can be performed.

[0085] Step S130, performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations.

[0086] In an embodiment of the present invention, the video processing-based object analysis platform may perform a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed (which may be obtained by a key information mining operation) and the original video feature representation to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations.

[0087] Step S140, performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed based on the original video feature representation and the merged video feature representation.

[0088] In an embodiment of the present invention, the video processing-based object analysis platform can perform a feature merging operation on the multiple aggregated video feature representations formed to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyze the important object monitoring video frame subsequences included in the object monitoring video frame sequence to be processed.

[0089] Step S150: taking the important object monitoring video frame subsequence as the representative video of the object to be processed.

[0090] In the embodiment of the present invention, the video processing-based object analysis platform may use the important object monitoring video frame subsequence as a representative video of the object to be processed.

[0091] Based on the foregoing content, before determining the important object monitoring video frame subsequence, the video frames and audio frames in the video will be separated first, so that the corresponding feature representations can be respectively mined, and then through multi-stage feature aggregation operations, the obtained merged video feature representation can be made more reliable and informative. Therefore, the determination accuracy of the important object monitoring video frame subsequence can be improved, thereby improving the reliability of object analysis and improving the deficiencies in the prior art.

[0092] Optionally, in some implementations, the step of performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations may further include the following details:

[0093] For the first stage of feature aggregation operation (that is, the first stage, i.e. the first time the feature aggregation operation is performed), it includes:

[0094] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed, i.e. mapping it to a feature space, etc., as in the aforementioned step S120, to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0095] Adding the original supplementary feature representation to the original video feature representation to form an intermediate feature representation corresponding to the original video feature representation;

[0096] Performing an aggregation operation, such as concatenation or superposition, on the intermediate feature representation and the original supplementary feature representation, and performing a key information mining operation on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation of the first stage;

[0097] For each stage of feature aggregation operation other than the first stage (i.e., each stage from the second stage to the end), it includes:

[0098] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0099] Adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation;

[0100] An aggregation operation is performed on the intermediate feature representation and the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, and a key information mining operation is performed on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation in the current stage.

[0101] Optionally, in some implementations, the step of performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed may further include the following details:

[0102] Performing a feature space mapping operation on the monitored audio frame sequence of the object to be processed to form a corresponding feature space mapping feature representation, and performing a row and column distribution adjustment operation on the feature representation parameters of the feature space mapping feature representation to form a first adjusted feature representation, that is, the row parameters of the feature representation parameters in the feature space mapping feature representation can be used as the column parameters in the first adjusted feature representation, and the column parameters of the feature representation parameters in the feature space mapping feature representation can be used as the row parameters in the first adjusted feature representation;

[0103] Performing a filtering operation on the first adjusted feature representation to form a filtered feature representation, wherein a filtering operator of the filtering operation may be determined during an optimization process of a corresponding neural network;

[0104] Performing a row-column distribution adjustment operation of feature representation parameters on the filtered feature representation, as described above, to form a corresponding second adjusted feature representation;

[0105] The second adjusted feature representation is subjected to a filtering operation, as described above, to form an original supplementary feature representation corresponding to the monitoring audio frame sequence of the object to be processed.

[0106] Optionally, in some implementations, the step of adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation may further include the following details:

[0107] Performing a row and column distribution adjustment operation on the aggregated video feature representation corresponding to the feature aggregation operation of the previous stage to form an adjusted aggregated video feature representation, that is, the row parameters in the aggregated video feature representation can be used as the column parameters of the adjusted aggregated video feature table, and the column parameters in the aggregated video feature representation can be used as the row parameters of the adjusted aggregated video feature table to achieve the exchange of rows and columns;

[0108] Performing an aggregation operation on the adjusted aggregated video feature representation and the original supplementary feature representation to form a corresponding optimized supplementary feature representation;

[0109] An aggregation operation is performed on the adjusted aggregated video feature representation and the optimized supplementary feature representation to form a corresponding intermediate feature representation.

[0110] Optionally, in some implementations, the step of performing an aggregation operation on the adjusted aggregated video feature representation and the original supplementary feature representation to form a corresponding optimized supplementary feature representation may further include the following details:

[0111] Performing a splicing operation on the adjusted aggregated video feature representation and the original supplementary feature representation, and performing a linear integration operation on the feature representation formed by the splicing operation to form a corresponding linear integrated first feature representation, wherein the linear integration operation can be implemented based on a multi-layer perceptron (MLP) layer;

[0112] The linear integrated first feature representation is subjected to parameter distribution adjustment and filtering operations to form a corresponding optimized linear integrated first feature representation, that is, the linear integrated first feature representation may first be subjected to a row and column distribution adjustment operation of feature representation parameters, and then a filtering operation is performed on the result of the operation to form an optimized linear integrated first feature representation;

[0113] A superposition operation is performed on the original supplementary feature representation and the optimized linearly integrated first feature representation, and a filtering operation is performed on the feature representation formed by the superposition operation to form a corresponding optimized supplementary feature representation.

[0114] Optionally, in some implementations, the step of performing an aggregation operation on the adjusted aggregated video feature representation and the optimized supplemental feature representation to form a corresponding intermediate feature representation may further include the following details:

[0115] Performing a splicing operation on the adjusted aggregated video feature representation and the optimized supplementary feature representation, and performing a linear integration operation on the feature representation formed by the splicing operation to form a corresponding linear integrated second feature representation, wherein the linear integration operation can be implemented based on a multi-layer perceptron (MLP) layer;

[0116] The linear integrated second feature representation is subjected to parameter distribution adjustment and filtering operations to form a corresponding optimized linear integrated second feature representation, that is, the linear integrated second feature representation may first be subjected to a row and column distribution adjustment operation of feature representation parameters, and then a filtering operation is performed on the result of the operation to form an optimized linear integrated second feature representation;

[0117] A superposition operation is performed on the optimized supplementary feature representation and the optimized linearly integrated second feature representation, and a filtering operation is performed on the feature representation formed by the superposition operation to form a corresponding intermediate feature table.

[0118] Optionally, in some implementations, the step of performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed based on the original video feature representation and the merged video feature representation, may further include the following details:

[0119] For each aggregated video feature representation in the plurality of aggregated video feature representations:

[0120] Extracting an overall distribution environment feature representation of an aggregated video feature representation (for example, during the processing, for each receptive field, the receptive field can be weightedly superimposed with adjacent receptive fields to achieve fusion of environmental parameters, thereby increasing the receptive field, making the overall distribution environment feature representation more informative), and, based on the overall distribution environment feature representation and the one aggregated video feature representation, determining a corresponding overall video feature representation, for example, the overall distribution environment feature representation and the one aggregated video feature representation can be aggregated to form a corresponding overall video feature representation;

[0121] Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, for example, the size update of the feature representation can be achieved by interpolation, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation;

[0122] Performing a dilated convolution operation on the merged video feature representation to form a dilated convolution video feature representation corresponding to the merged video feature representation;

[0123] The hole filtered video feature representation and the original video feature representation are aggregated to form a corresponding deep aggregated video feature representation, and the deep aggregated video feature representation is filtered and interpolated to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed.

[0124] Optionally, in some implementations, the step of performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation may further include the following details:

[0125] For each of the multiple hole parameters configured (such as dilation rate), performing a hole filtering operation on the merged video feature representation based on the hole parameter to form a local hole filtering video feature representation corresponding to the hole parameter, for example, for hole parameter 1, a hole filtering operation can be performed on the merged video feature representation based on the hole parameter 1 to form a local hole filtering video feature representation 1 corresponding to the hole parameter 1, for hole parameter 2, a hole filtering operation can be performed on the merged video feature representation based on the hole parameter 2 to form a local hole filtering video feature representation 2 corresponding to the hole parameter 2, for hole parameter 3, a hole filtering operation can be performed on the merged video feature representation based on the hole parameter 3 to form a local hole filtering video feature representation 3 corresponding to the hole parameter 3, for hole parameter 4, a hole filtering operation can be performed on the merged video feature representation based on the hole parameter 4 to form a local hole filtering video feature representation 4 corresponding to the hole parameter 4, and for hole parameter 5, a hole filtering operation can be performed on the merged video feature representation based on the hole parameter 5 to form a local hole filtering video feature representation 5 corresponding to the hole parameter 5;

[0126] An aggregation operation is performed on a plurality of local hole filter video feature representations corresponding to the plurality of hole parameters to form a hole filter video feature representation corresponding to the merged video feature representation.

[0127] Combination Figure 3 The embodiment of the present invention further provides an object analysis system based on video processing, which can be applied to the above-mentioned object analysis platform based on video processing. The object analysis system based on video processing may include the following functional modules:

[0128] A monitoring video separation module, used for separating a monitoring video of an object to be processed into a monitoring video frame sequence of the object to be processed and a monitoring audio frame sequence of the object to be processed in the monitoring video of the object to be processed, wherein the monitoring video of the object to be processed is formed based on video monitoring of the object to be processed;

[0129] A key information mining module is used to perform a key information mining operation on the monitoring video frame sequence of the object to be processed to output a corresponding original video feature representation;

[0130] A feature aggregation module, used for performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation, so as to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations;

[0131] An important video frame extraction module is used to perform a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyze the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed;

[0132] The object representative video determination module is used to use the important object monitoring video frame subsequence as the representative video of the object to be processed.

[0133] Optionally, in some implementations, the feature aggregation module is specifically used to:

[0134] The feature aggregation operations in the first stage include:

[0135] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0136] Adding the original supplementary feature representation to the original video feature representation to form an intermediate feature representation corresponding to the original video feature representation;

[0137] Performing an aggregation operation on the intermediate feature representation and the original supplementary feature representation, and performing a key information mining operation on the feature representation formed by the aggregation, so as to form an aggregated video feature representation corresponding to the feature aggregation operation of the first stage;

[0138] For each stage of feature aggregation operation except the first stage, the feature aggregation operation includes:

[0139] Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed;

[0140] Adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation;

[0141] An aggregation operation is performed on the intermediate feature representation and the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, and a key information mining operation is performed on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation in the current stage.

[0142] Optionally, in some implementations, the important video frame extraction module is specifically used to:

[0143] For each aggregated video feature representation in the plurality of aggregated video feature representations:

[0144] Extracting an overall distribution environment feature representation of an aggregated video feature representation, and determining a corresponding overall video feature representation based on the overall distribution environment feature representation and the aggregated video feature representation;

[0145] Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation;

[0146] Performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation;

[0147] The hole filtered video feature representation and the original video feature representation are aggregated to form a corresponding deep aggregated video feature representation, and the deep aggregated video feature representation is filtered and interpolated to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed.

[0148] In summary, the object analysis method and system based on video processing provided by the present invention can first separate the monitoring video frame sequence of the object to be processed and the monitoring audio frame sequence of the object to be processed; perform key information mining operation on the monitoring video frame sequence of the object to be processed to output the original video feature representation; perform multi-stage feature aggregation operation on the original supplementary feature representation and the original video feature representation corresponding to the monitoring audio frame sequence of the object to be processed to obtain multiple aggregated video feature representations; perform feature merging operation on multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyze the important object monitoring video frame subsequence; and use the important object monitoring video frame subsequence as the representative video of the object to be processed. Based on the above content, before determining the important object monitoring video frame subsequence, the video frame and audio frame in the video will be separated first, so that the corresponding feature representation can be mined respectively, and then through the multi-stage feature aggregation operation, the obtained merged video feature representation can be made more reliable and richer in information, so that the determination accuracy of the important object monitoring video frame subsequence can be improved, thereby improving the reliability of object analysis and improving the deficiencies in the prior art.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for object analysis based on video processing, characterized in that: include: For a surveillance video of an object to be processed, separating a surveillance video frame sequence of the object to be processed and a surveillance audio frame sequence of the object to be processed in the surveillance video of the object to be processed, wherein the surveillance video of the object to be processed is formed based on video surveillance of the object to be processed; Performing a key information mining operation on the surveillance video frame sequence of the object to be processed to output a corresponding original video feature representation; Performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations; Performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed based on the original video feature representation and the merged video feature representation; Using the important object monitoring video frame subsequence as the representative video of the object to be processed; The step of performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation to obtain the aggregated video feature representations formed by the multi-stage feature aggregation operations respectively includes: The feature aggregation operations in the first stage include: Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed; Adding the original supplementary feature representation to the original video feature representation to form an intermediate feature representation corresponding to the original video feature representation; Performing an aggregation operation on the intermediate feature representation and the original supplementary feature representation, and performing a key information mining operation on the feature representation formed by the aggregation, so as to form an aggregated video feature representation corresponding to the feature aggregation operation of the first stage; For each stage of feature aggregation operation except the first stage, the feature aggregation operation includes: Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed; Adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation; Performing an aggregation operation on the intermediate feature representation and the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, and performing a key information mining operation on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation in the current stage; The step of performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed according to the original video feature representation and the merged video feature representation, comprises: For each aggregated video feature representation in the plurality of aggregated video feature representations: Extracting an overall distribution environment feature representation of an aggregated video feature representation, and determining a corresponding overall video feature representation based on the overall distribution environment feature representation and the aggregated video feature representation; Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation; Performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation; Aggregating the hole-filtered video feature representation and the original video feature representation to form a corresponding deep aggregated video feature representation, and filtering and interpolating the deep aggregated video feature representation to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed; The step of performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation includes: For each of the configured multiple hole parameters, performing a hole filtering operation on the merged video feature representation based on the hole parameter to form a local hole filtering video feature representation corresponding to the hole parameter; An aggregation operation is performed on a plurality of local hole filter video feature representations corresponding to the plurality of hole parameters to form a hole filter video feature representation corresponding to the merged video feature representation.

2. The object analysis method based on video processing according to claim 1, characterized in that: The step of performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed includes: Performing a feature space mapping operation on the monitored audio frame sequence of the object to be processed to form a corresponding feature space mapping feature representation, and performing a row and column distribution adjustment operation of feature representation parameters on the feature space mapping feature representation to form a first adjusted feature representation; performing a filtering operation on the first adjusted feature representation to form a filtered feature representation; Performing a row-column distribution adjustment operation of feature representation parameters on the filtered feature representation to form a corresponding second adjusted feature representation; The second adjusted feature representation is filtered to form an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed.

3. The object analysis method based on video processing according to claim 1, characterized in that: The step of adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation includes: Performing a row-column distribution adjustment operation on the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, so as to form an adjusted aggregated video feature representation; Performing an aggregation operation on the adjusted aggregated video feature representation and the original supplementary feature representation to form a corresponding optimized supplementary feature representation; An aggregation operation is performed on the adjusted aggregated video feature representation and the optimized supplementary feature representation to form a corresponding intermediate feature representation.

4. The object analysis method based on video processing according to claim 3, characterized in that: The step of performing an aggregation operation on the adjusted aggregated video feature representation and the original supplementary feature representation to form a corresponding optimized supplementary feature representation comprises: Performing a splicing operation on the adjusted aggregated video feature representation and the original supplementary feature representation, and performing a linear integration operation on the feature representation formed by the splicing operation to form a corresponding linearly integrated first feature representation; Performing parameter distribution adjustment and filtering operations on the linearly integrated first feature representation to form a corresponding optimized linearly integrated first feature representation; A superposition operation is performed on the original supplementary feature representation and the optimized linearly integrated first feature representation, and a filtering operation is performed on the feature representation formed by the superposition operation to form a corresponding optimized supplementary feature representation.

5. The object analysis method based on video processing according to claim 3, characterized in that: The step of performing an aggregation operation on the adjusted aggregated video feature representation and the optimized supplementary feature representation to form a corresponding intermediate feature representation includes: Performing a splicing operation on the adjusted aggregated video feature representation and the optimized supplemental feature representation, and performing a linear integration operation on the feature representation formed by the splicing operation to form a corresponding linearly integrated second feature representation; Performing parameter distribution adjustment and filtering operations on the linearly integrated second feature representation to form a corresponding optimized linearly integrated second feature representation; A superposition operation is performed on the optimized supplementary feature representation and the optimized linearly integrated second feature representation, and a filtering operation is performed on the feature representation formed by the superposition operation to form a corresponding intermediate feature representation.

6. An object analysis system based on video processing, characterized in that: include: A monitoring video separation module, used for separating a monitoring video of an object to be processed into a monitoring video frame sequence of the object to be processed and a monitoring audio frame sequence of the object to be processed in the monitoring video of the object to be processed, wherein the monitoring video of the object to be processed is formed based on video monitoring of the object to be processed; A key information mining module is used to perform a key information mining operation on the monitoring video frame sequence of the object to be processed to output a corresponding original video feature representation; A feature aggregation module, used for performing a multi-stage feature aggregation operation on the original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed and the original video feature representation, so as to obtain an aggregated video feature representation formed by each of the multi-stage feature aggregation operations; An important video frame extraction module is used to perform a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and, based on the original video feature representation and the merged video feature representation, analyze the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed; An object representative video determination module, used for taking the important object monitoring video frame subsequence as the representative video of the object to be processed; The feature aggregation module is specifically used for: The feature aggregation operations in the first stage include: Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed; Adding the original supplementary feature representation to the original video feature representation to form an intermediate feature representation corresponding to the original video feature representation; Performing an aggregation operation on the intermediate feature representation and the original supplementary feature representation, and performing a key information mining operation on the feature representation formed by the aggregation, so as to form an aggregated video feature representation corresponding to the feature aggregation operation of the first stage; For each stage of feature aggregation operation except the first stage, the feature aggregation operation includes: Performing a key information mining operation on the monitored audio frame sequence of the object to be processed to output an original supplementary feature representation corresponding to the monitored audio frame sequence of the object to be processed; Adding the original supplementary feature representation to the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage to form a corresponding intermediate feature representation; Performing an aggregation operation on the intermediate feature representation and the aggregated video feature representation corresponding to the feature aggregation operation in the previous stage, and performing a key information mining operation on the aggregated feature representation to form an aggregated video feature representation corresponding to the feature aggregation operation in the current stage; The step of performing a feature merging operation on the formed multiple aggregated video feature representations to form a merged video feature representation, and analyzing the important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed based on the original video feature representation and the merged video feature representation, comprises: For each aggregated video feature representation in the plurality of aggregated video feature representations: Extracting an overall distribution environment feature representation of an aggregated video feature representation, and determining a corresponding overall video feature representation based on the overall distribution environment feature representation and the aggregated video feature representation; Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation; Performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation; Aggregating the hole-filtered video feature representation and the original video feature representation to form a corresponding deep aggregated video feature representation, and filtering and interpolating the deep aggregated video feature representation to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed; The step of performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation includes: For each of the configured multiple hole parameters, performing a hole filtering operation on the merged video feature representation based on the hole parameter to form a local hole filtering video feature representation corresponding to the hole parameter; An aggregation operation is performed on a plurality of local hole filter video feature representations corresponding to the plurality of hole parameters to form a hole filter video feature representation corresponding to the merged video feature representation.

7. The object analysis system based on video processing according to claim 6, characterized in that: The important video frame extraction module is specifically used for: For each aggregated video feature representation in the plurality of aggregated video feature representations: Extracting an overall distribution environment feature representation of an aggregated video feature representation, and determining a corresponding overall video feature representation based on the overall distribution environment feature representation and the aggregated video feature representation; Performing a size update operation on the determined multiple overall video feature representations to form multiple overall video feature representations of the same size, and performing an aggregation operation on the multiple overall video feature representations of the same size to form a corresponding merged video feature representation; Performing a hole filtering operation on the merged video feature representation to form a hole filtered video feature representation corresponding to the merged video feature representation; The hole filtered video feature representation and the original video feature representation are aggregated to form a corresponding deep aggregated video feature representation, and the deep aggregated video feature representation is filtered and interpolated to form an important object monitoring video frame subsequence included in the object monitoring video frame sequence to be processed.

Images