Behavior recognition method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a behavior recognition method, device, storage medium and electronic device, wherein the method comprises: extracting features from an image sequence containing an object to be processed to obtain image sequence features of the image sequence, wherein the image sequence features are used to characterize the motion information of the object to be processed in the image sequence; extracting features from a target image in the image sequence to obtain target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize the position information of the object to be processed in the target image; determining features to be processed based on the image sequence features and the target image features; and recognizing the behavior of the object to be processed in the image sequence based on the features to be processed. The present invention solves the problem of low accuracy of behavior recognition in the related art, and achieves the effect of improving the accuracy of behavior recognition.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular, to a behavior recognition method, device, storage medium and electronic device. Background Art

[0002] With the development of computer technology, the Internet and artificial intelligence, the scale of video images has grown exponentially. How to make machines "recognize" videos in the way humans think and realize automatic understanding of videos in different scenarios has always been an urgent problem to be solved in the field of machine vision. Taking the analysis and recognition of abnormal human behavior as an example, the abnormal human behavior analysis technology has a wide range of applications in public security, smart home and other fields. The application occasions include homes, ATM machines, banks, stores, parking lots, airports, government buildings, military bases, docks, etc. For example, in smart homes, it is used to monitor the falls and coma of elderly people living alone, and to monitor abnormal situations such as falling to the ground, fighting, abnormal crowd gathering, riots, etc. in crowded places. However, the existing abnormal human behavior analysis technology is not clear enough to distinguish abnormal behaviors, the accuracy of abnormal behavior recognition is not high, and the reliability of abnormal behavior alarm is not strong. Therefore, a new abnormal human behavior recognition technology is needed to solve the above problems.

[0003] Currently, no effective solution has been proposed to address the problem of low accuracy in behavior recognition in related technologies. Summary of the invention

[0004] The embodiments of the present invention provide a behavior recognition method, device, storage medium and electronic device to at least solve the problem of low accuracy of behavior recognition existing in the related art.

[0005] According to an embodiment of the present invention, a behavior recognition method is provided, comprising: extracting features from an image sequence containing an object to be processed to obtain image sequence features of the image sequence, wherein the image sequence features are used to characterize the motion information of the object to be processed in the image sequence; extracting features from a target image in the image sequence to obtain target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize the position information of the object to be processed in the target image; determining features to be processed based on the image sequence features and the target image features; and recognizing the behavior of the object to be processed in the image sequence based on the features to be processed. In an exemplary embodiment, before extracting features from an image sequence containing an object to be processed to obtain image sequence features of the image sequence, the method further comprises: acquiring a surveillance video containing the object to be processed; and performing frame extraction processing on the surveillance video to obtain the image sequence.

[0006] In an exemplary embodiment, the target images include at least two; determining the features to be processed based on the image sequence features and the target image features includes: determining reference image features based on the target image features of each target image included in the at least two target images; determining the features to be processed based on the image sequence features and the reference image features.

[0007] In an exemplary embodiment, determining the reference image feature based on the target image features of each target image included in the at least two target images includes: obtaining an average value of the target image features of each of the target images to obtain the reference image feature.

[0008] In an exemplary embodiment, based on the image sequence features and the target image features, determining the features to be processed includes: performing splicing and fusion processing on the image sequence features and the target image features to generate a first feature; and performing superposition processing on the first feature to generate the features to be processed.

[0009] In an exemplary embodiment, the image sequence feature and the target image feature are spliced ​​and fused to generate a first feature, including: setting a first parameter for identifying the number of frames of the image sequence included in the image sequence feature to 1 to obtain an image sequence adjustment feature; splicing the image sequence adjustment feature with the target image feature in a target channel dimension to obtain a spliced ​​feature; and performing feature fusion processing on the spliced ​​feature to generate the first feature.

[0010] In an exemplary embodiment, superimposing the first feature to generate the feature to be processed includes: adjusting the dimension of the first feature to obtain the second feature; performing inner product on the second feature on each channel to obtain the target type feature; processing the target type feature using softmax to obtain an attention mechanism graph; performing matrix multiplication processing on the attention mechanism graph and the second feature to obtain a third feature; adjusting the dimension of the third feature to obtain a fourth feature having the same dimension as the first feature; and superimposing the fourth feature with the first feature to generate the feature to be processed.

[0011] In an exemplary embodiment, based on the features to be processed, identifying the behavior of the object to be processed in the image sequence includes: performing regression processing on the features to be processed to determine the coordinate frame information and the classification score of the object to be processed; and determining the behavior of the object to be processed based on the coordinate frame information and the classification score.

[0012] In an exemplary embodiment, performing regression processing on the feature to be processed to determine the coordinate frame information and classification score of the object to be processed includes: inputting the feature to be processed into a convolutional layer to use the convolutional layer to regress the coordinate frame information and classification score of the object to be processed.

[0013] In an exemplary embodiment, using the convolution layer to regress the coordinate frame information of the object to be processed includes: using the supervisory frame coordinates pre-configured in the convolution layer to regress the coordinate frame information of the object to be processed, wherein the supervisory frame coordinates are generated in the following manner: based on the first area and the second area of ​​the annotation box of the object to be processed respectively contained in two consecutive frames of images included in the image sequence, obtaining a first category score and a second category score, wherein the first category score and the second category score are respectively used to indicate the classification scores of the annotation box of the object to be processed corresponding to the first area and the second area; determining the connection score of the two consecutive frames of images based on the first category score, the second category score, and the overlap IOU of the first area and the second area; and deriving the coordinate frame information based on a plurality of the connection scores using a predetermined algorithm.

[0014] In an exemplary embodiment, determining the connection score of the two consecutive frames of images based on the first category score, the second category score, and the overlap IOU of the first region and the second region includes: determining the connection score S of the two consecutive frames of images by the following formula: c (R t ,R t+1 ): S c (R t ,R t+1 )=ψ(x)·[S c (R t )+S c (R t+1 )]+α·S c (R t )·S c (R t+1 ), where S c (R t ) and S c (R t+1 ) is the first region R t and the second region R t+1 The first category score and the second category score, x is the IOU of the two regions, α is a scalar, when there is region overlap, ψ(x) is equal to 1, otherwise ψ(x) is equal to 0.

[0015] According to another embodiment of the present invention, there is also provided a behavior recognition device, comprising: a first acquisition module, used to perform feature extraction on an image sequence containing an object to be processed, and obtain image sequence features of the image sequence, wherein the image sequence features are used to characterize motion information of the object to be processed in the image sequence; an extraction module, used to perform feature extraction on a target image in the image sequence, and obtain target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize position information of the object to be processed in the target image; a determination module, used to determine features to be processed based on the image sequence features and the target image features; and an identification module, used to identify the behavior of the object to be processed in the image sequence based on the features to be processed.

[0016] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0017] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0018] Through the present invention, by extracting features from an image sequence containing an object to be processed, image sequence features are obtained, wherein the image sequence features are used to characterize the motion information of the object to be processed in the image sequence; feature extraction is performed on a target image in the image sequence to obtain a target image feature of the target image, wherein the target image is an image selected from the image sequence, and the target image feature is used to characterize the position information of the object to be processed in the target image; then based on the image sequence features and the target image features, features to be processed are determined; and then based on the features to be processed, the behavior of the object to be processed in the image sequence is identified. The purpose of identifying the behavior of the object to be processed contained in the image sequence by integrating the image sequence features and the target image features is achieved, and the accuracy of behavior recognition is improved. Therefore, the problem of low accuracy of behavior recognition existing in the related art is solved, and the effect of improving the accuracy of behavior recognition is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of the hardware structure of a mobile terminal of the behavior recognition method according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0021] Figure 3 is a flow chart of a behavior recognition method according to a specific embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a feature fusion process according to a specific embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of a flow chart of attention feature generation according to a specific embodiment of the present invention;

[0024] Figure 6 4 is a structural block diagram of a behavior recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a block diagram of the hardware structure of a mobile terminal of the behavior recognition method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the behavior recognition method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a behavior recognition method is provided. Figure 2 is a flow chart of a behavior recognition method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0031] Step S202, extracting features from an image sequence containing an object to be processed to obtain image sequence features of the image sequence, wherein the image sequence features are used to characterize motion information of the object to be processed in the image sequence;

[0032] Step S204, extracting features of the target image in the image sequence to obtain target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize position information of the object to be processed in the target image;

[0033] Step S206, determining features to be processed based on the image sequence features and the target image features;

[0034] Step S208: identifying the behavior of the object to be processed in the image sequence based on the feature to be processed.

[0035] Through the above steps, the image sequence features are obtained by extracting features from the image sequence containing the object to be processed, wherein the image sequence features are used to characterize the motion information of the object to be processed in the image sequence; the target image in the image sequence is subjected to feature extraction to obtain the target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize the position information of the object to be processed in the target image; then based on the image sequence features and the target image features, the features to be processed are determined; and then based on the features to be processed, the behavior of the object to be processed in the image sequence is identified. The purpose of identifying the behavior of the object to be processed contained in the image sequence by integrating the image sequence features and the target image features is achieved, and the accuracy of behavior recognition is improved. Therefore, the problem of low accuracy of behavior recognition existing in the related art is solved, and the effect of improving the accuracy of behavior recognition is achieved.

[0036] The execution subject of the above steps may be an image processing system, or a device with image analysis and processing capabilities, or an integrated image processing device, or a video analysis device, or a server with similar processing capabilities, or a processor with human-computer interaction capabilities configured on a storage device, or a processing device or processing unit with similar processing capabilities, but not limited thereto. The following takes the image processing device performing the above operations as an example (only an exemplary description, in actual operation, other devices or modules may also be used to perform the above operations) for description:

[0037] In the above embodiment, the image processing device extracts features from an image sequence containing an object to be processed to obtain image sequence features, wherein the image sequence features are used to characterize the motion information of the object to be processed in the image sequence, for example, a 3D convolutional neural network is used to extract image sequence features of the image sequence containing the object to be processed; extracts features from a target image in the image sequence to obtain target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize the position information of the object to be processed in the target image, for example, a 2D convolutional neural network is used to extract features of the target image included in the image sequence to obtain target image features; then based on the image sequence features and the target image features, features to be processed are determined; then based on the features to be processed, the behavior of the object to be processed in the image sequence is identified, for example, the image sequence features and the target image features are processed to determine the features to be processed. The purpose of identifying the behavior of the object to be processed contained in the image sequence by integrating the image sequence features and the target image features is achieved, and the accuracy of behavior recognition is improved. Therefore, the problem of low accuracy of behavior recognition existing in the related art is solved, and the effect of improving the accuracy of behavior recognition is achieved.

[0038] In an optional embodiment, before extracting features from an image sequence containing an object to be processed and obtaining image sequence features of the image sequence, the method further includes: obtaining a surveillance video containing the object to be processed; and performing frame extraction processing on the surveillance video to obtain the image sequence. In this embodiment, the image processing device can obtain real-time surveillance video data through a surveillance device, and perform frame extraction processing on the surveillance video, for example, performing frame skipping sampling on images in the surveillance video to obtain an image sequence. Through this embodiment, the purpose of obtaining an image sequence is achieved.

[0039] In an optional embodiment, the target images include at least two; the determining the feature to be processed based on the image sequence features and the target image features includes: determining the reference image feature based on the target image features of each target image included in the at least two target images; and determining the feature to be processed based on the image sequence features and the reference image features. In this embodiment, the target images selected from the image sequence include at least two, and the reference image feature is determined based on the target image features of each target image included in the at least two target images, so that the target image feature has more reference significance, and then the feature to be processed is determined based on the image sequence features and the reference image features, so that the feature to be processed integrates the information of the image sequence features and the image features, that is, integrates the spatiotemporal dimension information of the image sequence and the spatial dimension information of the target image.

[0040] In an optional embodiment, based on the target image features of each target image included in the at least two target images, determining the reference image feature includes: obtaining an average value of the target image features of each target image to obtain the reference image feature. In this embodiment, the reference image feature can be obtained by obtaining an average value of the target image features of each target image.

[0041] In an optional embodiment, based on the image sequence features and the target image features, determining the feature to be processed includes: splicing and fusing the image sequence features and the target image features to generate a first feature; and superimposing the first feature to generate the feature to be processed. In this embodiment, the image sequence features and the target image features are spliced ​​or fused to generate the first feature. For example, the image sequence features and the target image features are spliced ​​to obtain a spliced ​​feature, and the spliced ​​feature is then subjected to a convolutional layer for feature fusion to generate the first feature, and then the first feature is superimposed to generate the feature to be processed.

[0042] In an optional embodiment, the image sequence feature and the target image feature are spliced ​​and fused to generate a first feature, including: setting a first parameter for identifying the number of frames of the image sequence included in the image sequence feature to 1 to obtain an image sequence adjustment feature; splicing the image sequence adjustment feature with the target image feature in the target channel dimension to obtain a spliced ​​feature; and performing feature fusion processing on the spliced ​​feature to generate the first feature. In this embodiment, the first parameter of the number of frames for identifying the image sequence included in the image sequence feature is set to 1. For example, in practical applications, the image sequence features are extracted using a 3D convolutional neural network, and the target image features are extracted using a 2D convolutional neural network. The first parameter of the 3D convolutional neural network output feature map is set to 1. This is to enable the output of the 3D convolutional neural network to match the 2D convolutional neural network to obtain the image sequence adjustment feature, and then the image sequence adjustment feature and the target image feature are spliced ​​on the target channel dimension to obtain the spliced ​​feature. For example, the scale of the 3D convolutional neural network input is [C*T*H*W], where C is the number of input channels (such as C=3), T is the number of input frames, H and W are the height and width of the input image, respectively, and the input The output scale is [C′*T′*H′*W′], where C′ is the number of output channels, H′ and W′ are the height and width of the output feature map, respectively. In practical applications, H and W can be obtained by downsampling M times (for example, 32 times, or other multiples). In order to make the output of the 3D convolutional network match the 2D convolutional network, T′ is set to 1, that is, the output dimension of the 3D convolutional network is converted from [C′*T′*H′*W′] to [C′*H′*W′], while the input scale of the 2D convolutional neural network is [C*H*W], and the output scale is [Cʺ*Hʺ*Wʺ]. When the features of the 3D convolutional neural network and the features of the 2D convolutional neural network are spliced ​​in the channel dimension, because the output dimensions of H′ and Hʺ, W′ and Wʺ are the same, the output of the spliced ​​feature map is [(C′+ Cʺ)*T′*H′*W′] to obtain the splicing feature; then the splicing feature is subjected to feature fusion processing to generate the first feature. In practical applications, the spliced ​​feature is subjected to feature fusion through a convolution layer to generate a fusion feature. Through this embodiment, the purpose of effectively fusing the image sequence features of the multi-frame image and the target image features of the single-frame image is achieved, that is, the purpose of effectively fusing the spatiotemporal dimension information of the multi-frame image and the spatial dimension information of the single-frame image is achieved, and the dependency between channels is utilized to the greatest extent to improve the accuracy of behavior recognition.

[0043] In an optional embodiment, the first feature is superimposed to generate the feature to be processed, including: adjusting the dimension of the first feature to obtain the second feature; performing an inner product on the second feature on each channel to obtain a target type feature; processing the target type feature using softmax to obtain an attention mechanism graph; performing matrix multiplication on the attention mechanism graph and the second feature to obtain a third feature; adjusting the dimension of the third feature to obtain a fourth feature having the same dimension as the first feature; superimposing the fourth feature with the first feature to generate the feature to be processed. In this embodiment, the first feature is superimposed to generate the feature to be processed, including: adjusting the dimension of the first feature matrix to obtain the second feature, for example, the first feature output dimension is [Cʺ′*Hʺ′*Wʺ′], performing matrix dimension transformation on the first feature to generate a second feature having a dimension of [Cʺ′*N]; then performing an inner product on the second feature on each channel to obtain the target type feature, that is, generating a Gram matrix G, that is, G= N*N T ; Then use softmax to process the target type feature to obtain an attention mechanism graph, and then perform further matrix multiplication on the attention mechanism graph and the second feature to obtain a third feature, and then perform matrix dimension change on the third feature to obtain a fourth feature with the same dimension as the aforementioned first feature (i.e., the feature obtained after the fusion process), i.e., the dimension of the fourth feature is [Cʺ′*Hʺ′*Wʺ′], and finally, the fourth feature is superimposed on the first feature to generate a feature to be processed, i.e., the first feature of the fusion process is subjected to the attention mechanism to generate an attention feature. Through this embodiment, the purpose of generating an attention feature from the fusion feature through the attention mechanism is achieved.

[0044] In an optional embodiment, based on the feature to be processed, identifying the behavior of the object to be processed in the image sequence includes: regressing the feature to be processed to determine the coordinate frame information and classification score of the object to be processed; and determining the behavior of the object to be processed based on the coordinate frame information and the classification score. In this embodiment, regressing the feature moments to be processed to determine the coordinate frame information and classification score of the object to be processed can be achieved in the following manner: regressing the feature to be processed through a convolutional layer to obtain the coordinate frame information and classification score of each object to be processed, and then determining the behavior of the object to be processed based on the coordinate frame information and the classification score. Through this embodiment, the purpose of more accurately identifying the behavior of the object to be processed is achieved.

[0045] In an optional embodiment, performing regression processing on the features to be processed to determine the coordinate frame information and classification score of the object to be processed includes: inputting the features to be processed into a convolutional layer to use the convolutional layer to regress the coordinate frame information and classification score of the object to be processed, that is, regressing the features to be processed through the convolutional layer to obtain the coordinate frame information and classification score of each object to be processed.

[0046] In an optional embodiment, regressing the coordinate frame information of the object to be processed using the convolution layer includes: regressing the coordinate frame information of the object to be processed using the supervisory frame coordinates pre-configured in the convolution layer, wherein the supervisory frame coordinates are generated in the following manner: obtaining a first category score and a second category score based on the first area and the second area of ​​the annotation frame of the object to be processed respectively contained in two consecutive frames of images included in the image sequence, wherein the first category score and the second category score are used to indicate the classification scores of the annotation frame of the object to be processed corresponding to the first area and the second area respectively; determining the connection score of the two consecutive frames of images based on the first category score, the second category score, and the overlap IOU of the first area and the second area; and obtaining the coordinate frame information based on a predetermined algorithm using a plurality of the connection scores. In this embodiment, the connection score is used to indicate the action category of the object to be processed. In practical applications, after determining the connection score of two consecutive frames of images, the Viterbi algorithm can be used to find the best area for generating the annotation frames of multiple frames of the object to be processed, and the coordinate frame information can be obtained.

[0047] In an optional embodiment, determining the connection score of the two consecutive frames of images based on the first category score, the second category score, and the overlap IOU of the first region and the second region includes: determining the connection score S of the two consecutive frames of images by the following formula: c (R t ,R t+1 ): S c (R t ,R t+1 )=ψ(x)·[S c (R t )+S c (R t+1 )]+α·S c (R t )·S c (R t+1 ), where S c (R t ) and S c (R t+1 ) is the first region R t and the second region R t+1The first category score and the second category score are , x is the IOU of the two regions, α is a scalar, when there is region overlap, ψ(x) is equal to 1, otherwise ψ(x) is equal to 0. In this embodiment, in practical applications, considering that the class scores between two consecutive frames may change dramatically, an additional α·S is introduced c (R t )·S c (R t+1 ) Extending the definition of connection scores can improve the performance of video detection; and then obtaining coordinate frame information based on multiple connection scores using a predetermined algorithm. For example, after calculating all connection scores, the Viterbi algorithm is used to find the best area for generating coordinate frames of multiple frames of objects to be processed. Through this embodiment, based on a new multi-frame annotation frame connection strategy, the purpose of generating more accurate annotation frame information of the object to be processed can be achieved.

[0048] Obviously, the above-described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0049] The present invention will be specifically described below in conjunction with embodiments:

[0050] Figure 3 is a flow chart of a behavior recognition method according to a specific embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0051] S302, input video, for example, obtain real-time video data through monitoring equipment, perform frame skipping sampling on the pictures in the real-time video, and store them in a temporary space, storing a total of N frame picture sequences S (corresponding to the aforementioned image sequence), that is, S = (f i+1 ,f i+2 ,...,f i+N ), when a new image f is obtained i+N+1 When deleting the temporary storage space, i+1 Picture, save i+N+1 Picture, that is, S=(f i+2 ,f i+3 ,...,f i+N+1 );

[0052] S304, the image sequence S is first subjected to spatiotemporal feature extraction through a 3D convolutional network (corresponding to the aforementioned extraction of image sequence features). The input of the 3D convolutional network is a video clip, that is, a series of frames arranged in chronological order. The input scale is [C*T*H*W], and the output scale is [C′*T′*H′*W′], where C=3, T is the number of frames of the input clip, and H and W are the height and width of the input image, respectively. C′ is the number of output channels, and H′ and W′ are the height and width of the output feature map, respectively, which is obtained by downsampling H and W by 32 times. In order to make the output of the 3D convolutional network match that of the 2D convolutional network, T′ (corresponding to the first parameter mentioned above) is set to 1;

[0053] S306, select the middle frame of the sequence S (corresponding to the target image included in the aforementioned image sequence) and perform spatial feature extraction (corresponding to the aforementioned extraction of target image features) through a 2D convolutional neural network. The input of the 2D convolutional network is a picture, or a target picture selected from at least two pictures, or a reference picture obtained by integrating the features of each picture in at least two pictures. The middle frame of the clip is directly selected as the input picture. The input scale is [C*H*W], and the output scale is [Cʺ*Hʺ*Wʺ], where C=3, Cʺ is the number of output channels, Hʺ and Wʺ are the height and width of the output feature map, respectively, which is obtained by downsampling H and W by 32 times;

[0054] S308, concatenate the features of the 3D convolutional network and the features of the 2D convolutional network in the channel dimension. From the above step S304, we know that T'=1, so the output dimension of the 3D convolutional network [C'*T'*H'*W'] is converted to [C'*H'*W'], and then concatenated with the output of the 2D convolutional network [Cʺ*Hʺ*Wʺ]. Since the output dimensions of H' and Hʺ, W' and Wʺ are the same, the output of the concatenated feature map is [(C'+ Cʺ)*T'*H'*W'];

[0055] S310, the concatenated features are fused through a convolutional layer to generate a fused feature B (corresponding to the first feature mentioned above), and its output dimension is [Cʺ′*Hʺ′*Wʺ′];

[0056] S312, the fusion feature B generates the attention feature Fʺ′ (corresponding to the aforementioned feature to be processed) through the attention mechanism;

[0057] S314, finally, the feature Fʺ′ is regressed through the convolution layer to obtain the coordinate frame information and classification score of each person (corresponding to the aforementioned object to be processed);

[0058] The supervisory frame coordinates used in regressing the human frame coordinates are generated by the following steps, assuming that R t and R t+1S is the two labeled box regions of consecutive frames T and T+1, then the connection score of human action category C is defined as: c (R t ,R t+1 )=ψ(x)·[S c (R t )+S c (R t+1 )]+α·S c (R t )·S c (R t+1 ), where S c (R t ) and S c (R t+1 ) is the region R t and R t+1 The class scores of x are the IOUs of the two regions, α is a scalar, and (x>0) is a constraint. When there is region overlap, ψ(x) is equal to 1, otherwise ψ(x) is equal to 0. Considering that the class scores between two consecutive frames may change dramatically, an additional α·S is introduced. c (R t )·S c (R t+1 ) Extending the definition of connection scores can improve the performance of video detection; after calculating all connection scores, the Viterbi algorithm is used to find the best bbox area for generating multi-frame human body boxes.

[0059] S316, output the detection result;

[0060] The following is combined with Figure 4 The feature fusion in the above step S310 is explained. Figure 4 FIG. 1 is a schematic diagram of a feature fusion process according to a specific embodiment of the present invention, and the process includes:

[0061] S402, concatenating the 3D convolutional network features with the 2D convolutional network features to obtain concatenated features (corresponding to the concatenated features);

[0062] S404, input the features obtained after splicing in S402 into the convolution layer;

[0063] S406, the concatenated features are fused through a convolutional layer to generate fused features B;

[0064] Through the above steps S402-S406, fusion features can be generated;

[0065] Combined with Figure 5 The generation of the attention feature in step S312 is explained. Figure 5is a flowchart of attention feature generation according to a specific embodiment of the present invention. Figure 5 As shown, the process includes:

[0066] S502, firstly, reshape the feature matrix B to generate a feature matrix F (corresponding to the second feature) with a dimension of [Cʺ′*N], where N=Hʺ′*Wʺ′;

[0067] S504, then do the inner product of feature F on each channel to generate the Gram matrix G (corresponding to the aforementioned target type feature), that is, N*N T =G;

[0068] S506, then apply softmax to G to generate an attention mechanism graph M;

[0069] S508, further matrix multiplication is performed on the generated attention mechanism graph and F, and the result is reshaped into a feature F′ with the same output dimension as feature B, whose dimension is [Cʺ′*Hʺ′*Wʺ′];

[0070] S510, finally, feature F′ and feature B are superimposed to generate feature Fʺ′ (corresponding to the aforementioned feature to be processed);

[0071] Attention features may be generated through the above steps S502 - S510 .

[0072] Through the above embodiments, based on a new channel feature fusion method, the spatial dimension information of a single-frame image and the temporal dimension information of multiple frames are effectively fused to maximize the use of the dependencies between channels; a real-time single-stage network structure is used to locate abnormal human behavior in time and space, and end-to-end training can be achieved; at the same time, based on a new multi-frame annotation box connection strategy, the purpose of generating more accurate annotation box information of the target object can be achieved, thereby achieving the effect of improving the accuracy of target behavior recognition.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0074] In this embodiment, a behavior recognition device is also provided. Figure 6 is a structural block diagram of a behavior recognition device according to an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0075] A first acquisition module 602 is used to extract features from an image sequence containing an object to be processed, and obtain image sequence features of the image sequence, wherein the image sequence features are used to represent motion information of the object to be processed in the image sequence;

[0076] An extraction module 604 is used to extract features of a target image in the image sequence to obtain target image features of the target image, wherein the target image is an image selected from the image sequence, and the target image features are used to characterize position information of the object to be processed in the target image;

[0077] A determination module 606, configured to determine a feature to be processed based on the image sequence feature and the target image feature;

[0078] The recognition module 608 is used to recognize the behavior of the object to be processed in the image sequence based on the feature to be processed.

[0079] In an optional embodiment, the above-mentioned device also includes: a second acquisition module, used to extract features of an image sequence containing the object to be processed, and before obtaining the image sequence features of the image sequence, obtain a surveillance video containing the object to be processed; a third acquisition module, used to perform frame extraction processing on the surveillance video to obtain the image sequence.

[0080] In an optional embodiment, the above-mentioned target image includes at least two; the above-mentioned determination module 606 includes: a first determination submodule, used to determine the reference image features based on the target image features of each target image included in the at least two target images; a second determination submodule, used to determine the features to be processed based on the image sequence features and the reference image features.

[0081] In an optional embodiment, the first determining submodule includes: an acquiring unit, configured to acquire an average value of the target image features of each of the target images to obtain the reference image features.

[0082] In an optional embodiment, the above-mentioned determination module 606 includes: a first generation submodule, used to splice and fuse the image sequence features and the target image features to generate a first feature; a second generation submodule, used to superimpose the first feature to generate the feature to be processed.

[0083] In an optional embodiment, the above-mentioned first generation submodule includes: a setting unit, used to set the first parameter included in the image sequence feature and used to identify the number of frames of the image sequence to 1, so as to obtain an image sequence adjustment feature; a splicing unit, used to splice the image sequence adjustment feature with the target image feature in the target channel dimension to obtain a spliced ​​feature; and a fusion unit, used to perform feature fusion processing on the spliced ​​feature to generate the first feature.

[0084] In an optional embodiment, the above-mentioned second generation submodule includes: a first adjustment unit, used to adjust the dimension of the first feature to obtain a second feature; an acquisition unit, used to perform an inner product on the second feature matrix on each channel to obtain a target type feature; a first processing unit, used to process the target type feature using softmax to obtain an attention mechanism graph; a second processing unit, used to perform matrix multiplication processing on the attention mechanism graph and the second feature to obtain a third feature; a second adjustment unit, used to adjust the dimension of the third feature to obtain a fourth feature having the same dimension as the first feature; and a superposition unit, used to superimpose the fourth feature with the first feature to generate the feature to be processed.

[0085] In an optional embodiment, the above-mentioned identification module 608 includes: a third determination submodule, used to perform regression processing on the features to be processed to determine the coordinate frame information and classification score of the object to be processed; and a fourth determination submodule, used to determine the behavior of the object to be processed based on the coordinate frame information and the classification score.

[0086] In an optional embodiment, the third determination submodule includes: a regression unit, configured to input the feature to be processed into a convolution layer, so as to use the convolution layer to regress the coordinate frame information and the classification score of the object to be processed.

[0087] In an optional embodiment, the above-mentioned regression unit includes: a regression subunit, used to regress the coordinate frame information of the object to be processed using the supervisory frame coordinates pre-configured in the convolution layer, wherein the supervisory frame coordinates are generated in the following manner: based on the first area and the second area of ​​the labeled frame of the object to be processed respectively contained in two consecutive frames of images included in the image sequence, a first category score and a second category score are obtained, wherein the first category score and the second category score are used to indicate the classification scores of the labeled frame of the object to be processed corresponding to the first area and the second area respectively; based on the first category score, the second category score, and the overlap IOU of the first area and the second area, the connection score of the two consecutive frames of images is determined; based on multiple connection scores, the coordinate frame information is obtained using a predetermined algorithm.

[0088] In an optional embodiment, determining the connection score of the two consecutive frames of images based on the first category score, the second category score, and the overlap IOU of the first region and the second region includes: determining the connection score S of the two consecutive frames of images by the following formula: c (R t ,R t+1 ): S c (R t ,R t+1 )=ψ(x)·[S c (R t )+S c (R t+1 )]+α·S c (R t )·S c (R t+1 ), where S c (R t ) and S c (R t+1 ) is the first region R t and the second region R t+1 The first category score and the second category score, x is the IOU of the two regions, α is a scalar, when there is region overlap, ψ(x) is equal to 1, otherwise ψ(x) is equal to 0.

[0089] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0090] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0091] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0092] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0093] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0094] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0095] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A behavior recognition method, characterized in that: include: Extracting features from an image sequence containing an object to be processed to obtain image sequence features of the image sequence, wherein the image sequence features are used to characterize motion information of the object to be processed in the image sequence; Performing feature extraction on a target image in the image sequence to obtain a target image feature of the target image, wherein the target image is an image selected from the image sequence, and the target image feature is used to characterize position information of the object to be processed in the target image; Determining features to be processed based on the image sequence features and the target image features; Based on the features to be processed, identifying the behavior of the object to be processed in the image sequence; Wherein, based on the image sequence features and the target image features, determining the feature to be processed includes: performing splicing and fusion processing on the image sequence features and the target image features to generate a first feature, wherein the image sequence features are three-dimensional features and the target image features are two-dimensional features; performing superposition processing on the first feature to generate the feature to be processed; performing splicing and fusion processing on the image sequence features and the target image features to generate the first feature includes: setting a first parameter for identifying the number of frames of the image sequence included in the image sequence features to 1 to obtain an image sequence adjustment feature; splicing the image sequence adjustment feature with the target image feature in the target channel dimension to obtain a spliced ​​feature; performing feature fusion processing on the spliced ​​feature to generate the first feature.

2. The method according to claim 1, characterized in that Before extracting features from an image sequence containing the object to be processed and obtaining image sequence features of the image sequence, the method further includes: Acquire a surveillance video containing the object to be processed; The monitoring video is subjected to frame extraction processing to obtain the image sequence.

3. The method according to claim 1, characterized in that The target images include at least two; The step of determining the features to be processed based on the image sequence features and the target image features includes: Determining a reference image feature based on target image features of each target image included in the at least two target images; Based on the image sequence features and the reference image features, features to be processed are determined.

4. The method according to claim 3, characterized in that: Based on target image features of each target image included in the at least two target images, determining the reference image feature comprises: An average value of the target image features of each of the target images is obtained to obtain the reference image features.

5. The method according to claim 1, characterized in that Performing superposition processing on the first feature to generate the feature to be processed includes: Adjusting the dimension of the first feature to obtain a second feature; Performing an inner product on the second feature on each channel to obtain a target type feature; The target type features are processed using softmax to obtain an attention mechanism graph; Performing matrix multiplication on the attention mechanism graph and the second feature to obtain a third feature; Adjusting the dimension of the third feature to obtain a fourth feature having the same dimension as the first feature; The fourth feature is superimposed on the first feature to generate the feature to be processed.

6. The method according to claim 1, characterized in that Based on the feature to be processed, identifying the behavior of the object to be processed in the image sequence includes: Performing regression processing on the features to be processed to determine the coordinate frame information and classification score of the object to be processed; The behavior of the object to be processed is determined based on the coordinate frame information and the classification score.

7. The method according to claim 6, characterized in that Performing regression processing on the features to be processed to determine the coordinate frame information and classification score of the object to be processed includes: The features to be processed are input into a convolutional layer, so as to use the convolutional layer to regress the coordinate frame information and classification score of the object to be processed.

8. The method according to claim 7, characterized in that Regressing the coordinate frame information of the object to be processed using the convolutional layer includes: The coordinate frame information of the object to be processed is regressed using the supervisory frame coordinates pre-configured in the convolutional layer, wherein the supervisory frame coordinates are generated in the following manner: Based on a first area and a second area of ​​the annotation box of the object to be processed respectively contained in two consecutive frames of images included in the image sequence, obtaining a first category score and a second category score, wherein the first category score and the second category score are used to indicate classification scores of the annotation box of the object to be processed corresponding to the first area and the second area, respectively; Determine a connection score of the two consecutive image frames based on the first category score, the second category score, and the overlap IOU of the first region and the second region; The coordinate frame information is obtained by using a predetermined algorithm based on the plurality of connection scores.

9. The method according to claim 8, characterized in that Determining the connection score of the two consecutive frames of images based on the first category score, the second category score, and the overlap IOU of the first region and the second region includes: The connection score S of the two consecutive frames of images is determined by the following formula c (R t ,R t+1 ): S c (R t ,R t+1 )=ψ(x)·[S c (R t )+S c (R t+1 )]+α·S c (R t )·S c (R t+1 ), Among them, S c (R t ) and S c (R t+1 ) is the first region R t and the second region R t+1 The first category score and the second category score, x is the IOU of the two regions, α is a scalar, when there is region overlap, ψ(x) is equal to 1, otherwise ψ(x) is equal to 0.

10. A behavior recognition device, characterized in that: include: A first acquisition module is used to extract features from an image sequence containing an object to be processed, and obtain image sequence features of the image sequence, wherein the image sequence features are used to characterize motion information of the object to be processed in the image sequence; An extraction module, used for performing feature extraction on a target image in the image sequence to obtain target image features of the target image, wherein the target image is an image selected from the image sequence, the target image features are used to characterize position information of the object to be processed in the target image, and the target image includes at least two; A determination module, used for determining a feature to be processed based on the image sequence feature and the target image feature; A recognition module, used for recognizing the behavior of the object to be processed in the image sequence based on the feature to be processed; Wherein, the determination module includes: a first generation submodule, used for splicing and fusing the image sequence feature and the target image feature to generate a first feature, wherein the image sequence feature is a three-dimensional feature and the target image feature is a two-dimensional feature; a second generation submodule, used for superimposing the first feature to generate the feature to be processed; the first generation submodule includes: a setting unit, used for setting the first parameter included in the image sequence feature for identifying the number of frames of the image sequence to 1, so as to obtain an image sequence adjustment feature; a splicing unit, used for splicing the image sequence adjustment feature with the target image feature in the target channel dimension to obtain a spliced ​​feature; a fusion unit, used for performing feature fusion processing on the spliced ​​feature to generate the first feature.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 9 when executed by a processor.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 9 are implemented.

Images