Behavior recognition method and device, computer device, and storage medium
By constructing a skeletal temporal map in behavior recognition and using a two-dimensional convolutional network, the problem of low recognition efficiency caused by complex network structures is solved, and efficient behavior recognition is achieved on various devices.
Patent Information
- Application Number
- CN202210356842.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-06
- Publication Date
- 2026-02-13
- Estimated Expiration
- 2042-04-06
AI Technical Summary
In existing behavior recognition solutions, the complex network structure leads to low device recognition efficiency or even failure to achieve it, making it difficult to perform behavior recognition efficiently on devices with varying performance levels.
By acquiring multiple frames of behavioral video images, human skeletal points are extracted, a skeletal temporal map is constructed, and the map is input into a matching two-dimensional convolutional neural network for feature extraction, simplifying the network structure to improve recognition efficiency.
By utilizing the position and temporal information of the skeletal temporal map, efficient behavior recognition can be achieved using only a regular two-dimensional convolutional network, ensuring recognition accuracy and making it applicable to various devices.
Smart Images
Figure CN114821465B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of behavior recognition, and in particular, to a behavior recognition method and device, a computer device, and a storage medium. BACKGROUND
[0002] In the field of computer vision processing, a behavior recognition method is used to recognize the content of an image or a video through a pre-trained visual recognition model, so as to determine the behavior of a living body (such as a person or an animal) in the image or the video. This is a relatively common visual processing application scenario.
[0003] In the current behavior recognition scheme, a network structure with a time sequence relationship (such as a long short term memory (LSTM) network or a three-dimensional (3D) convolution network) or a graph convolution network is generally used to extract features, so as to output a behavior category. These behavior recognition schemes can ensure the accuracy of behavior recognition, but these network structures are usually complex and contain many operators that are not commonly used. When a device runs these networks, a relatively long operation time is often required, and the recognition efficiency is low. SUMMARY
[0004] The present application provides a behavior recognition method, device, computer device, and storage medium to solve the technical problem that a complex network structure has a low recognition efficiency or cannot be implemented on a device.
[0005] In a first aspect, a behavior recognition method is provided, comprising:
[0006] obtaining a plurality of frames of behavior video images;
[0007] performing human skeleton point extraction on each frame of behavior video image to obtain human skeleton identification information corresponding to each frame of behavior video image, wherein one human skeleton identification information is used to indicate one human skeleton point in one frame of behavior video image;
[0008] constructing at least one skeleton time sequence graph according to the human skeleton identification information corresponding to each frame of behavior video image, wherein the skeleton time sequence graph is used to reflect the position distribution information of different human skeleton points in the same frame of behavior video image and the time sequence change information of each human skeleton point, and the time sequence change information is information reflecting the position change of the same human skeleton point in different behavior video images;
[0009] inputting the at least one skeleton time sequence graph into a behavior recognition model matched with the at least one skeleton time sequence graph to obtain a behavior type corresponding to the plurality of frames of behavior video images.
[0010] In the technical solution, a plurality of behavior video images are acquired, human body skeleton points of each behavior video image are extracted, the human body skeleton points in each behavior video image are acquired, a skeleton time sequence diagram is constructed according to the human body skeleton points in each behavior video image, and finally the skeleton time sequence diagram is input into a behavior recognition model to obtain a behavior type corresponding to the plurality of behavior video images. Since the skeleton time sequence diagram can reflect position information of different human body skeleton points in the same behavior video image and change information of each human body skeleton in different behavior recognition images, it is equivalent to pre-processing the behavior video image carrying the skeleton information, so that the pre-processed skeleton time sequence diagram has skeleton time sequence information and position information. In this way, only a common two-dimensional convolution network can be used to extract the skeleton time sequence feature and the skeleton position feature, so as to complete the behavior recognition and ensure the recognition accuracy. The common two-dimensional convolution network has a simple network structure and can be implemented on various performance devices, and the required operation time is relatively short, so that the efficiency of the behavior recognition can be improved.
[0011] With reference to the first aspect, in a possible implementation manner, the constructing at least one skeleton time sequence diagram according to the human body skeleton identification information corresponding to the plurality of behavior video images comprises: determining at least one time sequence set corresponding to the plurality of behavior video images, each time sequence set comprising a plurality of time sequences; constructing a target time sequence channel array corresponding to a target time sequence set to obtain at least one time sequence channel array, wherein the target time sequence set is any time sequence set in the at least one time sequence set, and the target time sequence channel array is used to indicate arrangement of a plurality of human body skeleton points belonging to the same person on a two-dimensional matrix plane under a plurality of time sequences in the target time sequence set; and filling the at least one time sequence channel array by value according to the human body skeleton identification information corresponding to the plurality of behavior video images and by time sequence to obtain the at least one skeleton time sequence diagram. The time sequence channel sequence reflecting arrangement of a plurality of human body skeleton points belonging to the same person on a two-dimensional matrix plane under a plurality of time sequences corresponding to the behavior video image is constructed according to the time sequence of the behavior video image, and the time sequence channel sequence is filled by value, so that the skeleton time sequence diagram can have skeleton time sequence information and position information at the same time.
[0012] With reference to the first aspect, in a possible implementation manner, the constructing a target time sequence channel array corresponding to a target time sequence set comprises: rearranging a plurality of human body skeleton points belonging to the same person in structure to make the plurality of human body skeleton points distributed on a matrix two-dimensional plane to obtain a two-dimensional skeleton array corresponding to the plurality of human body skeleton points; and expanding each array position in the two-dimensional skeleton array by time sequence according to a plurality of time sequences in the target time sequence set to obtain the target time sequence channel array. The time sequence channel sequence can reflect position features and time sequence features of the human body skeleton points at the same time by rearranging the human body skeleton points in structure and expanding the human body skeleton points by time sequence.
[0013] With reference to the first aspect, in a possible implementation manner, the number of the at least one time sequence set is multiple, and the time sequences in each time sequence set are arranged in a time sequence order; wherein the first n time sequences in the (k+1)th time sequence set in the at least one time sequence set are the last n time sequences in the kth time sequence set in the at least one time sequence set, k and n are positive integers, 1≤n<N, 1≤k<K, N is the number of time sequences contained in one time sequence set, and K is the number of the at least one time sequence set. The number of time sequence sets is multiple, which can make the skeleton time sequence graph contain more time sequence information and improve the accuracy of behavior recognition; the first several time sequences in the latter time sequence set are the same as the latter several time sequences in the former time sequence set, which can make the skeleton time sequence graphs have close relevance, and can further improve the accuracy of behavior recognition.
[0014] With reference to the first aspect, in a possible implementation manner, the human body skeleton identification information includes position coordinates of the human body skeleton points in the one-frame behavior video image and confidence score of the position coordinates, and the confidence score is used to reflect the credibility of an image element corresponding to the position coordinates as a human body skeleton point; the assigning and filling of the at least one time sequence channel array according to the human body skeleton identification information corresponding to the multiple frames of behavior video images respectively to obtain the at least one skeleton time sequence graph includes: assigning and filling the at least one time sequence channel array respectively according to the position coordinates and the confidence score in the human body skeleton identification information corresponding to the multiple frames of behavior video images respectively in a time sequence to obtain the at least one skeleton time sequence graph. By using the position coordinates and the confidence score of the human body skeleton points, the time sequence channel sequence representing the arrangement of the human body skeleton points in multiple time sequences is assigned and filled to obtain the skeleton time sequence graph, which can make the skeleton time sequence graph contain more rich image features and be beneficial to improving the accuracy of behavior recognition.
[0015] With reference to the first aspect, in a possible implementation manner, the human body skeleton identification information corresponding to each of the plurality of behavior video images includes human body skeleton identification information of a plurality of human bodies; and the assigning and filling of the at least one time sequence channel array according to the human body skeleton identification information corresponding to each of the plurality of behavior video images to obtain the at least one skeleton time sequence image includes: assigning and filling the at least one time sequence channel array according to human body skeleton identification information corresponding to a target human body to obtain a skeleton filling image corresponding to the target human body, the target human body being any one of the plurality of human bodies; and determining the at least one skeleton time sequence image according to the skeleton filling image corresponding to each of the plurality of human bodies. By using the information of the human body skeleton points of the plurality of human bodies in the behavior recognition image to assign and fill the time sequence channel sequence representing the arrangement of the human body skeleton points at a plurality of time sequences to obtain the skeleton time sequence image, the skeleton time sequence image can contain richer image semantics, which is beneficial to improving the accuracy of behavior recognition.
[0016] With reference to the first aspect, in a possible implementation manner, the behavior recognition model includes M convolution layers, an i th convolution layer in the M convolution layers is connected with an (i-1) th convolution layer in the M convolution layers, and a first convolution layer in the M convolution layers is an input layer of the behavior recognition model, where a number of channels in the (i-1) th convolution layer is greater than a number of channels in the i th convolution layer, M is a positive integer greater than 4, and 1
[0017] With reference to the first aspect, in a possible implementation manner, the behavior recognition model further includes a max-pooling layer and a fully connected output layer, a group of adjacent convolution layers in the M convolution layers are connected through the max-pooling layer, used for inputting a feature map output by a previous convolution layer of the group of adjacent convolution layers to a subsequent convolution layer in the group of adjacent convolution layers after down-sampling, and the fully connected output layer is connected with an M th convolution layer in the M convolution layers, used for outputting the behavior type. By setting the max-pooling layer in the behavior recognition model, the dimensionality of the skeleton feature can be reduced and the perception field can be expanded, thereby facilitating feature extraction.
[0018] The second aspect provides a behavior recognition device, including:
[0019] An image acquisition module is configured to acquire a plurality of behavior video images.
[0020] extracting a human skeleton point of each row of behavior video images to obtain human skeleton identification information corresponding to each of the plurality of rows of behavior video images, wherein one human skeleton identification information is used to indicate one human skeleton point in one row of behavior video images;
[0021] constructing at least one skeleton time sequence diagram according to the human skeleton identification information corresponding to each of the plurality of rows of behavior video images, wherein the skeleton time sequence diagram is used to reflect position distribution information of different human skeleton points in the same row of behavior video images and time sequence change information of each human skeleton point, and the time sequence change information is information reflecting position change of the same human skeleton point in different behavior video images;
[0022] inputting the at least one skeleton time sequence diagram into a behavior recognition model matched with the at least one skeleton time sequence diagram to obtain a behavior type corresponding to the plurality of rows of behavior video images.
[0023] In a third aspect, a computer device is provided, including a memory and one or more processors, the one or more processors being configured to execute one or more computer programs stored in the memory, and the one or more processors, when executing the one or more computer programs, cause the computer device to implement the behavior recognition method in the first aspect.
[0024] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, the computer program including program instructions, and the program instructions, when executed by a processor, cause the processor to execute the behavior recognition method in the first aspect.
[0025] The present application can achieve the following technical effects: since the skeleton time sequence diagram can reflect position information of different human skeleton points in the same behavior video image and change information of each human skeleton in different behavior recognition images, it is equivalent to pre-processing the behavior video image carrying the skeleton information, so that the pre-processed skeleton time sequence diagram has the time sequence information and the position information of the skeleton. In this way, only a common two-dimensional convolution network can be used to extract the skeleton time sequence feature and the skeleton position feature, so as to complete the behavior recognition and ensure the recognition accuracy. The common two-dimensional convolution network has a simple network structure and can be implemented on various performance devices, and the required operation time is relatively short, so that the efficiency of the behavior recognition can be improved. BRIEF DESCRIPTION OF DRAWINGS
[0026] Figure 1 A flowchart of a behavior recognition method provided by an embodiment of the present application;
[0027] Figure 2 A human skeleton point diagram provided by an embodiment of the present application;
[0028] Figure 3 is a structural schematic diagram of a behavior recognition device provided by an embodiment of the present application;
[0029] Figure 4 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0030] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
[0031] The technical solutions of the present application can be applied to various scenes that need to recognize or monitor behaviors. For example, the technical solutions of the present application can be applied to a video platform to recognize behaviors in a video, to classify the video, or can be applied to a video platform to recognize behaviors in a video, to monitor bad behaviors in the video, or can be applied to video monitoring in public places to monitor illegal behaviors in public places, and the like, without being limited to the examples herein.
[0032] The technical solutions of the present application can be specifically applied to various types of computer devices, such as notebook computers, servers, and the like. In particular, the technical solutions of the present application can be applied to some computer devices with low processing performance, such as mobile phones, tablet computers, and the like.
[0033] In the present application, the general technical concept is as follows: in the process of recognizing behaviors in a behavior video based on the behavior video, after a plurality of behavior video images are obtained, human body skeleton points are first extracted from each behavior video image to obtain information of the human body skeleton points in each behavior video image, and a skeleton time sequence diagram is constructed according to the information of the human body skeleton points in each behavior video image to complete preprocessing of the plurality of behavior video images, so that the skeleton time sequence diagram has time sequence information and position information of the human body skeleton points; then the skeleton time sequence diagram having the time sequence information and the position information of the human body skeleton points is input into a common two-dimensional convolutional neural network to extract skeleton time sequence features and skeleton position features, thereby completing behavior recognition. The network structure of the common two-dimensional convolutional neural network is simple, the required operation performance is low, the common two-dimensional convolutional neural network can be deployed and used on various devices, is convenient to migrate and use, the required operation time is less, and the efficiency of behavior recognition can be improved; the behavior video images are preprocessed in advance, so that the skeleton time sequence diagram carries the time sequence information and the position information of the human body skeleton points, and the accuracy of recognition can also be ensured.
[0034] The technical solutions of the present application will be specifically introduced below.
[0035] Reference is made to Figure 1 , Figure 1 is a flowchart of a behavior recognition method provided by an embodiment of the present application, which can be applied to a computer device, such asFigure 1 As shown, the method comprises the following steps:
[0036] S101, acquire a plurality of behavior video images.
[0037] Here, the plurality of behavior video images refer to a plurality of images with time sequence relationship extracted from a certain behavior video and used to reflect the behavior in the behavior video. The behavior video refers to a video used to reflect a certain behavior. For example, the behavior video is a dancing video, and the plurality of behavior video images are a plurality of images extracted from the dancing video; or, the behavior video is a circus performance video, and the plurality of behavior recognition images are a plurality of images extracted from the circus performance.
[0038] Specifically, a plurality of behavior video images with time sequence continuity can be extracted from the behavior video as the plurality of behavior video images in the present application. For example, a plurality of behavior video images with time sequence T1-T300 can be extracted from the behavior video as the plurality of behavior video images, and the number of the plurality of behavior video images is 300.
[0039] S102, perform human skeleton point extraction on each behavior video image to acquire human skeleton identification information corresponding to each of the plurality of behavior video images.
[0040] In the embodiment of the present application, one human skeleton identification information is used to indicate one human skeleton point in one behavior video image. The human skeleton point feature extraction on the behavior video image refers to determining the position of the human skeleton point in the behavior video image through a human skeleton point extraction method. The same human skeleton point extraction method is used for each behavior video image, so that the positions of the human skeleton points in the plurality of behavior video images can be determined respectively, and the human skeleton identification information corresponding to each of the plurality of behavior video images is obtained. The human skeleton point extraction method includes but is not limited to distance transformation algorithm, image thinning algorithm, deep learning method, etc. The human skeleton identification information can be the human skeleton identification information of one person or the human skeleton identification information of multiple persons. Whether to extract the human skeleton identification information of one person or the human skeleton identification information of multiple persons depends on the specific design. When the design is to extract the human skeleton identification information of multiple persons, if only the human skeleton identification information of one person is extracted (i.e., only one person exists in the behavior video image), it is determined that the human skeleton identification information of the other person is all 0.
[0041] In one specific implementation, the human skeleton point extraction on each behavior video image can be performed through openpose based on convolutional neural network to acquire the human skeleton identification information corresponding to each of the plurality of behavior video images. Exemplarily, in the case of performing human skeleton point extraction on the behavior video image through openpose, for each behavior video image, the human skeleton identification information of one person can be obtained asFigure 2 The 18 human body skeleton points shown belong to the same human body, and the 18 human body skeleton identification information corresponding to each frame of the video image includes position coordinates and a confidence score. The position coordinates are used to indicate the position of the human body skeleton point in the video image, and the confidence score is used to reflect the credibility of the image element corresponding to the position coordinates as a human body skeleton point. The higher the confidence score, the greater the possibility that the image element corresponding to the position coordinates is a human body skeleton point. The lower the confidence score, the smaller the possibility that the image element corresponding to the position coordinates is a human body skeleton point. Specifically, the human body skeleton identification can be represented as (x, y, score), where (x, y) is the position coordinates and score is the confidence score. When multiple human body skeleton identification information needs to be extracted, for each frame of the video image, multiple human body skeleton points can be obtained, each human body contains 18 human body skeleton points, and each frame of the video image corresponds to 18*G human body skeleton identification information, where G is the number of human bodies. Assuming that the number of frames of the video image is H, a plurality of frames of the video image correspond to 18*G*H human body skeleton identification information.
[0042] It should be understood that the number of human body skeleton identification information extracted based on openpose described above is only an example of the present application. In actual application, a larger or smaller number of human body skeleton point identification information can be obtained for each frame of the video image, and the human body skeleton identification information can be extracted based on other methods, which is not limited in the present application.
[0043] In S103, at least one skeleton time sequence diagram is constructed according to the human body skeleton identification information corresponding to each of the plurality of frames of the video image.
[0044] In the embodiment of the present application, the skeleton time sequence diagram is used to reflect the position distribution information of different human body skeleton points in the same frame of the video image and the time sequence change information of each human body skeleton point, wherein the time sequence change information refers to the information reflecting the position change of the same human body skeleton point in different frames of the video image.
[0045] In a feasible implementation, the skeleton time sequence diagram can be constructed based on steps A1-A3 as follows.
[0046] A1, at least one time sequence set corresponding to the plurality of frames of the video image is determined.
[0047] A2, a target time sequence channel array corresponding to a target time sequence set is constructed to obtain at least one time sequence channel array.
[0048] A3, at least one skeleton time sequence diagram is obtained by assigning and filling at least one time sequence channel array according to the human body skeleton identification information corresponding to each of the plurality of frames of the video image.
[0049] The time sequence set corresponding to the multi-frame behavioral video image in the step A1 is a subset of a total time sequence set corresponding to the multi-frame behavioral video image, and each time sequence set includes a plurality of time sequences extracted from the total time sequence set. The total time sequence set refers to a time sequence set composed of all time sequences corresponding to the multi-frame behavioral video image. In order to sufficiently obtain the time sequence information of the multi-frame behavioral video image, the union of the at least one time sequence set should be the total time sequence set.
[0050] In a possible implementation, the time sequences of the multi-frame behavioral video image can be arranged in time sequence order to obtain the total time sequence set corresponding to the multi-frame behavioral video image, and the time sequences in the total time sequence set are extracted based on a sliding window to obtain the at least one time sequence set. In this case, one time sequence set is obtained each time the sliding window is extracted, the number of time sequences included in the time sequence set is the same as the length of the sliding window, and the length of the sliding window is less than or equal to the number of time sequences in the total time sequence set. The number of time sequence sets K is equal to 1+(a-N) / b, where a is the number of time sequences in the total time sequence set (equal to the number of frames of the multi-frame behavioral video image), N is the length of the sliding window (that is, the number of time sequences included in one sliding window), and b is the sliding step of the sliding window, which is equal to the number of time sequences between one position (such as the head or tail of the sliding window) of the sliding window before and after sliding plus one. For example, the total time sequence set corresponding to the multi-frame behavioral video image is {T1, T2,..., T300}, the time sequences in the sliding window before sliding are {T1, T2, T3, T4}, and the time sequences in the sliding window after sliding are {T4, T5, T6, T7}. Therefore, the sliding step is equal to the number of time sequences between T1 and T4 plus one, that is, 3.
[0051] In the case where the length of the sliding window is less than the number of time sequences in the total time sequence set, the number of time sequence sets is multiple. In some possible cases, the sliding step of the sliding window can be set to be less than the length of the sliding window, so that the first n time sequences in the (k+1)th time sequence set in the multiple time sequence sets are the last n time sequences in the kth time sequence set in the multiple time sequence sets, where k and n are positive integers, 1≤n
[0052] For example, the total time sequence set corresponding to the multi-frame behavioral video image is {T1, T2,..., T300}, the length of the sliding window is 9, and the sliding step is 6. Based on the sliding window, the time sequence sets {T1, T2,..., T9}, {T7, T8,..., T15}, {T13, T2,..., T9},..., and {T292, T292,..., T300} can be obtained in turn, and the number of time sequence sets is 50. As can be seen, the last 3 time sequences in the previous time sequence set are the first 3 time sequences in the next time sequence set.
[0053] By setting the number of the time sequence sets as multiple, the skeleton time sequence diagram constructed based on the time sequence sets can contain more time sequence information, and the accuracy of behavior recognition can be improved; the first several time sequences in the latter time sequence set are the same as the latter several time sequences in the former time sequence set, the association between the time sequence sets can be established, so that the skeleton time sequence diagrams constructed based on the time sequence sets have close association, and the accuracy of behavior recognition can be further improved.
[0054] Optionally, in other embodiments, the time sequences in the time sequence total set can also be extracted based on other manners to obtain at least one time sequence set, any at least one time sequence set extracted based on the time sequence total set and the union of which is the time sequence total set all belong to the protection scope of the present application; the embodiment that the first n time sequences in the (k+1)th time sequence set in the multiple time sequence sets are the latter n time sequences in the kth time sequence set in the multiple time sequence sets is not limited to the above-mentioned manner of extracting time sequences based on the sliding window. Any manner of extracting time sequences that can make the first n time sequences in the (k+1)th time sequence set in the multiple time sequence sets be the latter n time sequences in the kth time sequence set in the multiple time sequence sets all belong to the protection scope of the present application.
[0055] The target time sequence set involved in the above-mentioned step A2 is any time sequence set in the at least one time sequence set, and the target time sequence channel array is used to indicate the arrangement of the multiple human body skeletal points belonging to the same person on the two-dimensional matrix plane under the multiple time sequences in the target time sequence set. For example, the time sequences in the target time sequence set are the aforementioned T1-T9, and the target time sequence channel array is used to indicate the arrangement of the multiple human body skeletal points belonging to the same person on the two-dimensional matrix plane under the time sequences T1-T9.
[0056] Specifically, the above-mentioned step A2 can include the following steps A21-A22:
[0057] A21, structurally rearranging the multiple human body skeletal points belonging to the same person to make the multiple human body skeletal points belonging to the same person be distributed on the two-dimensional matrix plane to obtain the two-dimensional skeletal array corresponding to the multiple human body skeletal points belonging to the same person.
[0058] A22, time sequence expanding each sequence position in the two-dimensional skeletal array according to the multiple time sequences in the target time sequence set to obtain the target time sequence channel array.
[0059] The basic principle of distributing the plurality of human body skeletal points belonging to the same person on the two-dimensional matrix plane after structural rearrangement can be that the relative positional relationship of the plurality of human body skeletal points belonging to the same person on the two-dimensional matrix plane after structural rearrangement is generally consistent with the relative positional relationship of the plurality of human body skeletal points before structural rearrangement. The two-dimensional skeletal array can be used to reflect the relative positional relationship between the plurality of human body skeletal points belonging to the same person.
[0060] The plurality of human body skeletal points belonging to the same person can be Figure 2 For example, the 18 human body skeletal points in the above table can be distributed on a 5*6 two-dimensional matrix plane to obtain a 5*6 two-dimensional skeletal array. Exemplarily, Figure 2 The 5*6 two-dimensional skeletal array corresponding to the 18 human body skeletal points shown in the above table can be as shown in Table 1.
[0061] 16 14 0 15 17 2 1 5 3 4 8 11 7 9 12 10 13
[0062] Table 1
[0063] Alternatively, the 18 human body skeletal points can also be distributed on a 5*4 two-dimensional matrix plane to obtain a 5*4 two-dimensional skeletal array. Exemplarily, Figure 2 The 5*4 two-dimensional skeletal array corresponding to the 18 human body skeletal points shown in the above table can be as shown in Table 2.
[0064] 16 14 0 15 17 2 3 1 6 5 4 8 11 7 9 10 12 13
[0065] Table 2
[0066] It should be understood that the manner of structural rearrangement of the plurality of human body skeletal points belonging to the same person is not limited to the above two manners, and other manners of structural rearrangement can also be adopted to distribute the plurality of human body skeletal points belonging to the same person on a two-dimensional matrix plane, and to make the two-dimensional matrix array obtained by structural rearrangement be used to reflect the relative positional relationship between the plurality of human body skeletal points belonging to the same person.
[0067] The time sequence expansion of the sequence position refers to copying each sequence position according to a plurality of time sequences in the target time sequence set, so that each sequence position is expanded into a plurality of sequence positions, and the plurality of sequence positions obtained by expansion are sequence positions under the plurality of time sequences.
[0068] In a feasible implementation manner, after each sequence position in the two-dimensional skeletal array is expanded into a plurality of sequence positions, the plurality of sequence positions corresponding to the same sequence position can be arranged in an array manner to obtain a target time sequence channel array. Further, the plurality of sequence positions corresponding to the same sequence position can be arranged into an m*m array.
[0069] Taking the two-dimensional skeleton array as the above Table 1 as an example, assuming that the target time sequence set includes 9 time sequences T1-T9, the first sequence position (i.e., the position of the human body skeleton point 16) in Table 1 can be copied to expand the first sequence position to 9, and the 9 sequence positions obtained by expansion can be as shown in Table 3, representing the first sequence position at T1-T9.
[0070] T1 T2 T3 T4 T5 T6 T7 T8 T9
[0071] Table 3
[0072] Each sequence position in Table 1 is expanded and filled in the manner shown in Table 3, and a target time sequence channel array of 18*15 can be obtained. Exemplarily, the target time sequence channel array can be as shown in Table 4.
[0073]
[0074]
[0075] Table 4
[0076] The sequence positions obtained by expansion of the same sequence position arranged in the form of an array can facilitate extraction of the time sequence characteristics of the skeleton. By rearranging the structure and expanding the time sequence of the human body skeleton points, the time sequence channel sequence can reflect the position characteristics and time sequence characteristics of the human body skeleton points at the same time. For each time sequence set in the at least one time sequence set, the time sequence channel array is constructed in the manner of the above steps A21-A22, and at least one time sequence channel array can be obtained.
[0077] In some possible cases, the human body skeleton identification information includes position coordinates of a human body skeleton point in a behavior video image and a confidence score of the position coordinates, and the confidence score is used to reflect the credibility of the image element corresponding to the corresponding position coordinates as the human body skeleton point (for example, the case of extracting the human body skeleton point by openpose mentioned in the foregoing step S102). The above step A3 specifically includes: according to the position coordinates and the confidence scores in the human body skeleton identification information corresponding to each of the plurality of behavior video images, respectively assigning and filling at least one time sequence channel array in time sequence to obtain at least one skeleton time sequence diagram.
[0078] Taking the human body skeleton identification information in the form of (x, y, score) and the time sequence channel array as shown in Table 4 as an example, the x of the behavior video image corresponding to the time sequence 1-9 can be assigned to Table 4 according to the corresponding time sequence (i.e., time sequence 1-time sequence 9) in Table 4, to obtain a skeleton time sequence Figure 1; and the y corresponding to the behavior video image of the time sequence 1-9 is used to assign values to table 4, so that the bone sequence is obtained Figure 2 ; and the score corresponding to the behavior video image of the time sequence 1-9 is used to assign values to table 4, so that the bone sequence is obtained Figure 3 ; the bone sequence Figure 1 , the bone sequence Figure 2 , and the bone sequence Figure 3 comprise a three-channel bone sequence diagram. Among them, for the sequence position of the human body bone point not indicated in the to-be-filled plane array (such as the sequence position of the position of the 4 / 5 / 6 row and the 4 / 5 / 6 column in table 4, which is obtained by table 4 based on the position of the 2nd row and the 2nd column in table 1), it can be assigned as 0.
[0079] By using the position coordinates and confidence scores of the human body bone points, the two-dimensional distribution and time sequence arrangement of the human body bone points are assigned and filled to obtain the bone sequence diagram, which can make the bone sequence diagram contain more image features, and is beneficial to improve the accuracy of behavior recognition.
[0080] Optionally, in some possible cases, the human body bone identification information corresponding to each frame of behavior video image includes human body bone identification information of multiple human bodies, and the above step A3 specifically includes the following steps A31-A32:
[0081] A31, according to the human body bone identification information corresponding to the target human body, at least one time sequence channel array is assigned and filled to obtain a bone filling diagram corresponding to the target human body.
[0082] Among them, the target human body is any one of the multiple human bodies. For each human body in the multiple human bodies, the assignment and filling are respectively performed according to the assignment and filling mode based on the human body bone identification information of the same human body, so as to obtain a bone filling diagram corresponding to each human body. The implementation mode of the assignment and filling based on the human body bone identification information of the same human body will be described below.
[0083] A32, according to the bone filling diagram corresponding to each human body in the multiple human bodies, at least one bone sequence diagram is determined.
[0084] In one possible implementation, the skeleton filling maps of the plurality of human bodies can be combined to obtain at least one skeleton time sequence map. In this case, the number of skeleton time sequence maps is equal to the product of the number of time sequence channel arrays, the number of information contained in the human body skeleton identification information, and the number of human bodies. For example, the human body skeleton identification information corresponding to each frame of behavior video image includes human body skeleton identification information of two human bodies, the number of time sequence channel arrays is 50, and the number of information contained in the human body skeleton identification information is 3, and thus the number of skeleton time sequence maps is 300.
[0085] In another possible implementation, the skeleton filling maps of the plurality of human bodies can be combined to obtain at least one skeleton time sequence map. In this case, the number of skeleton time sequence maps is equal to the product of the number of time sequence channel arrays and the number of information contained in the human body skeleton identification information. For example, the number of time sequence channel arrays is 50, and the number of information contained in the human body skeleton identification information is 3, and thus the number of skeleton time sequence maps is 150.
[0086] By using the information of human body skeleton points of the plurality of human bodies in the behavior recognition image, the two-dimensional distribution and time sequence arrangement of the human body skeleton points are filled to obtain the skeleton time sequence map, so that the skeleton time sequence map contains richer image semantics, and the accuracy of behavior recognition is improved.
[0087] Optionally, the skeleton time sequence map can also be constructed based on other implementations, which are not limited in the present application.
[0088] S104, inputting the at least one skeleton time sequence map into a behavior recognition model matched with the at least one skeleton time sequence map to obtain a behavior type corresponding to the plurality of frames of behavior video image.
[0089] In the embodiments of the present application, the behavior recognition model matched with the at least one skeleton time sequence map is a two-dimensional convolutional neural network structure with an input layer matched with the skeleton time sequence map and capable of realizing behavior recognition. The input layer matched with the skeleton time sequence map means that the convolutional layer can perform convolution processing on the skeleton time sequence map.
[0090] In some possible cases, the behavior recognition model can include M convolutional layers, an i-th convolutional layer in the M convolutional layers is connected with an (i-1)-th convolutional layer in the M convolutional layers, a first convolutional layer in the M convolutional layers is an input layer of the behavior recognition model, a channel number of the (i-1)-th convolutional layer in the M convolutional layers is greater than a channel number of the i-th convolutional layer in the M convolutional layers, M is a positive integer greater than 4, and 1 < i ≤ M. The channel numbers of the plurality of convolutional layers in the behavior recognition model change in a trend from more to less, which can better extract the position features and the time sequence features of the skeleton time sequence diagram; and the number of the convolutional layers is greater than 4, which can flexibly realize compression of the feature map according to requirements and reduce the model calculation amount. In order to enable the behavior recognition model to cover enough skeleton space-time relationships, in the first few convolutional layers, a convolution kernel with a larger convolution field can be used to perform convolution processing on the skeleton time sequence diagram, so as to extract more abundant time sequence features. For example, in the first few convolutional layers, a 17*17 convolution kernel or a 15*15 convolution kernel can be used for convolution processing. Compared with ordinary 3*3 or 5*5 convolution kernels, a larger convolution can cover more time sequences, thereby ensuring that more time sequence features are extracted. Alternatively, in order to control the parameter amount and the inference speed of the model, a larger convolution kernel can be used in the first few convolutional layers, and a smaller convolution kernel can be used in the last few convolutional layers.
[0091] In a specific implementation manner, the M convolutional layers can be similar to the convolutional structure in the VGG11 network, and there are 8 convolutional layers, where the 8 convolutional layers can be as shown in Table 6.
[0092] Input conv7-1024 conv7-1024 conv5-512 conv5-512 conv5-512 conv5-256 conv3-128 conv3-64
[0093] Table 6
[0094] Alternatively, in some possible cases, the behavior recognition model can further include a max-pooling layer and a fully connected output layer, where a group of adjacent convolutional layers in the M convolutional layers are connected through the max-pooling layer, the max-pooling layer is used to input, after down-sampling, a feature map output by a previous convolutional layer in the group of adjacent convolutional layers to a subsequent convolutional layer in the group of adjacent convolutional layers, and the fully connected output layer is connected with an M-th convolutional layer in the M convolutional layers and is used to output a behavior type. By setting the max-pooling layer in the behavior recognition model, dimension reduction of the skeleton features and expansion of a perception field can be realized, thereby facilitating feature extraction.
[0095] For example, the network structure of the behavior recognition model including the max-pooling layer and the fully connected output layer can be as shown in Table 7.
[0096] Input conv7-1024 conv7-1024 Max pooling layer conv5-512 conv5-512 conv5-512 conv5-512 Max pooling layer conv3-128 conv3-128 FC+softmax (Fully connected output layer)
[0097] Table 7
[0098] In the technical solution, a plurality of behavior video images are acquired, human body skeleton points of each behavior video image are extracted, human body skeleton points in each behavior video image are acquired, a skeleton time sequence graph is constructed according to the human body skeleton points in each behavior video image, and finally the skeleton time sequence graph is input into a behavior recognition model to obtain a behavior type corresponding to the plurality of behavior video images. Since the skeleton time sequence graph can reflect position information of different human body skeleton points in the same behavior video image and change information of each human body skeleton in different behavior recognition images, it is equivalent to pre-processing the behavior video image carrying the skeleton information, so that the pre-processed skeleton time sequence graph has skeleton time sequence information and position information. In this way, only a common two-dimensional convolution network can be used to extract the skeleton time sequence feature and the skeleton position feature, so as to complete the behavior recognition and ensure the recognition accuracy. The common two-dimensional convolution network has a simple network structure, and can also improve the efficiency of behavior recognition.
[0099] The method of the application is introduced above. In order to better implement the method of the application, the device of the application is introduced below.
[0100] Referring to Figure 3 , Figure 3 is a structural schematic diagram of a behavior recognition device provided by an embodiment of the application. The behavior recognition device can be a computer device or a part of a computer device. As shown in Figure 3 , the behavior recognition device 20 includes:
[0101] An image acquisition module 201 is configured to acquire a plurality of behavior video images.
[0102] An extraction module 202 is configured to extract human body skeleton points of each behavior video image to acquire human body skeleton identification information corresponding to each of the plurality of behavior video images. One human body skeleton identification information is used to indicate one human body skeleton point in one behavior video image.
[0103] An image construction module 203 is configured to construct at least one skeleton time sequence graph according to the human body skeleton identification information corresponding to each of the plurality of behavior video images. The skeleton time sequence graph is used to reflect position distribution information of different human body skeleton points in the same behavior video image and time sequence change information of each human body skeleton point. The time sequence change information is information reflecting position change of the same human body skeleton point in different behavior video images.
[0104] A type output module 204 is configured to input the at least one skeleton time sequence graph into a behavior recognition model matched with the at least one skeleton time sequence graph to obtain a behavior type corresponding to the plurality of behavior video images.
[0105] In some possible design, the image construction module 203 is specifically configured to: determine at least one time sequence set corresponding to the multi-frame behavioral video image, each time sequence set including a plurality of time sequences; construct a target time sequence channel array corresponding to a target time sequence set to obtain at least one time sequence channel array, where the target time sequence set is any time sequence set in the at least one time sequence set, and the target time sequence channel array is used to indicate arrangement of a plurality of human body skeleton points belonging to the same person on a two-dimensional matrix plane at the plurality of time sequences in the target time sequence set; and fill and value the at least one time sequence channel array according to human body skeleton identification information corresponding to each of the multi-frame behavioral video images and according to time sequence to obtain the at least one skeleton time sequence image.
[0106] In some possible design, the image construction module 203 is specifically configured to: perform structural rearrangement on a plurality of human body skeleton points belonging to the same person to distribute the plurality of human body skeleton points on a matrix two-dimensional plane to obtain a two-dimensional skeleton array corresponding to the plurality of human body skeleton points; and perform time sequence expansion on each array position in the two-dimensional skeleton array according to the plurality of time sequences in the target time sequence set to obtain the target time sequence channel array.
[0107] In some possible design, the number of the at least one time sequence set is a plurality, and the time sequences in each time sequence set are arranged in time sequence order; where the first n time sequences in the (k+1)th time sequence set in the at least one time sequence set are the last n time sequences in the kth time sequence set in the at least one time sequence set, k and n are positive integers, 1≤nN, 1≤kK, N is the number of time sequences included in a time sequence set, and K is the number of the at least one time sequence set.
[0108] In some possible design, a human body skeleton identification information includes a position coordinate of the human body skeleton point in the one-frame behavioral video image and a confidence score of the position coordinate, and the confidence score is used to reflect credibility of an image element corresponding to the position coordinate as a human body skeleton point; and the image construction module 203 is specifically configured to: fill and value the at least one time sequence channel array according to the position coordinate and the confidence score in the human body skeleton identification information corresponding to each of the multi-frame behavioral video images and according to time sequence respectively to obtain the at least one skeleton time sequence image.
[0109] In a possible design, the human body skeleton identification information corresponding to each frame of the behavior video image includes human body skeleton identification information of multiple human bodies; and the image construction module 203 is specifically configured to: fill the at least one time sequence channel array according to the human body skeleton identification information corresponding to a target human body in time sequence, to obtain a skeleton filling image corresponding to the target human body, the target human body being any one of the multiple human bodies; and determine the at least one skeleton time sequence image according to the skeleton filling images corresponding to the multiple human bodies respectively.
[0110] In some possible designs, the behavior recognition model includes M convolution layers, an i th convolution layer in the M convolution layers is connected with an (i-1) th convolution layer in the M convolution layers, and a first convolution layer in the M convolution layers is an input layer of the behavior recognition model, where a number of channels in the (i-1) th convolution layer is greater than a number of channels in the i th convolution layer, M is a positive integer greater than 4, and 1 < i ≤ M.
[0111] In some possible designs, the behavior recognition model further includes a max-pooling layer and a fully connected output layer, a group of adjacent convolution layers in the M convolution layers are connected through the max-pooling layer, used for inputting a feature map output by a previous convolution layer of the group of adjacent convolution layers to a subsequent convolution layer in the group of adjacent convolution layers after down-sampling, and the fully connected output layer is connected with an M th convolution layer in the M convolution layers, used for outputting the behavior type.
[0112] It should be noted that, Figure 3 The content not mentioned in the corresponding embodiments can be referred to the foregoing Figure 1 description of the method embodiments, which will not be repeated here.
[0113] The apparatus described above, by acquiring multiple frames of behavior video images, performing human body skeleton point extraction on each frame of behavior video image, obtaining human body skeleton points in each frame of behavior video image, then constructing a skeleton time sequence image according to the human body skeleton points in each frame of behavior video image, and finally inputting the skeleton time sequence image into a behavior recognition model to obtain a behavior type corresponding to the multiple frames of behavior video images. Since the skeleton time sequence image can reflect position information of different human body skeleton points in the same behavior video image and change information of each human body skeleton in different behavior recognition images, it is equivalent to pre-processing the behavior video image carrying skeleton information, so that the pre-processed skeleton time sequence image has skeleton time sequence information and position information. In this way, only a common two-dimensional convolution network can be used to extract skeleton time sequence features and skeleton position features, thereby completing behavior recognition and ensuring recognition accuracy. The common two-dimensional convolution network has a simple network structure, and can also improve the efficiency of behavior recognition.
[0114] Reference can be made toFigure 4 , Figure 4 is a structural schematic diagram of a computer device provided by an embodiment of the present application. The computer device 30 comprises a processor 301 and a memory 302. The processor 301 is connected to the memory 302, for example, the processor 301 can be connected to the memory 302 through a bus.
[0115] The processor 301 is configured to support the computer device 30 to perform the corresponding functions in the methods in the method embodiments described above. The processor 301 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The hardware chip described above can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD described above can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0116] The memory 302 is used to store program codes and the like. The memory 302 can include a volatile memory (VM), such as a random access memory (RAM); the memory 302 can also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 302 can also include a combination of the above-mentioned kinds of memories.
[0117] The processor 301 can invoke the program codes to perform the following operations:
[0118] Obtain a plurality of frame behavior video images;
[0119] Perform human skeleton point extraction on each frame behavior video image to obtain respective human skeleton identification information of the plurality of frame behavior video images, and one human skeleton identification information is used to indicate one human skeleton point in one frame behavior video image;
[0120] According to the human body skeleton point information corresponding to each of the multiple frame behavior video images, at least one skeleton time sequence graph is constructed, the skeleton time sequence graph is used to reflect position distribution information of different human body skeleton points in the same frame behavior video image, and time sequence change information of each human body skeleton point, the time sequence change information refers to information reflecting position change of the same human body skeleton point in different behavior video images;
[0121] The at least one skeleton time sequence graph is input into a behavior recognition model matched with the at least one skeleton time sequence graph, so as to obtain a behavior type corresponding to the multiple frame behavior video images.
[0122] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program includes program instructions, and the program instructions make the computer execute the method as described in the foregoing embodiments when the computer executes the program instructions.
[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM) and the like.
[0124] The above only discloses preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application, so equivalent changes made according to the claims of the present application still fall within the scope of the present application.
Claims
1. A behavior recognition method, characterized by, The method comprises: acquiring a plurality of behavior video images; performing human skeleton point extraction on each behavior video image to obtain respective human skeleton identification information of the plurality of behavior video images, wherein one human skeleton identification information is used to indicate one human skeleton point in one behavior video image; constructing at least one skeleton time sequence diagram according to the respective human skeleton identification information of the plurality of behavior video images, wherein the skeleton time sequence diagram is used to reflect position distribution information of different human skeleton points in the same behavior video image and time sequence change information corresponding to each human skeleton point, and the time sequence change information is information reflecting position change of the same human skeleton point in different behavior video images; wherein the constructing at least one skeleton time sequence diagram according to the respective human skeleton identification information of the plurality of behavior video images comprises: determining at least one time sequence set corresponding to the plurality of behavior video images, wherein each time sequence set comprises a plurality of time sequences, the number of the at least one time sequence set is a plurality, and the time sequences in each time sequence set are arranged in time sequence order, wherein the first n time sequences in the (k+1)th time sequence set in the at least one time sequence set are the last n time sequences in the kth time sequence set in the at least one time sequence set, k and n are positive integers, 1≤n<N, 1≤k<K, N is the number of time sequences contained in one time sequence set, and K is the number of the at least one time sequence set; constructing a target time sequence channel array corresponding to a target time sequence set to obtain at least one time sequence channel array, wherein the target time sequence set is any time sequence set in the at least one time sequence set, and the target time sequence channel array is used to indicate arrangement of a plurality of human skeleton points belonging to the same person on a two-dimensional matrix plane under a plurality of time sequences in the target time sequence set; and filling and assigning the at least one time sequence channel array in time sequence according to the respective human skeleton identification information of the plurality of behavior video images to obtain the at least one skeleton time sequence diagram; inputting the at least one skeleton time sequence diagram into a behavior recognition model matched with the at least one skeleton time sequence diagram to obtain a behavior type corresponding to the plurality of behavior video images.
2. The method of claim 1, wherein, The constructing a target time sequence channel array corresponding to a target time sequence set comprises: performing structural rearrangement on a plurality of human skeleton points belonging to the same person to make the plurality of human skeleton points distributed on a matrix two-dimensional plane to obtain a two-dimensional skeleton array corresponding to the plurality of human skeleton points; performing time sequence expansion on each array position in the two-dimensional skeleton array according to a plurality of time sequences in the target time sequence set to obtain the target time sequence channel array.
3. The method of claim 1, wherein, One human skeleton identification information comprises position coordinates of the one human skeleton point in the one behavior video image and a confidence score of the position coordinates, and the confidence score is used to reflect credibility of an image element corresponding to the position coordinates as a human skeleton point; The filling and assigning the at least one time sequence channel array in time sequence according to the respective human skeleton identification information of the plurality of behavior video images to obtain the at least one skeleton time sequence diagram comprises: According to position coordinates and confidence scores in the human body skeleton identification information corresponding to each of the multiple-frame behavior video images, the at least one time sequence channel array is sequentially assigned and filled to obtain the at least one skeleton time sequence image.
4. The method of claim 1, wherein, The human body skeleton identification information corresponding to each of the multiple-frame behavior video images comprises human body skeleton identification information of multiple human bodies. The assigning and filling of the at least one time sequence channel array according to the human body skeleton identification information corresponding to each of the multiple-frame behavior video images comprises: According to the human body skeleton identification information corresponding to a target human body, the at least one time sequence channel array is sequentially assigned and filled to obtain a skeleton filling image corresponding to the target human body, the target human body being any one of the multiple human bodies. According to the skeleton filling images corresponding to the multiple human bodies respectively, the at least one skeleton time sequence image is determined.
5. The method of claim 1, wherein, The behavior recognition model comprises M convolution layers, a first convolution layer in the M convolution layers being an input layer of the behavior recognition model, an i-th convolution layer in the M convolution layers being connected with an (i-1)-th convolution layer in the M convolution layers, a number of channels in the (i-1)-th convolution layer being greater than a number of channels in the i-th convolution layer, M being a positive integer greater than 4, and 1 6. A behavior recognition apparatus characterized by comprising: The method comprises: An image acquisition module is configured to acquire multiple-frame behavior video images; An extraction module is configured to perform human body skeleton point extraction on each of the multiple-frame behavior video images to obtain human body skeleton identification information corresponding to each of the multiple-frame behavior video images, one human body skeleton identification information being used to indicate one human body skeleton point in one behavior video image. An image construction module is configured to construct at least one skeleton time sequence image according to the human skeleton identification information corresponding to each of the multiple frame behavior video images, the skeleton time sequence image being configured to reflect the position distribution information of different human skeleton points in the same frame behavior video image and the time sequence change information of each human skeleton point, the time sequence change information being information reflecting the position change of the same human skeleton point in different behavior video images. A type output module is configured to input the at least one skeleton time sequence image into a behavior recognition model matched with the at least one skeleton time sequence image, so as to obtain the behavior type corresponding to the multiple frame behavior video images.
7. A computer device, characterized by The computer device includes a memory and a processor, the processor being configured to execute one or more computer programs stored in the memory, and the processor, when executing the one or more computer programs, causes the computer device to implement the method according to any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program including program instructions, and the program instructions, when executed by a processor, cause the processor to execute the method according to any one of claims 1-5.