Target object behavior recognition method, device, electronic device and medium

Abstract

The embodiments of this specification provide a method, device, electronic device and medium for target object behavior recognition, which divides the target behavior of the target object in the scene into sub-behaviors, and on this basis, samples the input video to obtain a sequence of images to be recognized arranged in chronological order, and then extracts features to obtain a feature data sequence, and then, based on the feature data sequence, predicts sub-behaviors to obtain sub-behavior prediction results of the target object in the input video, which include sub-behavior categories generated by the target object in the time period corresponding to the image sequence to be recognized; then, based on the sub-behavior prediction results, obtain target behavior category recognition results of the target object in the image sequence to be recognized, which is beneficial to improving the accuracy of the behavior recognition results.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular, to a method, device, electronic device, and medium for identifying target object behavior. Background Art

[0002] Nowadays, video behavior recognition technology, as a research hotspot in the field of computer vision, has been widely used in many fields such as intelligent monitoring, human-computer interaction such as gesture recognition, etc. Taking the in-vehicle behavior recognition scenario as an example, by collecting video images of people in the car, especially the driver, the behavior of the people in the car can be analyzed in real time, and reminders can be given when dangerous behaviors of people in the car are identified, so as to avoid safety accidents as much as possible and improve driving safety. In this process, if there are too many missed or misidentified cases, it will not only affect the safety of the vehicle, but also easily cause unnecessary disturbances to the user and affect the user experience. Therefore, the accuracy of the behavior recognition results is particularly important. Summary of the invention

[0003] The embodiments of this specification provide a method, device, electronic device and medium for identifying target object behavior.

[0004] In a first aspect, an embodiment of the present specification provides a target object behavior recognition method for recognizing a target behavior of a target object in a video, wherein the target behavior is divided into multiple sub-behaviors, and the method includes:

[0005] Sampling the input video to obtain a sequence of images to be identified, wherein the sequence of images to be identified includes multiple frames of images arranged in chronological order;

[0006] Extracting features from the image sequence to be identified to obtain a feature data sequence;

[0007] Based on the feature data sequence, a sub-behavior prediction result of the target object in the input video is obtained, wherein the sub-behavior prediction result includes: a sub-behavior category generated by the target object in a time period corresponding to the image sequence to be identified;

[0008] Based on the sub-behavior prediction result, a target behavior category recognition result of the target object in the to-be-recognized image sequence is obtained.

[0009] Furthermore, after obtaining the target behavior category recognition result, it also includes:

[0010] If the recognition result is that the target behavior has not occurred, and the sub-behavior prediction result includes the characteristic sub-behavior among the multiple sub-behaviors, then the sampling period is increased according to the preset step size, and the image sequence to be recognized is resampled according to the increased sampling period, and the target object behavior recognition is performed on the resampled image sequence to be recognized to obtain a new target behavior category recognition result;

[0011] The starting time point of the sampling period after the increase is before the time point corresponding to the characteristic sub-behavior, and the ending time point is after the ending time point of the sampling period before the increase.

[0012] Furthermore, after obtaining the target behavior category recognition result, it also includes:

[0013] If the recognition result is that the target behavior did not occur, and the sub-behavior prediction result does not include the characteristic sub-behavior among the multiple sub-behaviors, or the target behavior recognition result is that the target behavior occurred, then the target object behavior recognition of the next cycle will continue according to the current sampling cycle.

[0014] Furthermore, feature extraction is performed on the image sequence to be identified to obtain a feature data sequence, including:

[0015] Extracting features from each frame of the image in the sequence of images to be identified to obtain a feature map;

[0016] Extracting features from the feature graph, and splitting the extracted features according to the time dimension to obtain a time feature sequence;

[0017] The time feature sequence is input into a pre-trained recurrent neural network to obtain the feature data sequence, wherein the recurrent neural network is trained using a connection temporal classification loss function.

[0018] Furthermore, the feature extraction is performed on each frame of the image in the sequence of images to be identified to obtain a feature map, including:

[0019] Extracting features of each frame of the image in the sequence of images to be identified to obtain sequence features of each frame of the image;

[0020] The obtained sequence features are concatenated in chronological order, and the concatenated results are dimensionally transformed to obtain the feature graph.

[0021] Furthermore, obtaining a target behavior category recognition result of the target object in the to-be-recognized image sequence based on the sub-behavior prediction result includes:

[0022] Combining the sub-behavior categories included in the sub-behavior prediction results to obtain sub-behavior combination results;

[0023] The target behavior category recognition result of the target object in the to-be-recognized image sequence is determined by matching the sub-behavior combination result with a preset behavior combination strategy, wherein the behavior combination strategy includes a sub-behavior combination strategy corresponding to each target behavior.

[0024] Furthermore, the sub-behavior prediction results are also distributed with separation placeholders, and the sub-behavior categories included in the sub-behavior prediction results are combined to obtain sub-behavior combination results, including:

[0025] Deduplication processing is performed on adjacent and repeated sub-behavior categories in the sub-behavior prediction results;

[0026] After completing the deduplication process, the separation placeholders in the sub-behavior prediction results are deleted to obtain the sub-behavior combination results.

[0027] Furthermore, obtaining a sub-behavior prediction result of the target object based on the feature data sequence includes:

[0028] The feature data sequence is input into a preset activation function to obtain a sub-behavior prediction result of the target object in the input video.

[0029] Furthermore, the input video is video data of a target person inside or outside the vehicle collected by a vehicle-mounted camera, and the target object is the target person inside or outside the vehicle.

[0030] In a second aspect, an embodiment of the present specification provides a target object behavior recognition device for recognizing a target behavior of a target object in a video, wherein the target behavior is divided into a plurality of sub-behaviors, and the device comprises:

[0031] A sampling module, used for sampling the input video to obtain a sequence of images to be identified, wherein the sequence of images to be identified includes multiple frames of images arranged in chronological order;

[0032] A feature extraction module is used to extract features from the image sequence to be identified to obtain a feature data sequence;

[0033] A prediction module, configured to obtain a sub-behavior prediction result of a target object in the input video based on the feature data sequence, wherein the sub-behavior prediction result includes: a sub-behavior category generated by the target object in a time period corresponding to the image sequence to be identified;

[0034] The recognition module is used to obtain a target behavior category recognition result of the target object in the image sequence to be recognized based on the sub-behavior prediction result.

[0035] In a third aspect, an embodiment of the present specification provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the target object behavior recognition method provided in the first aspect are implemented.

[0036] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the target object behavior recognition method provided in the first aspect above.

[0037] The target object behavior recognition method provided by an embodiment of the present specification divides the target behavior of the target object in the video scene into sub-behaviors, obtains a sequence of images to be recognized arranged in chronological order by sampling the input video, and then obtains a feature data sequence through feature extraction, and then performs sub-behavior prediction based on the feature data sequence to obtain a sub-behavior prediction result of the target object in the input video, which includes the sub-behavior category generated by the target object in the time period corresponding to the image sequence to be recognized; and then obtains the target behavior category recognition result of the target object in the image sequence to be recognized based on the sub-behavior prediction result. Compared with the scheme of directly outputting the behavior category, this scheme disassembles and predicts the target behavior of the target object, first predicts the sub-behavior category, such as the opening and closing sub-behavior actions of the driver's yawning behavior in the in-vehicle behavior recognition scenario, and then parses the final target behavior category based on the sub-behavior prediction result, which can effectively reduce the prediction errors caused by the partial similarity of the two target behaviors, which is conducive to improving the accuracy of the behavior recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a target object behavior recognition method provided in the first aspect of the embodiment of this specification;

[0039] Figure 2 This is a diagram of the first-stage feature extraction network structure in the embodiments of this specification;

[0040] Figure 3 This is a diagram of the second stage feature extraction network structure in the embodiment of this specification;

[0041] Figure 4 A processing flow chart for input of different lengths of sampling periods for the conventional scheme;

[0042] Figure 5 A schematic diagram of sub-behavior prediction results corresponding to sampling periods of different lengths in the embodiments of this specification;

[0043] Figure 6 A module block diagram of a target object behavior recognition device provided in the second aspect of the embodiment of this specification;

[0044] Figure 7 A schematic diagram of the structure of an electronic device provided in the third aspect of the embodiments of this specification. DETAILED DESCRIPTION

[0045] In order to better understand the technical solutions provided by the embodiments of this specification, the technical solutions of the embodiments of this specification are described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0046] It should be noted that the target object behavior recognition method provided in the embodiments of this specification can be applied to various scenarios that require video behavior recognition. This embodiment mainly uses the in-vehicle behavior recognition scenario and the gesture recognition scenario as examples for illustration, and is not intended to limit the application scenarios.

[0047] For example, in a vehicle behavior recognition scenario, a vehicle camera can be used to collect video images of target persons such as drivers or pedestrians inside or outside the vehicle, and the target person can be used as the target object. The internal chip executes the target object behavior recognition method provided in the embodiment of this specification for the collected video image to realize the target behavior recognition of the target person inside or outside the vehicle, and respond to the recognition result according to the actual scenario needs. For example, in one application scenario, the target behavior can be configured as a behavior that endangers driving safety. In this way, when a dangerous behavior of a target person inside or outside the vehicle is identified, a warning function can be triggered to remind relevant personnel to stop the dangerous behavior in time to ensure driving safety.

[0048] In the gesture recognition scenario, the video image of the device user can be captured by the camera set in the device, and then the device executes the target object behavior recognition method provided in the embodiment of this specification for the captured video image to realize the recognition of the user's target behavior in the video, and respond to the recognition result according to the actual scene needs. For example, in the scenario where the user controls the volume and song control of the device through gestures, the target behavior can be configured as the operation behavior required in the scenario, such as volume increase operation, previous song operation, next song operation, volume increase and next song operation, etc. When these operation behaviors are recognized, the corresponding function processing process can be entered.

[0049] The specific technical solutions provided in the embodiments of this specification are described in detail below.

[0050] In the first aspect, the embodiments of this specification provide a method for identifying target object behavior, which is used to identify the target behavior of a target object in a video. In the specific implementation, it is necessary to predetermine the target behavior involved in the scene, and pre-split the target behavior into multiple sub-behaviors according to the behavioral action process, that is, determine the sub-behavior category, and the determination of the specific target behavior and the splitting of the sub-behavior can be carried out according to the actual application scenario. It should be noted that in a specific application scenario, one target behavior can be set, or a variety of different target behaviors can be set according to the recognition needs. In the same scene, the sub-behavior combinations obtained by splitting different target behaviors are not exactly the same, and there may be partial overlaps. Among them, the incompleteness can include two situations, one is that there is at least one different sub-behavior, and the other is that the sub-behavior categories split into are exactly the same but the combination order of each sub-behavior is different.

[0051] Taking the behavior recognition scenario of target persons inside or outside the car as an example, dangerous behaviors can be configured as target behaviors in advance, and then these target behaviors can be split to determine the sub-behavior categories. For example, if the target object is the driver, the target behaviors may include yawning, picking up things and other behaviors that are not conducive to driving safety. Yawning can be further split into two actions: opening the mouth + closing the mouth, and picking up things can be split into three actions: squatting + reaching out + standing up, and each action is a sub-behavior. Taking the volume and song control scenario as an example, the target behavior can include various operation behaviors needed in the scenario, such as increasing the volume and playing the next song. For example, the volume increase and playing the next song operation can be split into four types of gestures: up, up, left, and left, and each type of gesture is a sub-behavior.

[0052] On this basis, we can collect image sequences, extract features and predict sub-behaviors, and finally parse the target behavior of the target object. Figure 1 The target object behavior recognition method provided in the embodiments of this specification may at least include the following steps S101 to S104.

[0053] Step S101 , sampling the input video to obtain a sequence of images to be recognized, wherein the sequence of images to be recognized includes a plurality of frames of images arranged in chronological order.

[0054] The input video is the video frame whose target object behavior needs to be identified. The specific source can be determined according to the actual application scenario, and this embodiment does not limit this. For example, in the scene of identifying the behavior of a target person inside or outside the car, the input video can be the video data of the target person inside or outside the car collected in real time by the on-board camera; in the scene of gesture recognition, the input video can be the gesture video data of the device user collected in real time by the camera in the recognition device.

[0055] In specific implementation, the input video data can be sampled through a sliding window of a set time length to obtain a sequence of images to be identified in each sampling period. It should be noted that the initial sampling period and the sampling interval can be configured according to the actual application scenario and multiple tests, such as 2 seconds or 3 seconds. The sequence of images to be identified includes multiple frames of images arranged in chronological order, and the specific number of frames is determined according to the length of the sampling period. After determining the sequence of images to be identified in the current period, the following step S102 can be performed, that is, entering the feature extraction stage.

[0056] Step S102: extracting features from the image sequence to be identified to obtain a feature data sequence.

[0057] Specifically, feature extraction of the image sequence to be identified is to obtain the spatiotemporal feature expression of each frame image in the image sequence to be identified. There are many specific implementation methods, which can be configured according to the computing power factor of the actual scene.

[0058] In an optional embodiment, feature extraction can be performed in two stages. In the first stage, feature extraction is performed on each frame of the image sequence to be identified to obtain a feature map. In the second stage, feature extraction is performed on the feature map, and the extracted features are split according to the time dimension to obtain a time feature sequence. The time feature sequence is input into a pre-trained recurrent neural network to obtain the above-mentioned feature data sequence.

[0059] Among them, the feature extraction of the first stage is to send the images contained in the image sequence to be identified into the preset first feature extraction network in sequence for feature extraction, and the obtained features are more of an expression of the spatial semantics of the input information. Therefore, in addition to being used in this solution, the feature extraction network of the first stage can also be used as network input for other tasks such as in-vehicle face detection and facial key point detection, that is, the feature extraction network of the first stage can be used for feature extraction of other deep learning tasks at the same time, thereby greatly improving the efficiency of the product. The feature extraction of the second stage is to perform deep spatiotemporal feature extraction on the features output by the first stage, which increases the temporal correlation and is conducive to obtaining a more comprehensive spatiotemporal feature expression, so that the spatial and temporal dimension features of the input information can be used to achieve more accurate recognition of the target behavior.

[0060] In an optional embodiment, the feature extraction process of the first stage may include: performing feature extraction on each frame of the image sequence to be identified to obtain sequence features of each frame of the image; splicing the obtained sequence features in chronological order, and performing dimensionality conversion on the splicing results to obtain a feature map.

[0061] For example, Figure 2As shown, each frame of the image sequence to be identified (for example, the dimension is: w*h) can be input into the first feature extraction network composed of backbone networks such as Resnet34-150 series, Inception or VGG, and the output tensor corresponding to each frame of the image (for example, the dimension is: c*w′*h′) can be obtained. Then, through the pooling operation, the sequence features corresponding to each frame of the image are obtained, such as a series of c*1*1 feature expressions shown in the figure. Then, this series of sequence features are spliced ​​in chronological order to obtain features of size c*1*T. Among them, T is the time dimension. Then, the splicing result is dimensionally converted to obtain a feature tensor of dimension 1*c*T. In this way, the feature tensor can be regarded as a feature map, which is convenient for feature extraction in the subsequent second stage.

[0062] like Figure 3 As shown, the obtained feature map is further input into the preset second feature extraction network. For example, the second feature extraction network can be implemented by a series of convolutional networks and downsampling. Depending on the different hardware computing power during the specific implementation, the number of convolutional layers here can be selected between 3 and 7 layers. Taking three downsampling as an example, the input 1*c*T feature tensor is successively downsampled three times (such as Figure 3 After layer1, layer2 and layer3 shown in ), the output tensor size is c1*c / 8*T / 8. For example, each layer can be composed of con2d(3*3)+BatchNorm+Relu+Maxpool2d structure. Then, after another layer of convolution (such as conv+BN+Relu structure), the c dimension is globally pooled, and the time series dimension is retained to obtain an output feature of size c2*1*T / 8. Furthermore, the output features of the second feature extraction network are split according to the time dimension to obtain a time feature sequence. For example, in the above example, the time feature sequence includes T / 8 groups of feature data corresponding to different time nodes. If T=80, there are 10 groups of feature data for different time nodes.

[0063] Furthermore, the time feature sequence of size c2*1*T / 8 obtained in the above example is input into a pre-trained recurrent neural network such as LSTM (Long Short-Term Memory) network to better express the feature correlation between spatial features and temporal features, and provide performance support for better prediction of results later. For example, a recurrent neural network can output a feature data sequence of size num_class*1*T / 8, which contains feature data of T / 8 groups of different time nodes. Among them, num_class represents the number of sub-behavior categories.

[0064] In addition, in an optional embodiment, in order to further improve the accuracy of the recognition result, the above-mentioned recurrent neural network can be trained based on the connectionist temporal classification (Connectionist Temporal Classification, CTC) loss function. That is to say, in the process of training the network structure, after the feature extraction output of the above-mentioned second stage, the CTC loss function is used to calculate the loss for gradient backtransmission (when predicting, the feature data sequence output by the second stage feature extraction network is connected to the activation function to realize the classification prediction of the sub-behavior category). In this way, a very long result sequence can be trained corresponding to a shorter label, that is, the number of dimensions of the prediction result (T / 8 as shown in the figure)> the number of categories (num_class as shown in the figure) is realized. Moreover, according to the characteristics of the shared weight of the recurrent neural network, it is possible to extract sequence features of different lengths. In this way, the length of the sampling period can be adjusted without changing the feature extraction network structure, thereby improving the applicability of the network structure. It is convenient to adaptively increase the sampling period as needed in the subsequent use process, that is, to realize the variable length sampling period strategy, so as to reduce the situation where the behavior sampled to half leads to recognition errors or even cannot be recognized, thereby further improving the accuracy of the recognition result.

[0065] It is understandable that in conventional behavior recognition methods, the settings of sampling period and sampling interval are limited by the network's own parameters and the limited computing power of the chip, and generally will not be changed after selection. If sampling and recognition are performed based on fixed sampling period and sampling interval, it is easy to sample half of the behavior and cause recognition errors or even fail to recognize it. Even if the sampling period is increased and the sampling interval strategy is optimized, there is still a situation where the long sampling period leads to the loss of key frames, which in turn leads to model prediction errors and poor generalization.

[0066] The specific reason is that Figure 4 For example, for input sequences of different lengths, according to the conventional solution, a fixed number of frames are usually extracted at P1 and then input into the subsequent network, which easily leads to the risk of key frame information loss. If a recurrent neural network is used only in the P2 part, i.e., the fully connected layer classification part, although the sampling problem of the P1 part can be avoided, the output dimension of the fully connected layer also needs to be changed, that is, the network structure still needs to be changed, and multiple networks with different fully connected layers need to be deployed in actual deployment, which is inefficient.

[0067] The embodiment of this specification adopts the above-mentioned two-stage feature extraction and CTC constraint model training scheme, which can subdivide the sampling period. The feature extraction of the first stage is not limited by the sampling period, and the extraction of spatiotemporal features in the second stage introduces a recurrent neural network. According to the characteristics of the recurrent neural network sharing weights, it can achieve the extraction of sequence features of different lengths. At the same time, because the penalty term uses the ctc constraint, the trainability of the network designed by the scheme and the stability of the prediction are guaranteed.

[0068] It should be noted that during the model training process, the feature extraction networks in the above two stages can be trained separately, or they can be trained as a whole. The specific training method can be selected according to the needs of the actual scenario. For example, when the feature extraction network in the first stage needs to be applied to other deep learning tasks, separate training can be adopted.

[0069] It should also be noted that, in addition to the feature extraction method exemplified above, in other implementations of this specification, other applicable feature extraction networks may also be sampled to complete feature extraction of the image sequence to be identified, and this embodiment does not impose any limitation on this.

[0070] After the feature extraction stage is completed, the following step S103 may be further performed, that is, entering the sub-behavior prediction stage.

[0071] Step S103, based on the feature data sequence, obtaining a sub-behavior prediction result of the target object in the input video, the sub-behavior prediction result including: a sub-behavior category generated by the target object in a time period corresponding to the image sequence to be identified.

[0072] Specifically, the feature data sequence may be input into a preset activation function to determine the sub-behavior category generated by the target object in the time period corresponding to the image sequence to be identified.

[0073] Taking the feature data sequence (num_class*1*T / 8) obtained in the above example as an example, by inputting it into the activation function, T / 8 groups of sub-behavior category probability distributions can be obtained. Then, the sub-behavior category with the largest probability in each group of sub-behavior category probability distributions is used as the predicted sub-behavior category, and a sub-behavior prediction result consisting of T / 8 predicted sub-behavior categories can be obtained. For example, the activation function can use a softmax function suitable for multi-classification. Of course, other applicable activation functions can also be used, and this embodiment does not limit this.

[0074] Further, the following step S104 may be executed to enter the target behavior category parsing stage.

[0075] Step S104: obtaining a target behavior category recognition result of the target object in the to-be-recognized image sequence based on the sub-behavior prediction result.

[0076] For example, behavior combination strategies can be pre-set and stored according to the sub-behavior splitting method. The behavior combination strategies include sub-behavior combination strategies corresponding to each target behavior. For example, the sub-behavior combination strategy of yawning behavior is "open mouth and close mouth"; the sub-behavior combination strategy of picking up behavior is "squat down, stretch out your hands and stand up"; the sub-behavior combination strategy of increasing the volume and playing the next song behavior is "up, up, left, left".

[0077] In specific implementation, the target behavior category recognition result of the target object in the image sequence to be recognized can be determined based on the sub-behavior prediction result and the preset behavior combination strategy. For example, the sub-behavior prediction result can be matched with the behavior combination strategy, and the successfully matched target behavior can be used as the target behavior category recognition result of the target object in the image sequence to be recognized.

[0078] In an optional embodiment, the above process of determining the target behavior category recognition result based on the sub-behavior prediction result and the preset behavior combination strategy may include: combining the sub-behavior categories contained in the sub-behavior prediction result to obtain the sub-behavior combination result; and then matching the sub-behavior combination result with the preset behavior combination strategy to determine the target behavior category recognition result occurring in the image sequence to be identified.

[0079] For example, considering that some sub-behaviors may last for a long time, repeated sub-behaviors are likely to continue to exist in the sub-behavior prediction results. Therefore, in order to facilitate subsequent matching, the above combination processing may include deduplication processing. For example, if the sub-behavior prediction result is: open mouth, open mouth, close mouth, close mouth, after deduplication processing, the sub-behavior combination result obtained is: open mouth, close mouth.

[0080] In addition, if the above-mentioned two-stage feature extraction plus CTC constraint solution is adopted in the feature extraction stage, there are also separator placeholders distributed in the sub-behavior prediction results, for example, " / " can be used as an indication in this article. In this case, the above-mentioned combination processing process may include: performing deduplication processing on adjacent and repeated sub-behavior categories in the sub-behavior prediction results; after completing the deduplication processing, deleting the separator placeholders in the sub-behavior prediction results to obtain the sub-behavior combination results.

[0081] Taking gesture recognition as an example, if the sub-behavior prediction result is: up / / / up / / left / / / / left, after the above preprocessing, the sub-behavior combination result can be obtained: up-up-left-left (i.e., the actual sub-behavior combination), and then it can be matched with the preset behavior combination strategy, and the category recognition result of the target behavior can be matched: volume increase and previous song, so as to respond to the operation behavior and enter the pre-configured function processing process. Because of the existence of the separator placeholder, the sampling cycle length and interval have a certain degree of adaptability during training, and there is no need to deliberately consider cutting out the entire behavior exactly, which has important value in practical applications.

[0082] Furthermore, after obtaining the target behavior category recognition result, the target object behavior recognition method provided in the embodiment of this specification also includes: judging whether the target object has the target behavior according to the target behavior category recognition result, and judging whether the sub-behavior prediction result contains characteristic sub-behaviors among the multiple sub-behaviors that have been split out. It should be noted that the two judgment processes can be performed sequentially in the aforementioned order, that is, when it is determined that the target object has not had the target behavior, it is judged whether the sub-behavior prediction result contains characteristic sub-behaviors; or, the two judgment processes can also be performed simultaneously, or in the opposite order, and this embodiment does not limit this.

[0083] If the recognition result is that the target behavior has not occurred, and the sub-behavior prediction result includes characteristic sub-behaviors among multiple sub-behaviors, it means that the current sampling period may have only sampled a part of the target behavior occurrence time period. Therefore, the sub-behavior prediction result is incomplete, resulting in the inability to recognize the target behavior. At this time, the sampling period can be increased according to the preset step length, and the image sequence to be recognized can be resampled according to the increased sampling period, and the target object behavior recognition is performed on the resampled image sequence to be recognized to obtain a new target behavior category recognition result. In other words, the above steps S102 to S104 are performed for the resampled image sequence to be recognized, and the target behavior category recognition result after the expanded sampling period is obtained again. For example, if the target behavior can be recognized from the resampled image sequence to be recognized, the sampling and behavior recognition of the next cycle can be continued according to the increased sampling period. If the target behavior still cannot be recognized, the above steps of increasing the sampling period according to the preset step length, resampling the image sequence to be recognized according to the increased sampling period, and recognizing the target object behavior on the resampled image sequence to be recognized can be repeated to obtain a new target behavior category recognition result. During specific implementation, an upper limit value of the increase of the sampling period can be pre-configured, that is, the maximum sampling period cannot exceed the upper limit value. It can be specifically configured according to the actual scenario and will not be described in detail here.

[0084] Therefore, the embodiments of the present invention can well identify sub-behaviors within a small time series range, effectively improving the problem of difficult model prediction or prediction errors caused by the sampling period ending at the beginning of the target behavior or the sampling period starting at the end of the target behavior in the actual production process, which is conducive to further improving the accuracy of the recognition results.

[0085] Among them, the characteristic sub-behavior can be set according to the needs of the actual application scenario. For example, in one application scenario, all sub-behaviors split from the target sub-behavior can be used as characteristic sub-behaviors. At this time, if the sub-behavior prediction result contains any sub-behavior, it is considered that the sub-behavior prediction result contains characteristic sub-behaviors. For another example, the starting sub-behavior of the target behavior, and the combination of several consecutive sub-behaviors (except the last sub-behavior) starting from the starting sub-behavior can be used as characteristic sub-behaviors. If the target behavior 1 can be split into sub-behaviors "abc", then sub-behavior "a" and sub-behavior "ab" can be used as target sub-behaviors. At this time, the sub-behavior prediction result only contains the starting sub-behavior of any target behavior, or contains a combination of several consecutive sub-behaviors starting from the starting sub-behavior of any target behavior, and the combination does not contain other sub-behaviors, then the sub-behavior prediction result is considered to contain characteristic sub-behaviors.

[0086] During specific implementation, the preset step size can be set by the duration of the target behavior involved in the specific application scenario, for example, it can be set to 1 second, 2 seconds or 3 seconds, etc. Moreover, in the above sampling period lengthening strategy, the starting time point of the sampling period after the increase is located before the time point corresponding to the characteristic sub-behavior, and the end time point is located after the end time point of the sampling period before the increase. In this way, the first frame of the image in the resampled image sequence to be identified can be located before the image frame corresponding to the characteristic sub-behavior, and the last frame of the image in the resampled image sequence to be identified can be located after the last frame of the image in the image sequence to be identified before resampling, thereby ensuring that the resampled image sequence to be identified covers the entire target behavior occurrence stage of the target object as much as possible.

[0087] For example, the start time point of the increased sampling period can be moved forward by a preset fixed time length compared to the start time point before the update. Alternatively, the length of time the start time point is moved forward compared to before the update can also be determined based on the position of the characteristic sub-behavior in the sub-behavior prediction result. Still taking target behavior 1 as an example, assuming that the sub-behavior prediction result before the update is " / / / aa", the result cannot identify target behavior 1, but contains characteristic sub-behavior a. From the last to the first position, characteristic sub-behavior a is located at the 2 / 5 position, then the start time point can be moved forward by 2 / 5*T1+ΔT, where T1 represents the sampling period before the update, and ΔT is the time adjustment amount set in advance based on multiple experiments.

[0088] Of course, if the recognition result is that the target behavior has not occurred, and the sub-behavior prediction result does not contain the characteristic sub-behaviors among the multiple sub-behaviors, the target object behavior recognition of the next cycle will continue according to the current sampling cycle. If the target behavior recognition result is that the target behavior has occurred, on the one hand, the corresponding subsequent operations can be performed according to the identified target behavior category. The specific operations are set according to the actual application scenario. For example, in the in-vehicle driver behavior recognition scenario, a voice reminder can be initiated to the driver, such as "XX behavior is detected in you, please pay attention to driving safety"; on the other hand, the target object behavior recognition of the next cycle will continue according to the current sampling cycle. It should be noted that in scenarios where there are multiple target behaviors, the above-mentioned occurrence of the target behavior can be the occurrence of any one of the target behaviors.

[0089] In order to understand the above process more clearly, refer to Figure 5 Give an example. Figure 5 The figure shows the schematic diagram of the sub-behavior prediction results obtained under sampling periods of different lengths. In the figure, the black area represents the time period when the target behavior 1 occurs, the gray area represents the time series, the dotted box represents the sampling period, and the overlapping area of ​​two adjacent dotted boxes represents the intersection area of ​​two adjacent sampling periods. It can be understood that the longer the sampling period, the more corresponding time nodes, and the longer the sub-behavior prediction results obtained.

[0090] from Figure 5 It can be seen from the figure that when the sampling period is T1, the image sequence to be identified is sampled in this period. If the sub-behavior prediction result is " / / / / / ", it means that no behavior occurs. As the sampling progresses, if the sub-behavior prediction result obtained in a certain period is " / / / aa", where a represents a characteristic sub-behavior, the sampling of the expanded period will be performed again, so that the expanded sampling period covers the entire time period when the target behavior 1 occurs. For example, if it is expanded to Figure 5 In the T2 period shown in the figure, the sub-behavior prediction result of the resampled image sequence to be identified is " / / / aabbc / / / ", and the target object can be identified as having target behavior 1 based on the result. It should be noted that since the shared weight recurrent neural network is used in the feature extraction network of the above two stages, and the CTC constraint is used in the training process, the sub-behavior results under different sampling period lengths can be predicted.

[0091] Depend on Figure 5It can be seen that after a conventional sampling cycle enters the network designed in this scheme, it undergoes temporal sub-modularization, which can effectively avoid recognition errors caused by half of the behavior. After a suspected behavior is found, the sampling cycle can be expanded to fully recognize the entire behavior process. No additional network structure changes are required, nor is there a risk of losing key frames due to expanding the sampling cycle like 3D convolution. This not only improves recognition accuracy, but also ensures timeliness.

[0092] In summary, this solution predicts the target behavior of the target object by breaking it down into sub-behavior categories, such as the opening and closing of the mouth in the driver's yawning behavior in the in-vehicle behavior recognition scenario, and then parses the final target behavior category based on the sub-behavior prediction results. Compared with the solution that directly outputs behavior categories such as yawning and picking up things through 3D convolution or CNN+RNN network end-to-end, the prediction of combined behaviors is more accurate. In addition, it can effectively reduce the prediction errors caused by the partial similarity of two target behaviors, which is conducive to improving the accuracy of behavior recognition results.

[0093] In addition, the feature extraction network and CTC constraints of the above two sampling stages can well identify sub-behaviors within a small time series range, effectively solving the problem of difficult model prediction or prediction errors caused by the sampling period ending at the beginning of the target behavior or the sampling period starting at the end of the target behavior in the actual production process. In addition, the feature extraction network of the first stage can be used as network input for other tasks such as in-vehicle face detection and face key point detection, that is, the feature extraction network of the first stage can be used for feature extraction of other deep learning tasks at the same time, greatly improving the efficiency of the product.

[0094] In the second aspect, based on the same inventive concept as the target object behavior recognition method provided in the first aspect embodiment, the embodiment of this specification also provides a target object behavior recognition device for recognizing the target behavior of the target object in the video, wherein the target behavior is divided into multiple sub-behaviors. Figure 6 As shown, the target object behavior recognition device 60 includes:

[0095] The sampling module 601 is used to sample the input video to obtain a sequence of images to be identified, wherein the sequence of images to be identified includes multiple frames of images arranged in chronological order;

[0096] A feature extraction module 602 is used to extract features from the image sequence to be identified to obtain a feature data sequence;

[0097] A prediction module 603 is used to obtain a sub-behavior prediction result of the target object in the input video based on the feature data sequence, wherein the sub-behavior prediction result includes: a sub-behavior category generated by the target object in the time period corresponding to the image sequence to be identified;

[0098] The recognition module 604 is used to obtain a target behavior category recognition result of the target object in the to-be-recognized image sequence based on the sub-behavior prediction result.

[0099] In an optional implementation, the identification module 604 is further configured to:

[0100] If the recognition result is that the target behavior has not occurred, and the sub-behavior prediction result includes the characteristic sub-behavior among the multiple sub-behaviors, then the sampling period is increased according to the preset step size, and the image sequence to be recognized is resampled according to the increased sampling period, and the target object behavior recognition is performed on the resampled image sequence to be recognized to obtain a new target behavior category recognition result;

[0101] The starting time point of the sampling period after the increase is before the time point corresponding to the characteristic sub-behavior, and the ending time point is after the ending time point of the sampling period before the increase.

[0102] In an optional implementation, the identification module 604 is further configured to:

[0103] If the recognition result is that the target behavior did not occur, and the sub-behavior prediction result does not include the characteristic sub-behavior among the multiple sub-behaviors, or the target behavior recognition result is that the target behavior occurred, then the target object behavior recognition of the next cycle will continue according to the current sampling cycle.

[0104] In an optional implementation, the feature extraction module 602 includes:

[0105] A first extraction submodule is used to extract features from each frame of the image in the sequence of images to be identified to obtain a feature map;

[0106] The second extraction submodule is used to extract features from the feature graph and split the extracted features according to the time dimension to obtain a time feature sequence;

[0107] The input submodule is used to input the time feature sequence into a pre-trained recurrent neural network to obtain the feature data sequence, wherein the recurrent neural network is trained using a connection time series classification loss function.

[0108] In an optional implementation, the first extraction submodule is used to:

[0109] Extracting features of each frame of the image in the sequence of images to be identified to obtain sequence features of each frame of the image;

[0110] The obtained sequence features are concatenated in chronological order, and the concatenated results are dimensionally transformed to obtain the feature graph.

[0111] In an optional implementation, the identification module 604 is specifically used for:

[0112] Combining the sub-behavior categories included in the sub-behavior prediction results to obtain sub-behavior combination results;

[0113] The target behavior category recognition result of the target object in the to-be-recognized image sequence is determined by matching the sub-behavior combination result with a preset behavior combination strategy, wherein the behavior combination strategy includes a sub-behavior combination strategy corresponding to each target behavior.

[0114] In an optional implementation manner, the sub-behavior prediction results are further distributed with separation placeholders, and the identification module 604 is specifically used for:

[0115] Deduplication processing is performed on adjacent and repeated sub-behavior categories in the sub-behavior prediction results;

[0116] After completing the deduplication process, the separation placeholders in the sub-behavior prediction results are deleted to obtain the sub-behavior combination results.

[0117] In an optional implementation, the prediction module 603 is specifically used for:

[0118] The feature data sequence is input into a preset activation function to obtain a sub-behavior prediction result of the target object in the input video.

[0119] In an optional implementation, the input video is video data of a target person inside or outside the vehicle captured by a vehicle-mounted camera, and the target object is the target person inside or outside the vehicle.

[0120] It should be noted that the target object behavior recognition device 60 provided in the embodiment of this specification, wherein the specific manner in which each module performs operations has been described in detail in the method embodiment provided in the above-mentioned first aspect, and the specific implementation process can refer to the method embodiment provided in the above-mentioned first aspect, and will not be elaborated here.

[0121] In a third aspect, based on the same inventive concept as the target object behavior recognition method provided in the above embodiment, the embodiment of this specification also provides an electronic device. Figure 7As shown, the electronic device includes a memory 704, one or more processors 702, and a computer program stored in the memory 704 and executable on the processor 702. When the processor 702 executes the program, the steps of any embodiment of the target object behavior recognition method provided in the first aspect above are implemented. For example, the electronic device may be a vehicle-mounted terminal device, or may be an electronic device equipped with a camera and having a gesture recognition function, such as a smart phone, a smart TV, a virtual reality device, or a head-mounted smart device, etc., which is not limited in this embodiment.

[0122] Among them, Figure 7 In the embodiment of the present invention, a bus architecture (represented by bus 700) is shown, which may include any number of interconnected buses and bridges, and bus 700 links various circuits including one or more processors represented by processor 702 and memory represented by memory 704. Bus 700 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 705 provides an interface between bus 700 and receiver 701 and transmitter 703. Receiver 701 and transmitter 703 may be the same element, namely a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 702 is responsible for managing bus 700 and general processing, while memory 704 may be used to store data used by processor 702 when performing operations.

[0123] Understandably, Figure 7 The structure shown is for illustration only. The electronic device provided in the embodiments of this specification may also include Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown. Figure 7 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0124] In the fourth aspect, based on the same inventive concept as the target object behavior identification method provided in the aforementioned embodiment, the embodiment of this specification also provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of any embodiment of the target object behavior identification method provided in the first aspect above.

[0125] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] Although the preferred embodiments of this specification have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of this specification.

[0130] Obviously, those skilled in the art can make various changes and modifications to this specification without departing from the spirit and scope of this specification. Thus, if these modifications and variations of this specification fall within the scope of the claims of this specification and their equivalents, this specification is also intended to include these modifications and variations.

Claims

1. A method for identifying target object behavior, characterized in that: For identifying a target behavior of a target object in a video, the target behavior being divided into a plurality of sub-behaviors, the method comprising: Sampling the input video to obtain a sequence of images to be identified, wherein the sequence of images to be identified includes multiple frames of images arranged in chronological order; Extracting features from the image sequence to be identified to obtain a feature data sequence, including: extracting features from each frame of the image sequence to be identified to obtain a feature map; extracting features from the feature map, and splitting the extracted features according to the time dimension to obtain a time feature sequence; inputting the time feature sequence into a pre-trained recurrent neural network to obtain the feature data sequence, wherein the recurrent neural network is trained using a connection time series classification loss function; Based on the feature data sequence, a sub-behavior prediction result of the target object in the input video is obtained, wherein the sub-behavior prediction result includes: a sub-behavior category generated by the target object in a time period corresponding to the image sequence to be identified; Based on the sub-behavior prediction result, a target behavior category recognition result of the target object occurring in the to-be recognized image sequence is obtained; if the recognition result is that the target behavior has not occurred, and the sub-behavior prediction result includes characteristic sub-behaviors among the multiple sub-behaviors, the sampling period is increased according to a preset step size, and the to-be recognized image sequence is resampled according to the increased sampling period, and the target object behavior is recognized on the resampled image sequence to be recognized, so as to obtain a new target behavior category recognition result; Among them, feature extraction is performed on each frame of the image in the image sequence to be identified to obtain a feature map, including: feature extraction is performed on each frame of the image in the image sequence to be identified to obtain sequence features of each frame of the image; the obtained sequence features are spliced ​​in chronological order, and the splicing results are dimensionalized to obtain the feature map.

2. The method according to claim 1, characterized in that The starting time point of the sampling period after the increase is located before the time point corresponding to the characteristic sub-behavior, and the ending time point is located after the ending time point of the sampling period before the increase.

3. The method according to claim 1, characterized in that After obtaining a target behavior category recognition result of the target object in the to-be-recognized image sequence based on the sub-behavior prediction result, the method further includes: If the recognition result is that the target behavior did not occur, and the sub-behavior prediction result does not include the characteristic sub-behavior among the multiple sub-behaviors, or the target behavior recognition result is that the target behavior occurred, then the target object behavior recognition of the next cycle will continue according to the current sampling cycle.

4. The method according to claim 1, characterized in that: The step of obtaining a target behavior category recognition result of the target object in the to-be-recognized image sequence based on the sub-behavior prediction result includes: Combining the sub-behavior categories included in the sub-behavior prediction results to obtain sub-behavior combination results; The target behavior category recognition result of the target object in the to-be-recognized image sequence is determined by matching the sub-behavior combination result with a preset behavior combination strategy, wherein the behavior combination strategy includes a sub-behavior combination strategy corresponding to each target behavior.

5. The method according to claim 4, characterized in that The sub-behavior prediction results are also distributed with separation placeholders, and the sub-behavior categories included in the sub-behavior prediction results are combined to obtain sub-behavior combination results, including: Deduplication processing is performed on adjacent and repeated sub-behavior categories in the sub-behavior prediction results; After completing the deduplication process, the separation placeholders in the sub-behavior prediction results are deleted to obtain the sub-behavior combination results.

6. The method according to claim 1, characterized in that The obtaining, based on the feature data sequence, a sub-behavior prediction result of the target object in the input video includes: The feature data sequence is input into a preset activation function to obtain a sub-behavior prediction result of the target object in the input video.

7. The method according to claim 1, characterized in that The input video is video data of a target person inside or outside the vehicle collected by a vehicle-mounted camera, and the target object is the target person inside or outside the vehicle.

8. A target object behavior recognition device, characterized in that: The device is used to identify a target behavior of a target object in a video, wherein the target behavior is divided into a plurality of sub-behaviors, and comprises: A sampling module, used for sampling the input video to obtain a sequence of images to be identified, wherein the sequence of images to be identified includes multiple frames of images arranged in chronological order; A feature extraction module is used to extract features from the image sequence to be identified to obtain a feature data sequence, and the feature extraction module includes: a first extraction submodule, used to extract features from each frame of the image sequence to be identified to obtain a feature map; a second extraction submodule, used to extract features from the feature map, and split the extracted features according to the time dimension to obtain a time feature sequence; an input submodule, used to input the time feature sequence into a pre-trained recurrent neural network to obtain the feature data sequence, wherein the recurrent neural network is trained using a connection time series classification loss function; A prediction module, configured to obtain a sub-behavior prediction result of a target object in the input video based on the feature data sequence, wherein the sub-behavior prediction result includes: a sub-behavior category generated by the target object in a time period corresponding to the image sequence to be identified; A recognition module, for obtaining a target behavior category recognition result of the target object in the to-be-recognized image sequence based on the sub-behavior prediction result; if the recognition result is that the target behavior does not occur, and the sub-behavior prediction result includes a characteristic sub-behavior among the multiple sub-behaviors, then increasing the sampling period according to a preset step length, and re-sampling the to-be-recognized image sequence according to the increased sampling period, performing target object behavior recognition on the re-sampled to-be-recognized image sequence, and obtaining a new target behavior category recognition result; Among them, feature extraction is performed on each frame of the image in the image sequence to be identified to obtain a feature map, including: feature extraction is performed on each frame of the image in the image sequence to be identified to obtain sequence features of each frame of the image; the obtained sequence features are spliced ​​in chronological order, and the splicing results are dimensionalized to obtain the feature map.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images