Target behavior candidate box generation method and device, electronic equipment, and storage medium

Abstract

The present disclosure provides a target behavior candidate box generation method and device, electronic equipment and storage medium, and relates to the field of machine learning. The target behavior candidate box generation method comprises: acquiring a to-be-detected video frame sequence, and inputting the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map; inputting the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map; inputting the multi-scale time feature map into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map; and performing filtering processing on the initial behavior candidate box to obtain a target behavior candidate box of the to-be-detected video sequence. The technical scheme of the embodiment of the present disclosure can generate a behavior candidate box of the to-be-detected video frame sequence, and efficiently predict a behavior label, a behavior start time and a behavior end time of the behavior candidate box.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of machine learning, and in particular, to a target behavior candidate box generation method, a target behavior candidate box generation apparatus, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the deepening of the research in the field of autonomous driving, the autonomous driving system mainly includes an autonomous driving perception system and an autonomous driving decision system, and the autonomous driving decision system relies on the feedback information of the autonomous driving perception system; therefore, constructing a high-level autonomous driving perception system in the autonomous driving system is the key to the development of the field of autonomous driving.

[0003] However, in the related autonomous driving perception system, either a preset window sliding window is used to detect and identify the video frame sequence, or a behavior probability score combination method is used to detect and identify the video frame sequence, and the former has the problem of inaccurate time positioning, and the latter has the problem of low detection efficiency.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the embodiments of the present disclosure is to provide a target behavior candidate box generation method, a target behavior candidate box generation apparatus, an electronic device, and a computer readable storage medium, thereby at least partially overcoming the problems of low detection efficiency of video frame sequence and inaccurate time positioning.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to a first aspect of the embodiments of the present disclosure, a target behavior candidate box generation method is provided, including: obtaining a to-be-detected video frame sequence, and inputting the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map; inputting the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map; inputting the multi-scale time feature map into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map; the initial behavior candidate box includes a behavior confidence, a behavior start prediction time, and a behavior end prediction time; and performing filtering processing on the initial behavior candidate box to obtain a target behavior candidate box of the to-be-detected video sequence.

[0008] In some example embodiments of the present disclosure, based on the foregoing scheme, the prediction network comprises a standard time prediction network and a behavior label prediction network, and the inputting the multi-scale time feature map into the pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map comprises: inputting the multi-scale time feature map into the standard time prediction network to obtain a time feature point of the multi-scale time feature map, and obtaining a behavior start prediction time and a behavior end prediction time of the time feature point; obtaining a behavior confidence of the time feature point based on the behavior label prediction network; and constructing an initial behavior candidate box corresponding to the time feature point according to the behavior confidence, the behavior start prediction time and the behavior end prediction time.

[0009] In some example embodiments of the present disclosure, based on the foregoing scheme, the obtaining the behavior start prediction time and the behavior end prediction time of the time feature point comprises: obtaining a left offset and a right offset of the time feature point output by the standard time prediction network; calculating the behavior start prediction time of the time feature point based on the time feature point and the left offset; and calculating the behavior end prediction time of the time feature point based on the time feature point and the right offset.

[0010] In some example embodiments of the present disclosure, based on the foregoing scheme, the filtering processing of the initial behavior candidate box to obtain a target behavior candidate box of the video sequence to be detected comprises: performing the following loop process until all behavior candidate boxes in the initial behavior candidate box are traversed, and constructing a target behavior candidate box set according to the target behavior candidate box determined each time the loop is executed: sorting the initial behavior candidate box according to the behavior confidence, and determining a target behavior candidate box with the maximum behavior confidence in the initial behavior candidate box; obtaining a candidate box overlap threshold, and calculating an overlap degree of the target behavior candidate box and other behavior candidate boxes in the initial behavior candidate box sequence; performing confidence reduction processing on the confidence corresponding to the behavior candidate box in the other behavior candidate boxes whose overlap degree is greater than or equal to the candidate box overlap threshold, and deleting the behavior candidate box in the other behavior candidate boxes whose overlap degree is less than the candidate box overlap threshold to obtain a remaining behavior candidate box; and taking the remaining behavior candidate box as the initial behavior candidate box of the next loop.

[0011] In some example embodiments of the present disclosure, based on the foregoing scheme, the method further comprises: obtaining a multi-scale sample time feature map, and obtaining a sample behavior start time and a sample behavior end time of the initial time feature point by mapping the initial time feature point in the multi-scale sample time feature map to a sample video frame sequence corresponding to the multi-scale sample time feature map; calculating a sample left offset of the initial time feature point and the sample behavior start time, and a sample right offset of the initial time feature point and the sample behavior end time; calling an initial time prediction network and a target function corresponding to the initial time prediction network; training the initial time prediction network based on the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset, and using the trained initial time prediction network as a standard time prediction network when the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset satisfy the target function.

[0012] In some example embodiments of the present disclosure, based on the foregoing scheme, the method further comprises: obtaining a sample behavior label of the initial time feature point in the sample video frame sequence; determining a global time feature point corresponding to the initial time feature point within a time range of the sample behavior start time and the sample behavior end time, and labeling the global time feature point based on the sample behavior label to obtain a sample time feature point; obtaining an initial behavior label prediction network, and training the initial behavior label prediction network based on the sample time feature point to obtain the behavior label prediction network.

[0013] In some example embodiments of the present disclosure, based on the foregoing scheme, the labeling of the global time feature point based on the behavior label sample to obtain a sample time feature point comprises: obtaining a time boundary regulation parameter, and a left offset and a right offset of the global time feature point; labeling the global time feature point based on the sample behavior label, and screening a key time feature point from the global time feature point; the key time feature point is obtained by deleting a time feature point with a left offset and / or a right offset greater than the time boundary regulation parameter in the global time feature point; and using the key time feature point as a sample time feature point.

[0014] According to a second aspect of the embodiments of the present disclosure, a barrier identification model training apparatus is provided, comprising: a time feature map determination module configured to obtain a to-be-detected video frame sequence, and input the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map; a multi-scale feature map determination module configured to input the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map; an initial behavior candidate box generation module configured to input the multi-scale time feature map into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map; the initial behavior candidate box comprises a behavior confidence, and a behavior start prediction time and a behavior end prediction time; and a target behavior candidate box generation module configured to filter the initial behavior candidate box to obtain a target behavior candidate box of the to-be-detected video sequence.

[0015] In some example embodiments of the present disclosure, based on the foregoing scheme, the initial behavior candidate box generation module comprises an initial behavior candidate box generation unit configured to obtain a time feature point of the multi-scale time feature map by inputting the multi-scale time feature map into the standard time prediction network, and obtain a behavior start prediction time and a behavior end prediction time of the time feature point; obtain a behavior confidence of the time feature point based on the behavior label prediction network; and construct an initial behavior candidate box corresponding to the time feature point according to the behavior confidence, the behavior start prediction time and the behavior end prediction time.

[0016] In some example embodiments of the present disclosure, based on the foregoing scheme, the initial behavior candidate box generation module comprises a time calculation unit configured to obtain a left offset and a right offset of the time feature point output by the standard time prediction network; obtain a behavior start prediction time of the time feature point based on the time feature point and the left offset; and obtain a behavior end prediction time of the time feature point based on the time feature point and the right offset.

[0017] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior proposal generation module comprises a loop execution unit configured to perform the following loop process until all behavior proposals in the initial behavior proposal are traversed, and build a target behavior proposal set according to the target behavior proposal determined in each loop: sort the initial behavior proposals according to the behavior confidence, and determine a target behavior proposal with the maximum behavior confidence in the initial behavior proposals; obtain a proposal overlap threshold, and calculate the overlap of the target behavior proposal and other behavior proposals in the sequence of initial behavior proposals; perform confidence reduction processing on the confidence corresponding to the behavior proposals in the other behavior proposals with the overlap greater than or equal to the proposal overlap threshold, and delete the behavior proposals in the other behavior proposals with the overlap less than the proposal overlap threshold to obtain remaining behavior proposals; and take the remaining behavior proposals as the initial behavior proposals for the next loop.

[0018] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior proposal generation apparatus further comprises a standard time prediction network training module configured to obtain a multi-scale sample time feature map, and obtain a sample behavior start time and a sample behavior end time of an initial time feature point in the multi-scale sample time feature map by mapping the initial time feature point to a sequence of sample video frames corresponding to the multi-scale sample time feature map; calculate a sample left offset of the initial time feature point and the sample behavior start time, and a sample right offset of the initial time feature point and the sample behavior end time; call an initial time prediction network and a target function corresponding to the initial time prediction network; train the initial time prediction network based on the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset, and take the trained initial time prediction network as a standard time prediction network when the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset satisfy the target function.

[0019] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior candidate box generation apparatus further comprises a label prediction network training module, configured to obtain a sample behavior label of the initial time feature point in the sequence of sample video frames; determine a global time feature point corresponding to the initial time feature point within a time range of the sample behavior start time and the sample behavior end time, and label the global time feature point based on the sample behavior label to obtain a sample time feature point; obtain an initial behavior label prediction network, and train the initial behavior label prediction network based on the sample time feature point to obtain the behavior label prediction network.

[0020] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior candidate box generation apparatus comprises a sample time feature point determination module, configured to obtain a time boundary regulation parameter, and a left offset and a right offset of the global time feature point; label the global time feature point based on the sample behavior label, and select a key time feature point from the global time feature point; the key time feature point is obtained by deleting a time feature point with a left offset and / or a right offset greater than the time boundary regulation parameter in the global time feature point; and the key time feature point is taken as a sample time feature point.

[0021] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the target behavior candidate box generation method according to any one of the preceding aspects.

[0022] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executed by a processor to implement the target behavior candidate box generation method according to any one of the preceding aspects.

[0023] The technical solutions provided by the embodiments of the present disclosure can have the following beneficial effects:

[0024] The target behavior candidate box generation method in the example embodiment of the present disclosure, acquires a to-be-detected video frame sequence, and inputs the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map; inputs the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map; inputs the multi-scale time feature map into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map; the initial behavior candidate box includes a behavior confidence, a behavior start prediction time and a behavior end prediction time; and performs filtering processing on the initial behavior candidate box to obtain a target behavior candidate box of the to-be-detected video sequence. On the one hand, the multi-scale time feature map can be detected in parallel by the prediction network, thereby improving the detection efficiency of the to-be-detected video frame sequence; on the other hand, the initial behavior candidate box corresponding to the multi-scale time feature map, and the behavior prediction label, the behavior start prediction time and the behavior end prediction time of the initial behavior candidate box can be generated by the prediction network, thereby avoiding setting a sliding parameter manually, moving a preset window according to the sliding parameter to determine the initial behavior candidate box in the video frame sequence, and the behavior start time and the behavior end time of the initial behavior candidate box, and improving the time positioning accuracy of the initial behavior candidate box; and on the other hand, the target behavior candidate box of the to-be-detected video frame sequence can be determined by filtering the initial behavior candidate box, thereby improving the accuracy of identifying the behavior candidate box in the to-be-detected video frame sequence.

[0025] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0026] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. It is clear that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art. In the drawings:

[0027] Figure 1 The schematic diagram of the target behavior candidate box generation method flow according to some embodiments of the present disclosure is schematically shown;

[0028] Figure 2 The schematic diagram of the time change unit structure in the time feature pyramid network is exemplarily shown;

[0029] Figure 3 The schematic diagram of the initial behavior candidate box generation method flow according to some embodiments of the present disclosure is schematically shown;

[0030] Figure 4A schematic diagram of a network structure for predicting initial behavior candidate boxes is shown according to some embodiments of the present disclosure;

[0031] Figure 5 A schematic diagram of a time prediction method flow is shown according to some embodiments of the present disclosure;

[0032] Figure 6 A schematic diagram of an initial behavior candidate box filtering method flow is shown according to some embodiments of the present disclosure;

[0033] Figure 7 A schematic diagram of a standard time prediction network training method flow is shown according to some embodiments of the present disclosure;

[0034] Figure 8 A schematic diagram of a behavior label prediction network training method flow is shown according to some embodiments of the present disclosure;

[0035] Figure 9 A schematic diagram of a sample time feature point determination method flow is shown according to some embodiments of the present disclosure;

[0036] Figure 10 A schematic diagram of a target behavior candidate box generation apparatus is shown according to some embodiments of the present disclosure;

[0037] Figure 11 A schematic diagram of a computer system of an electronic device is shown according to some embodiments of the present disclosure;

[0038] Figure 12 A schematic diagram of a computer readable storage medium is shown according to some embodiments of the present disclosure.

[0039] In the drawings, identical or corresponding reference signs indicate identical or corresponding parts. DETAILED DESCRIPTION

[0040] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0041] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the

[0042] Moreover, the accompanying drawings are only schematic and are non-limiting exact representations of embodiments. The drawings in the present description are only schematic and are non-limiting exact representations of the described premises; they show only the essential features of the embodiments, the drawings themselves being non-limiting exact representations thereof. In the drawings, the size of some of the elements can be exaggerated, and the figures are not drawn to scale in order to distinctly show the elements. Identical or similar elements are denoted by the same reference numerals. Elements that have totally the same function are designated with the same reference signs.

[0043] In the present example embodiment, a target behavior candidate box generation method is first provided, which can be applied to a terminal device. Figure 1 An exemplary schematic diagram of a target behavior candidate box generation method flow according to some embodiments of the present disclosure is shown. Referring to Figure 1 As shown, the target behavior candidate box generation method can include the following steps:

[0044] In step S110, a sequence of video frames to be detected is obtained, and the sequence of video frames to be detected is input into a pre-trained feature extraction network to obtain a temporal feature map;

[0045] In step S120, the temporal feature map is input into a pre-trained temporal feature pyramid network to obtain a multi-scale temporal feature map;

[0046] In step S130, the multi-scale temporal feature map is input into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale temporal feature map; the initial behavior candidate box includes a behavior confidence, and a behavior start prediction time and a behavior end prediction time;

[0047] In step S140, the initial behavior candidate box is filtered to obtain a target behavior candidate box of the sequence of video frames to be detected.

[0048] According to the target behavior candidate box generation method in the example embodiment, on the one hand, the detection efficiency of the video frame sequence to be detected can be improved by parallel detection of the multi-scale time feature map through the prediction network; on the other hand, the initial behavior candidate box corresponding to the multi-scale time feature map, the behavior prediction label of the initial behavior candidate box, the behavior start prediction time, and the behavior termination prediction time can be generated through the prediction network, which avoids setting the sliding parameter manually, moving the preset window according to the sliding parameter to determine the initial behavior candidate box in the video frame sequence, and the behavior start time and the behavior termination time of the initial behavior candidate box, and improves the time positioning accuracy of the initial behavior candidate box; on the other hand, the target behavior candidate box of the video frame sequence to be detected can be determined by filtering the initial behavior candidate box, which improves the accuracy of identifying the behavior candidate box in the video frame sequence to be detected.

[0049] In the following, the target behavior candidate box generation method in the example embodiment will be further described.

[0050] In step S110, the video frame sequence to be detected is obtained, and the video frame sequence to be detected is input into the pre-trained feature extraction network to obtain a time feature map.

[0051] In some example embodiments of the present disclosure, the feature extraction network can refer to a network for extracting the spatio-temporal features of the video frame sequence to be detected. For example, the feature extraction network can be a network for extracting the spatio-temporal features of the video frame sequence to be detected, which includes a three-dimensional convolution layer, a maximum pooling layer, a full connection layer, and a normalized exponential function output layer. Of course, the feature extraction network can also be a network for extracting the spatio-temporal features of the video frame sequence to be detected, which includes other network structures, and the example embodiment does not make special limitations on this.

[0052] The time feature map can refer to a feature map obtained by performing spatial pooling processing on the spatio-temporal feature map output by the feature extraction network through the pooling layer. For example, the time feature map can be a feature image containing ROI (Region Of Interest, region of interest) region features obtained by performing spatial pooling processing on the spatio-temporal feature map output by the feature extraction network through the pooling layer. Of course, the time feature map can also be a feature vector matrix containing ROI (Region Of Interest, region of interest) region features, and the example embodiment does not make special limitations on this.

[0053] The spatio-temporal features of the video frame sequence to be detected can be extracted by a feature extraction network to obtain a spatio-temporal feature map, and the spatio-temporal feature map can be subjected to spatial pooling processing to obtain a time feature map. In this way, a batch normalization layer and a ReLU (Recitified Linear Units) activation layer can be added after each convolutional layer in the feature extraction network to avoid the distribution corresponding to the input of each layer of the neural network from shifting or moving during the training of the feature extraction network, thereby improving the convergence speed of the feature extraction network and further improving the training speed of the feature extraction network.

[0054] For example, the feature extraction network can be a network composed of 8 three-dimensional convolutional layers, 5 maximum pooling layers, 2 fully connected layers, and 1 normalized exponential function output layer. The first two pooling layers are located after the first two convolutional layers, the third pooling layer is located after the fourth convolutional layer, the fourth pooling layer is located after the sixth convolutional layer, and the fifth pooling layer is located after the eighth convolutional layer. The size of all three-dimensional convolution kernels is 3x3x3, the step length in the time domain and the spatial domain is 1, the number of channels of the first three-dimensional convolutional layer is 64, the number of channels of the second three-dimensional convolutional layer is 128, the number of channels of the third three-dimensional convolutional layer and the number of channels of the fourth three-dimensional convolutional layer are both 256, and the number of channels of the fifth three-dimensional convolutional layer, the number of channels of the sixth three-dimensional convolutional layer, the number of channels of the seventh three-dimensional convolutional layer, and the number of channels of the eighth three-dimensional convolutional layer are all 512. The size of the pooling kernel of the first pooling layer is 1x2x2, and the size of the pooling kernel of the remaining four pooling layers is all 2x2x2. If the size of the video frame sequence to be detected is T*H*W*3, the video frame sequence to be detected is input into the feature extraction network, and a spatio-temporal feature map of can be obtained. Wherein, T represents the number of frames of the video, H and W are the height and width of the image in the video respectively, and C is the number of channels of the output feature map. Different backbone networks have different numbers of output channels. After obtaining the spatio-temporal feature map of the video frame sequence, a three-dimensional deconvolution network can be used to up-sample the output feature map in the time dimension, and a spatio-temporal feature map of can be obtained. Finally, a two-dimensional average pooling layer can be used to pool the time feature map, and a time feature map of is finally obtained.

[0055] In step S120, the time feature map is input into a pre-trained time feature pyramid network to obtain a multi-scale time feature map.

[0056] In some example embodiments of the present disclosure, the time feature pyramid network can refer to a feature pyramid network for adjusting the receptive field of the time feature map to obtain time feature maps of different scales. For example, the time feature pyramid network can be a feature pyramid network that obtains time feature maps of different time scales by scaling the time feature map through multiple time change units.

[0057] The multi-scale time feature map can refer to a feature map corresponding to different receptive fields obtained by size transformation of the time feature map through the time feature pyramid network. For example, the multi-scale feature map can be a feature image corresponding to different receptive fields obtained by size transformation of the time feature map through the time feature pyramid network. The multi-scale feature map can also be a vector matrix corresponding to different receptive fields obtained by size transformation of the time feature map through the time feature pyramid network. This example embodiment does not make special limitations.

[0058] The time feature map obtained by performing spatial pooling processing on the spatio-temporal feature map output by the feature extraction network can be used as the bottom layer feature map of the time feature pyramid network, and the bottom layer feature map can be convoluted through the time change unit in the time feature pyramid network to obtain the first scale time feature map. Similarly, the first scale feature map can be used as the bottom layer network, and the first scale feature map can be convoluted through the time change unit in the time feature pyramid network to obtain the second scale time feature map, and a plurality of time feature maps of different scales can be obtained in turn.

[0059] Figure 2 An example of the time change unit in the time feature pyramid network is shown. The time change unit can be composed of 4 convolutional layers, and the convolution kernel of the 1st convolutional layer, the 2nd convolutional layer, and the 3rd convolutional layer can be 1*3, the channel number can be 256, and the step length can be 1. The convolution kernel of the 4th convolutional layer can be 1*3, the channel number can be 256, and the step length can be 2.

[0060] The time feature map can be scaled through the time change unit in the time feature pyramid network to obtain a multi-scale time feature map containing ROI region features.

[0061] In step S130, the multi-scale time feature map is input into the pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map; the initial behavior candidate box includes a behavior confidence, a behavior start prediction time, and a behavior end prediction time.

[0062] In some example embodiments of the present disclosure, the prediction network can refer to a special network for predicting the behavior candidate box in the multi-scale time feature map, and the behavior start time and the behavior end time of the behavior candidate box. For example, the prediction network can be a special network comprising a binary classification network and a regression network, wherein the binary classification network can be a network for predicting the foreground score and the background score of the behavior candidate box in the multi-scale time feature map, and the regression network can be a network for predicting the time left offset and the time right offset of the behavior candidate box in the multi-scale time feature map. Of course, the prediction network can also be a special network comprising other network structures and used for predicting the behavior candidate box in the multi-scale time feature map, and the behavior start time and the behavior end time of the behavior candidate box, which is not specially limited in the example embodiments.

[0063] The initial behavior candidate box can refer to a behavior candidate box corresponding to each time feature point in the multi-scale time feature map predicted by the prediction network. For example, the initial behavior candidate box can be a behavior candidate box obtained by merging the behavior confidence of each time feature point in the multi-scale time feature map predicted by the prediction network, and the behavior start prediction time and the behavior end prediction time of each time feature point. For example, the initial behavior candidate box can be a behavior candidate box comprising a behavior prediction confidence formed by a plurality of time feature points with the same behavior start prediction time and behavior end prediction time. Of course, the initial behavior candidate box can also be a behavior candidate box formed by time feature points with the same prediction time left offset and prediction time right offset, which is not specially limited in the example embodiments.

[0064] The behavior confidence can refer to a parameter for determining the behavior prediction label of the initial behavior candidate box. For example, the behavior confidence can be reference data for determining the behavior prediction label of the initial behavior candidate box, such as the predicted probability of the entity object represented by the initial behavior candidate box to perform the behavior. The behavior confidence can also be the foreground score and the background score for determining the behavior prediction label of the initial behavior candidate box. Of course, the behavior confidence can also be other parameters for determining the behavior prediction label of the initial behavior candidate box, which is not specially limited in the example embodiments.

[0065] The behavior start prediction time can refer to a predicted time calculated according to the time feature point in the multi-scale time feature map and the left offset of the time feature point. The behavior end prediction time can refer to a predicted time calculated according to the time feature point in the multi-scale time feature map and the right offset of the time feature point.

[0066] In step S140, the initial behavior candidate box is filtered to obtain the target behavior candidate box of the video sequence to be detected.

[0067] In some example embodiments of the present disclosure, the target behavior candidate box can refer to a behavior candidate box obtained by filtering the initial behavior candidate box. For example, the target behavior candidate box can be the behavior candidate box with the maximum behavior confidence in the initial behavior candidate box, i.e., the behavior candidate box with the maximum behavior confidence obtained by screening the initial behavior candidate box after sorting according to the behavior confidence, and reducing the confidence of other behavior candidate boxes in the initial behavior candidate box, and the target behavior candidate box can be the behavior candidate box with the maximum behavior confidence obtained by screening the other behavior candidate boxes after the confidence reduction.

[0068] Since one behavior candidate box can correspond to multiple scale time feature maps of different layers, after predicting the behavior confidence, the behavior start time and the behavior end time of the time feature points in the multiple scale time feature maps of different layers by the prediction network, the behavior candidate boxes corresponding to the multiple scale time feature maps of different layers can be obtained. The same behavior candidate boxes of the multiple scale feature maps of different layers can be merged by filtering the initial behavior candidate boxes, and then the target behavior candidate boxes corresponding to the multiple scale time feature maps can be obtained.

[0069] Figure 3 An illustrative diagram of a method flow of generating an initial behavior candidate box according to some embodiments of the present disclosure is shown. Referring to FIG. 3, the method of generating an initial behavior candidate box can include the following steps: Figure 3 As shown, the method of generating an initial behavior candidate box can include the following steps:

[0070] In step S310, the time feature points of the multiple scale time feature maps are obtained by inputting the multiple scale time feature maps into the standard time prediction network, and the behavior start prediction time and the behavior end prediction time of the time feature points are obtained.

[0071] In step S320, the behavior confidence of the time feature points is obtained based on the behavior label prediction network.

[0072] In step S330, the initial behavior candidate box corresponding to the time feature points is constructed according to the behavior confidence, the behavior start prediction time and the behavior end prediction time.

[0073] Wherein, the standard time prediction network can refer to a regression network for predicting the behavior start time and the behavior end time of the time feature points in the multiple scale time feature maps. The behavior label prediction network can refer to a binary classification network for the behavior label of the time feature points in the multiple scale time feature maps. The time feature point can refer to the interest point with time features in the multiple scale time feature maps.

[0074] The time feature points in the multi-scale time feature map can be extracted by a standard time prediction network in the prediction network, and the left offset and the right offset of the time feature points can be predicted, and the behavior start prediction time of the time feature points can be determined based on the time feature and the left offset of the time feature points, and the behavior end time of the time feature points can be determined based on the time feature and the right offset of the time feature points. In addition, the behavior confidence of the time feature points in the multi-scale time feature map can also be obtained by the behavior label prediction network in the prediction network, and the behavior label of the time feature points can be determined based on the behavior confidence. After obtaining the behavior confidence of the time feature points in the multi-scale time feature map, and the behavior start prediction time and the behavior end prediction time, the initial behavior candidate box can be constructed according to the time feature points with the same behavior start prediction time and behavior end prediction time and the same behavior confidence in the multi-scale time feature map, so as to detect and recognize the sequence of video frames to be detected.

[0075] Figure 4 The network structure for predicting the initial behavior candidate box is schematically shown according to some embodiments of the present disclosure. Referring to Figure 4 The network structure 400 for predicting the initial behavior candidate box mainly includes a sequence of video frames to be detected input module 410, a feature extraction network 420, a spatial pooling module 430, a time feature pyramid network 440, a prediction network 450, and a prediction result output module 460.

[0076] The sequence of video frames to be detected input module 410 is mainly used for inputting the sequence of video frames to be detected into the feature extraction network; the feature extraction network 420 is mainly used for extracting the space-time features of the sequence of video frames to be detected, and outputting the space-time feature map of the sequence of video frames to be detected; the spatial pooling module 430 is mainly used for performing pooling processing on the space-time feature map output by the feature extraction network, and outputting the time feature map of the sequence of video frames to be detected; the time feature pyramid network 440 is mainly used for changing the scale of the time feature map, and obtaining the multi-scale time feature map with different receptive fields; the prediction network 450 is mainly used for extracting the time feature points in the multi-scale time feature map, and predicting the behavior start time, the behavior end time and the behavior confidence of the time feature points; and the prediction result output module 460 is mainly used for inputting the predicted behavior confidence (foreground score and background score) of the time feature points in the multi-scale time feature map of different layers, and the behavior start prediction time (left offset) and the behavior end prediction time (right offset).

[0077] Preferably, the behavior candidate frame corresponding to the to-be-detected video frame sequence can be generated by the feature extraction network, the time feature pyramid network, and the prediction network, which breaks the limitation of the detection and recognition of the to-be-detected video frame sequence based on the sliding window or the combination method based on the behavior probability score, improves the detection efficiency of the to-be-detected video frame sequence, and through the prediction network, the behavior start time and the behavior end time of the behavior candidate frame in the multi-scale time feature map corresponding to the to-be-detected video frame sequence can be accurately positioned in time, thereby improving the detection accuracy of the to-be-detected video frame sequence.

[0078] Figure 5 A schematic diagram of a time prediction method flow according to some embodiments of the present disclosure is schematically shown. Referring to Figure 5 The time prediction method can include the following steps:

[0079] In step S510, the left offset and the right offset of the time feature point output by the standard time prediction network are obtained;

[0080] In step S520, the behavior start prediction time of the time feature point is calculated based on the time feature point and the left offset;

[0081] In step S530, the behavior end prediction time of the time feature point is calculated based on the time feature point and the right offset.

[0082] The left offset can refer to the time offset between the time feature point and the behavior start prediction time of the time feature point, and the right offset can refer to the time offset between the time feature point and the behavior end prediction time of the time feature point.

[0083] The time feature point in the multi-scale time feature map and the left offset and the right offset of each time feature point can be extracted by the standard time prediction network, and then the behavior start prediction time of the initial behavior candidate corresponding to the time feature point can be determined based on the time feature point and the left offset of the time feature point. Meanwhile, the behavior end prediction time of the initial behavior candidate frame corresponding to the time feature point can be determined based on the time feature point and the right offset of the time feature point.

[0084] For example, the time feature point in the multi-scale time feature map can be extracted by the standard time prediction network, and the left offset between the time feature point and the behavior start prediction time and the right offset between the time feature point and the behavior end prediction time can be predicted, and then the time difference between the time feature point and the left offset can be calculated, the time difference is taken as the behavior start prediction time of the time feature point, and the time sum between the time feature point and the right offset is calculated, and the time sum is taken as the behavior end prediction time of the time feature point.

[0085] After obtaining the time feature points in the multi-scale time feature map, and the behavior confidence, the behavior start prediction time, and the behavior end prediction time of each time feature point, the time feature points in the multi-scale time feature map can be merged, that is, the initial behavior candidate box is formed by the time points with the same behavior confidence, the behavior start prediction time, and the behavior end prediction time.

[0086] Figure 6 A schematic diagram of an initial behavior candidate box filtering method flow according to some embodiments of the present disclosure is schematically shown. Referring to Figure 6 As shown, the initial behavior candidate box filtering method can include the following steps:

[0087] In step S610, the following loop process is performed until all behavior candidate boxes in the initial behavior candidate box are traversed, and a target behavior candidate box set is constructed according to the target behavior candidate box determined each time the loop is determined;

[0088] In step S620, the initial behavior candidate boxes are sorted according to the behavior confidence, and a target behavior candidate box with the maximum behavior confidence in the initial behavior candidate box is determined;

[0089] In step S630, a candidate box overlap threshold is obtained, and the overlap of the target behavior candidate box and other behavior candidate boxes in the initial behavior candidate box sequence is calculated;

[0090] In step S640, the confidence of the behavior candidate box corresponding to the other behavior candidate box with the overlap greater than or equal to the candidate box overlap threshold is subjected to confidence reduction processing, and the other behavior candidate box with the overlap less than the candidate box overlap threshold is deleted, to obtain a remaining behavior candidate box; and

[0091] In step S650, the remaining behavior candidate box is taken as the initial behavior candidate box of the next loop.

[0092] Wherein, the target behavior candidate box set can refer to a set for storing target behavior candidate boxes. The candidate box overlap threshold can refer to a threshold of the time window overlap rate between each behavior candidate box in the initial behavior candidate box.

[0093] After obtaining the initial behavior candidate box set through the prediction network A confidence reduction strategy can be used to suppress redundant candidate boxes in the initial behavior candidate box. Wherein, φ P represents the initial behavior candidate box set, represents all behavior candidate boxes in the initial behavior candidate box set, t s represents the behavior start prediction time of each behavior candidate box, t ebehavior termination prediction time of each behavior candidate box, p f behavior confidence of each behavior candidate box. Specifically, all behavior candidate boxes in the initial behavior candidate box set can be sorted according to the behavior confidence, and the behavior candidate box with the highest behavior confidence in the initial behavior candidate box set is determined as a target behavior candidate box, and other behavior candidate boxes in the initial behavior candidate box set except the target behavior candidate box are processed according to the confidence reduction rule. The behavior confidence reduction rule of other behavior candidate boxes can be expressed by the calculation expression (1):

[0094]

[0095] wherein p f,i the behavior confidence of the behavior candidate box in the initial behavior candidate box set with the overlap degree less than the candidate box overlap degree threshold with the target behavior candidate box, the IOU (Intersection over Union) between the candidate box with the highest confidence and the remaining candidate boxes, i.e., the overlap degree of the time window between two candidate boxes, p' f,i the behavior confidence of the behavior candidate box after the behavior confidence reduction, ε can represent the parameter of the Gaussian function, and θ can represent the candidate box overlap degree threshold.

[0096] The target behavior candidate box with the highest behavior confidence can be extracted from the initial behavior candidate box set by performing a loop operation on the initial behavior candidate box, and the time window overlap degree between other behavior candidate boxes in the initial behavior candidate box set and the target behavior candidate box is calculated, the redundant behavior candidate boxes with the time window overlap degree less than the candidate box overlap degree threshold with the target behavior candidate box are deleted from other behavior candidate boxes, and the behavior confidence of the remaining behavior candidate boxes in other behavior candidate boxes is reduced according to the behavior candidate box confidence reduction rule, so as to select the behavior candidate box with the highest behavior confidence from the remaining behavior candidate boxes as the target behavior candidate box, and other behavior candidate boxes in the remaining behavior candidate boxes as the initial behavior candidate box. The above loop operation is performed until all behavior candidate boxes in the initial behavior candidate box set are traversed, and finally a target behavior candidate box set containing multiple target behavior candidate boxes can be obtained, realizing the filtering of the initial behavior candidate box corresponding to the multi-scale time feature map.

[0097] Figure 7 The schematic diagram schematically shows the standard time prediction network training method flow according to some embodiments of the present disclosure. Referring to Figure 7 , the standard time prediction network training method can include the following steps:

[0098] In step S710, a multi-scale sample time feature map is obtained, and sample behavior start time and sample behavior end time of the initial time feature point are obtained by mapping the initial time feature point in the multi-scale sample time feature map to a sample video frame sequence corresponding to the multi-scale sample time feature map.

[0099] In step S720, a sample left offset of the initial time feature point and the sample behavior start time, and a sample right offset of the initial time feature point and the sample behavior end time are calculated.

[0100] In step S730, an initial time prediction network and a target function corresponding to the initial time prediction network are called.

[0101] In step S740, the initial time prediction network is trained based on the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset, and when it is detected that the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset satisfy the target function, the trained initial time prediction network is used as a standard time prediction network.

[0102] The multi-scale sample time feature map can refer to a sample of a multi-scale time feature map output by a time feature pyramid. The initial time feature point can refer to a time feature point with behavior change in the multi-scale sample time feature map. The sample behavior start time can refer to the real behavior start time of the initial time feature point. The sample behavior end time can refer to the real behavior end time of the initial time feature point. The initial time prediction network can refer to a neural network for extracting the initial time feature point in the multi-scale sample time feature map and predicting the behavior start time and the behavior end time of the initial time feature point. The sample left offset can refer to the time offset between the initial time feature point and the sample behavior start time. The sample right offset can refer to the time offset between the initial time feature point and the sample behavior end time. The target function can refer to a function for measuring the training degree of the initial time prediction network.

[0103] The sample video frame sequence can be obtained, and the time features of the sample video frame sequence are extracted through a feature extraction network to obtain a time feature map, and the time feature map is scale transformed through a time feature pyramid to obtain a multi-scale time feature map, and initial time feature points in the multi-scale time feature map are extracted. By mapping the initial time feature points in the multi-scale time feature map to the sample video frame sequence, the real behavior start time and the real behavior end time of the initial time feature points are determined. Further, the real behavior start time of the initial time feature points can be collected as a sample behavior start time, and the real behavior end time of the initial time feature points can be collected as a sample behavior end time, and the left offset of the initial time point and the sample behavior start time, and the right offset of the initial time point and the sample behavior end time are calculated.

[0104] Thus, the initial time prediction network can be called, and the initial time prediction network can be trained based on the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset. By inputting the multi-scale test time feature map to the initial time prediction network, the time feature points of the multi-scale test time feature map output by the initial time prediction network, the behavior start prediction time, and the behavior end prediction time of the time feature points are determined. Further, the test left offset of the time feature points is determined according to the time feature points and the behavior start prediction time in the multi-scale test time feature map, and the test right offset is determined according to the time feature points and the behavior end prediction time in the multi-scale test time feature map. When the time feature points in the multi-scale test time feature map, and the test left offset and the test right offset satisfy the objective function of the initial time network, the initial time network after the current training is completed can be used as a standard time prediction network. The objective function of the initial time prediction network can be represented by the calculation expressions (2) and (3):

[0105]

[0106]

[0107] wherein S1 can represent the left offset between the time feature point and the behavior start prediction time, S2 can represent the right offset between the time feature point and the behavior end prediction time, t s The behavior start prediction time can be identified, t e The behavior end prediction time can be represented, I can identify the number of layers of the multi-scale time feature map output by the time feature pyramid, t can represent the time feature of the time feature point in the multi-scale time feature map, and λ can represent a scale parameter for controlling the network loss of the standard prediction time.

[0108] The offset of the initial time feature point in the sample video frame sequence can be determined by subtracting the sample behavior start time from the initial time feature point projected to the sample video frame sequence, and subtracting the sample behavior end time from the initial time feature point projected to the sample video frame sequence, and then projecting the initial time feature point back to the multi-scale sample time feature map, so that the offset of the initial time feature point near the receptive field of the multi-scale sample time feature map can be centered to about 1. After using the logarithmic operation, the offset of the initial time feature point near the receptive field of the multi-scale sample time feature map will be further pulled to about 0, reducing the prediction error of the offset of the initial time feature point in the multi-scale sample time feature map and improving the accuracy of the behavior start prediction time and the behavior end prediction time of the time feature point in the multi-scale time feature map.

[0109] Figure 8 A schematic diagram of a behavior label prediction network training method according to some embodiments of the present disclosure is shown schematically. Referring to Figure 8 The behavior label prediction network training method can include the following steps, as shown in the figure.

[0110] In step S810, the sample behavior label of the initial time feature point in the sample video frame sequence is obtained.

[0111] In step S820, the global time feature point corresponding to the initial time feature point within the time range of the sample behavior start time and the sample behavior end time is determined, and the global time feature point is labeled based on the sample behavior label to obtain a sample time feature point.

[0112] In step S830, an initial behavior label prediction network is obtained, and the initial behavior label prediction network is trained based on the sample time feature point to obtain the behavior label prediction network.

[0113] The sample behavior label can refer to the behavior label of the sample behavior candidate box corresponding to the initial behavior time point obtained by mapping the initial time point to the sample video frame sequence. The global time feature point can refer to all time feature points within the time range of the sample behavior start time and the sample behavior end time corresponding to the initial time feature point. The sample time feature point can refer to the time feature point obtained by labeling the global time feature point based on the sample behavior label of the initial time feature point.

[0114] The initial time feature point in the multi-scale sample time feature map can be mapped to a sample video frame sequence, the real behavior label of the initial time feature point is determined, and the global time feature points within the time range of the sample behavior start time and the sample behavior end time of the initial time feature point are labeled based on the real behavior label. The labeled global time feature points are used as positive samples for training of the behavior label prediction network. Wherein, the loss function L of the behavior prediction network can be expressed by the calculation expression (4). loc The calculation expression (4) can be expressed by the calculation expression (4).

[0115]

[0116] Wherein, a i The behavior label corresponding to the i-th sample can be represented, and the value range is {0, 1}, q i The output value of the behavior label prediction network after the normalization exponential function can be represented, and q i It is a two-dimensional vector, including the background score and the foreground score; The foreground score corresponding to the i-th sample can be represented; The background score corresponding to the i-th sample can be represented.

[0117] Preferably, by mapping the initial time feature point to the sample video frame sequence, the real behavior start time and the real behavior end time of the initial time feature point are determined, and the real behavior label is determined. The real behavior label is directly matched with the global time feature points within the time range of the real behavior start time and the real behavior end time, and the global time feature points matched successfully are used as positive samples. Avoiding the label assignment strategy based on the time overlap rate to define the positive samples and the corresponding behavior labels of the non-positive samples, the training speed of the behavior label network is improved; at the same time, in the process of detecting the behavior label of the time feature point in the multi-scale time feature map corresponding to the to-be-detected video frame sequence, the behavior label prediction network also does not need to predict the behavior label of the time feature point in the multi-scale feature map based on the time overlap rate, which improves the efficiency of predicting the behavior label of the time feature point in the multi-scale time feature map.

[0118] Figure 9 The schematic diagram of the sample time feature point determination method flow according to some embodiments of the present disclosure is shown schematically. Referring to Figure 9 The sample time feature point determination method can include the following steps:

[0119] In step S910, the time boundary regulation parameters, and the left offset and the right offset of the global time feature point are obtained;

[0120] In step S920, the global time feature points are labeled based on the sample behavior label, and key time feature points are selected from the global time feature points; the key time feature points are obtained by deleting time feature points with left offset and / or right offset greater than the time boundary control parameter from the global time feature points;

[0121] In step S930, the key time feature points are taken as sample time feature points.

[0122] The time boundary control parameter can refer to a parameter for controlling the time prediction range of the standard time prediction network. Since the size of the convolution sum of the time change unit in the time feature pyramid is fixed, the receptive field of different time feature points of the multi-scale time feature map at the same layer of the time feature pyramid network is the same, and the higher the convolution layer of the time feature pyramid network, the larger the receptive field of the time feature point of the multi-scale time feature map. A basic length can be assigned to each convolution layer of the time feature pyramid network, for example, the basic length can be the same as the time mapping parameter of the multi-scale feature map mapping to the video frame sequence to be detected. For example, the basic length of the Lth layer multi-scale time feature map can be represented by calculation expression (5).

[0123] L l =2 l+1 (5)

[0124] Wherein, L l may represent the basic length of the lth layer feature map.

[0125] The time boundary control parameter can be set based on the basic length of the multi-scale sample time feature map to filter out invalid time feature points with left offset and / or right offset exceeding the time boundary control parameter from the global time feature points in the multi-scale sample time feature map, so as to ensure that the left offset and right offset of the time feature point sample participating in the training of the standard time prediction network are near the basic length of the multi-scale time feature map, avoid training the standard time prediction network through the global time feature points in the multi-scale sample time feature map, and cause the standard time prediction network to have low accuracy, long training period and resource waste. The loss function of the standard time prediction network can be a loss function for boundary regression, such as a smooth first-order regularization loss function (Smooth L1), and the sum of the left offset loss and the right offset loss constitutes the boundary offset loss of the standard time prediction network. The loss function L loc of the standard time prediction network can be represented by calculation expression (4).

[0126]

[0127] Wherein, N posmay represent the number of samples in the multi-scale sample time feature map, r i 1 may represent the left offset of sample i in the multi-scale sample time feature map, may represent the left offset regression target data of sample i in the multi-scale sample time feature map, r i 2 may represent the right offset of sample i in the multi-scale sample time feature map, may represent the right offset regression target data of sample i in the multi-scale sample time feature map.

[0128] Preferably, whether the left offset and / or the right offset of the global time feature point in the multi-scale sample time feature map exceeds the time boundary control parameter can be detected, if the left offset and / or the right offset of the global time feature point exceeds the time boundary control parameter, the time feature point with the left offset and / or the right offset exceeding the time boundary control parameter can be regarded as an invalid time feature point, and the invalid time feature point is deleted from the global time feature point, the remaining time feature points in the global time feature point are regarded as sample time feature points, and the standard time prediction network is trained based on the valid sample time feature points, so as to improve the accuracy of the standard time prediction network.

[0129] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0130] In addition, in the present example embodiment, a target behavior candidate box generation apparatus is also provided. Referring to Figure 10As shown, the target behavior candidate box generation apparatus 1000 comprises: a time feature map determination module 1010, a multi-scale feature map determination module 1020, an initial behavior candidate box generation module 1030, and a target behavior candidate box generation module 1040. The time feature map determination module 1010 is configured to obtain a to-be-detected video frame sequence, and input the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map. The multi-scale feature map determination module 1020 is configured to input the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map. The initial behavior candidate box generation module 1030 is configured to input the multi-scale time feature map into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map. The initial behavior candidate box comprises a behavior confidence, a behavior start prediction time, and a behavior end prediction time. The target behavior candidate box generation module 1040 is configured to filter the initial behavior candidate box to obtain a target behavior candidate box of the to-be-detected video sequence.

[0131] In some example embodiments of the present disclosure, based on the foregoing scheme, the initial behavior candidate box generation module 1030 comprises an initial behavior candidate box generation unit configured to obtain a time feature point of the multi-scale time feature map and a behavior start prediction time and a behavior end prediction time of the time feature point by inputting the multi-scale time feature map into the standard time prediction network. The behavior label prediction network is used to obtain a behavior confidence of the time feature point. An initial behavior candidate box corresponding to the time feature point is constructed according to the behavior confidence, the behavior start prediction time, and the behavior end prediction time.

[0132] In some example embodiments of the present disclosure, based on the foregoing scheme, the initial behavior candidate box generation module 1030 comprises a time calculation unit configured to obtain a left offset and a right offset of the time feature point output by the standard time prediction network. The behavior start prediction time of the time feature point is calculated based on the time feature point and the left offset. The behavior end prediction time of the time feature point is calculated based on the time feature point and the right offset.

[0133] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior proposal generation module 1040 comprises a loop execution unit configured to execute the following loop process until all behavior proposals in the initial behavior proposal are traversed, and build a target behavior proposal set according to the target behavior proposal determined in each loop: sort the initial behavior proposals according to the behavior confidence, and determine the target behavior proposal with the maximum behavior confidence in the initial behavior proposals; obtain a proposal overlap threshold, and calculate the overlap of the target behavior proposal and other behavior proposals in the sequence of initial behavior proposals; perform confidence reduction processing on the confidence corresponding to the behavior proposals in the other behavior proposals with the overlap greater than or equal to the proposal overlap threshold, and delete the behavior proposals in the other behavior proposals with the overlap less than the proposal overlap threshold, to obtain remaining behavior proposals; and take the remaining behavior proposals as the initial behavior proposals for the next loop.

[0134] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior proposal device 1000 further comprises a standard time prediction network training module configured to obtain a multi-scale sample time feature map, and obtain a sample behavior start time and a sample behavior end time of an initial time feature point in the multi-scale sample time feature map by mapping the initial time feature point to a sequence of sample video frames corresponding to the multi-scale sample time feature map; calculate a sample left offset of the initial time feature point and the sample behavior start time, and a sample right offset of the initial time feature point and the sample behavior end time; call an initial time prediction network and a target function corresponding to the initial time prediction network; train the initial time prediction network based on the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset, and when it is detected that the sample behavior start time, the sample behavior end time, the sample left offset, and the sample right offset satisfy the target function, take the trained initial time prediction network as a standard time prediction network.

[0135] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior candidate box generation apparatus 1000 further comprises a label prediction network training module, configured to obtain a sample behavior label of the initial time feature point in the sequence of sample video frames; determine a global time feature point corresponding to the initial time feature point within a time range of the sample behavior start time and the sample behavior end time, and label the global time feature point based on the sample behavior label to obtain a sample time feature point; obtain an initial behavior label prediction network, and train the initial behavior label prediction network based on the sample time feature point to obtain the behavior label prediction network.

[0136] In some example embodiments of the present disclosure, based on the foregoing scheme, the target behavior candidate box generation apparatus 1000 comprises a sample time feature point determination module, configured to obtain a time boundary regulation parameter, and a left offset and a right offset of the global time feature point; label the global time feature point based on the sample behavior label, and select a key time feature point from the global time feature point; the key time feature point is obtained by deleting a time feature point whose left offset and right offset are both greater than the time boundary regulation parameter; and take the key time feature point as a sample time feature point.

[0137] The specific details of the modules of the target behavior candidate box generation apparatus described above have been described in detail in the corresponding target behavior candidate box generation method, and therefore will not be described here again.

[0138] It should be noted that although several modules or units of the target behavior candidate box generation apparatus are mentioned in the foregoing detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into embodied by multiple modules or units.

[0139] In addition, in the example embodiments of the present disclosure, an electronic device capable of implementing the target behavior candidate box generation method described above is also provided.

[0140] Those skilled in the art can understand that each aspect of the present disclosure can be implemented as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as “circuitry”, “module” or “system” here.

[0141] The following reference Figure 11 To describe an electronic device 1100 according to such an embodiment of the present disclosure. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0142] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0143] The storage unit stores program code that can be executed by the processing unit 1110, causing the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1110 can perform actions such as... Figure 1 In step S110, the video frame sequence to be detected is obtained and input into a pre-trained feature extraction network to obtain a temporal feature map; in step S120, the temporal feature map is input into a pre-trained temporal feature pyramid network to obtain a multi-scale temporal feature map; in step S130, the multi-scale temporal feature map is input into a pre-trained prediction network to obtain an initial action candidate box corresponding to the multi-scale temporal feature map; the initial action candidate box includes action confidence, action start prediction time, and action end prediction time; in step S140, the initial action candidate box is filtered to obtain the target action candidate box of the video sequence to be detected.

[0144] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1121 and / or cache memory 1122, and may further include read-only memory (ROM) 1123.

[0145] Storage unit 1120 may also include a program / utility 1124 having a set (at least one) program module 1125, such program module 1125 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0146] Bus 1130 can be one or more of several types of bus structure including a memory bus or memory controller, a peripheral bus, a graphics bus, a processor or local bus using any of a variety of bus architectures.

[0147] Electronic device 1100 can also communicate with one or more external devices 1170 such as a keyboard or pointing device, a Bluetooth device, etc.; other devices associated with electronic device 1100; and / or one or more devices that enable provide user access to electronic device 1100 and / or one or more devices that enable electronic device 1100 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface 1150. Still yet, electronic device 1100 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 1160. As depicted, network adapter 1160 communicates with the other components of electronic device 1100 via bus 1130. It should be appreciated that although not shown, other hardware and / or software components could be used in conjunction with electronic device 1100. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0148] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by hardware coupled with software, as described above. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0149] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores a program product capable of implementing the method described above. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps described in the “example method” section of the present disclosure according to various example embodiments of the present disclosure when the program product is run on the terminal device.

[0150] Reference Figure 12As shown, a program product 1200 for implementing the above-mentioned target behavior candidate box generation method according to embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and include program codes, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in the present document, the readable storage medium can be any tangible medium containing or storing a program, which can be used or combined with an instruction execution system, device or apparatus.

[0151] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0152] The computer readable signal medium can include a data signal borne in a baseband or as part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, device or apparatus.

[0153] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0154] Program code to implement an operation of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0155] In addition, the above-described flowcharts are merely illustrative of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to limit the purpose. It is easily understood that the processes shown in the above-described flowcharts do not indicate or limit the time sequence of the processes. In addition, it is also easily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0156] Through the above description of the embodiments, it is easily understood by those skilled in the art that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0157] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0158] It is to be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be effected therein by those skilled in the art without departing from the scope of the application. The scope of the application should be limited only by the appended claims.

Claims

1. A method for generating a target behavior bounding box, characterized in that, The method comprises: obtaining a to-be-detected video frame sequence, and inputting the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map; the time feature map is a feature map obtained by performing spatial pooling processing on a space-time feature map output by the feature extraction network through a pooling layer; inputting the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map; inputting the multi-scale time feature map into a pre-trained prediction network to obtain an initial behavior candidate box corresponding to the multi-scale time feature map; the initial behavior candidate box comprises a behavior confidence, a behavior start prediction time and a behavior end prediction time; the obtaining of the initial behavior candidate box corresponding to the multi-scale time feature map comprises: obtaining a left offset and a right offset of a time feature point output by a standard time prediction network; based on the time feature point and the left offset, a behavior start prediction time of the time feature point is calculated; based on the time feature point and the right offset, a behavior end prediction time of the time feature point is calculated; performing filtering processing on the initial behavior candidate box to obtain a target behavior candidate box of the to-be-detected video sequence; the method further comprises: obtaining a multi-scale sample time feature map, and obtaining a sample behavior start time and a sample behavior end time of an initial time feature point in the multi-scale sample time feature map by mapping the initial time feature point to a sample video frame sequence corresponding to the multi-scale sample time feature map; calculating a sample left offset of the initial time feature point and the sample behavior start time, and a sample right offset of the initial time feature point and the sample behavior end time; calling an initial time prediction network and a target function corresponding to the initial time prediction network; training the initial time prediction network based on the sample behavior start time, the sample behavior end time, the sample left offset and the sample right offset, and using the initial time prediction network trained when the sample behavior start time, the sample behavior end time, the sample left offset and the sample right offset meet the target function as a standard time prediction network.

2. The target behavior candidate frame generation method according to claim 1, characterized in that, The prediction network comprises a standard time prediction network and a behavior label prediction network, and the inputting of the multi-scale time feature map into the pre-trained prediction network to obtain the initial behavior candidate box corresponding to the multi-scale time feature map comprises: by inputting the multi-scale time feature map into the standard time prediction network, obtaining a time feature point of the multi-scale time feature map, and obtaining a behavior start prediction time and a behavior end prediction time of the time feature point; obtaining a behavior confidence of the time feature point based on the behavior label prediction network; constructing an initial behavior candidate box corresponding to the time feature point according to the behavior confidence, the behavior start prediction time and the behavior end prediction time. 3.The target behavior candidate box generation method of claim 1, wherein, the filtering processing of the initial behavior candidate box to obtain the target behavior candidate box of the to-be-detected video sequence comprises: performing the following loop process until all action candidate boxes in the initial action candidate box are traversed, and constructing an action candidate box set according to the target action candidate box determined in each loop; sorting the initial action candidate boxes according to the action confidence, and determining a target action candidate box with the maximum action confidence in the initial action candidate box; obtaining a candidate box overlap threshold, and calculating the overlap degree of the target action candidate box and other action candidate boxes in the initial action candidate box sequence; performing confidence reduction processing on the confidence corresponding to the action candidate box with the overlap degree greater than or equal to the candidate box overlap threshold in the other action candidate boxes, and deleting the action candidate box with the overlap degree less than the candidate box overlap threshold in the other action candidate boxes, to obtain a remaining action candidate box; and taking the remaining action candidate box as the initial action candidate box of the next loop.

4. The target behavior candidate frame generation method according to claim 2, characterized in that, The method further comprises: obtaining a sample action label of the initial time feature point in the sample video frame sequence; determining a global time feature point corresponding to the initial time feature point within the time range of the sample action start time and the sample action end time, and labeling the global time feature point based on the sample action label to obtain a sample time feature point; obtaining an initial action label prediction network, and training the initial action label prediction network based on the sample time feature point to obtain the action label prediction network.

5. The target behavior candidate frame generation method according to claim 4, characterized in that, The labeling of the global time feature point based on the action label sample to obtain a sample time feature point comprises: obtaining a time boundary regulation parameter, and a left offset and a right offset of the global time feature point; labeling the global time feature point based on the sample action label, and screening a key time feature point from the global time feature point; the key time feature point is obtained by deleting the time feature point with the left offset and / or the right offset greater than the time boundary regulation parameter in the global time feature point; taking the key time feature point as a sample time feature point.

6. A target behavior candidate frame generation apparatus characterized by comprising: comprises: a time feature map determination module, configured to obtain a to-be-detected video frame sequence, and input the to-be-detected video frame sequence into a pre-trained feature extraction network to obtain a time feature map; the time feature map is a feature map obtained by performing spatial pooling processing on a space-time feature map output by the feature extraction network through a pooling layer; a multi-scale feature map determination module, configured to input the time feature map into a pre-trained time feature pyramid network to obtain a multi-scale time feature map; an initial action candidate box generation module, configured to input the multi-scale time feature map into a pre-trained prediction network to obtain an initial action candidate box corresponding to the multi-scale time feature map; the initial action candidate box comprises an action confidence, an action start prediction time, and an action end prediction time; the initial action candidate box corresponding to the multi-scale time feature map comprises being configured to obtain a left offset and a right offset of a time feature point output by a standard time prediction network. Based on the time feature point and the left offset, a behavior start prediction time of the time feature point is calculated; Based on the time feature point and the right offset, a behavior end prediction time of the time feature point is calculated; The target behavior candidate box generation module is configured to filter the initial behavior candidate boxes to obtain target behavior candidate boxes of the video sequence to be detected. The device is further configured to: obtain a multi-scale sample time feature map, and map initial time feature points in the multi-scale sample time feature map to a sample video frame sequence corresponding to the multi-scale sample time feature map to obtain sample behavior start times and sample behavior end times of the initial time feature points; calculate sample left offsets of the initial time feature points and the sample behavior start times, and sample right offsets of the initial time feature points and the sample behavior end times; invoke an initial time prediction network and a target function corresponding to the initial time prediction network; train the initial time prediction network based on the sample behavior start times, the sample behavior end times, the sample left offsets and the sample right offsets, and use the trained initial time prediction network as a standard time prediction network when the sample behavior start times, the sample behavior end times, the sample left offsets and the sample right offsets satisfy the target function.

7. An electronic device, comprising: comprise: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the target behavior candidate box generation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, a computer program stored thereon, the computer program being executed by a processor to implement the target behavior candidate box generation method according to any one of claims 1 to 5.

Images