Video processing method and device, electronic device and storage medium

By performing multi-level feature extraction and upsampling on the video stream, combined with temporal flipping and feature sequence fusion, the problem of low accuracy in temporal nomination detection technology is solved, and the accuracy and confidence of temporal action nomination are improved.

CN112183588BActive Publication Date: 2024-11-26SHANGHAI SENSETIME INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010955752.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-11
Publication Date
2024-11-26
Estimated Expiration
2040-09-11

AI Technical Summary

Technical Problem

Existing temporal nomination detection techniques have low accuracy in selecting temporal action nominations containing target actions from massive amounts of video data, particularly regarding temporal boundary probability sequences.

Method used

By performing n-level feature extraction and m-level upsampling on the video stream, combined with temporal reversal and feature sequence fusion, the first and second temporal boundary probability sequences are obtained, and temporal action nominations are determined by similarity and confidence calculations.

Benefits of technology

It improves the accuracy of temporal boundary probability sequences and the confidence of temporal action nominations, thereby enhancing the accuracy of identifying target actions from videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112183588B_ABST
    Figure CN112183588B_ABST
Patent Text Reader

Abstract

The present application discloses a video processing method and device, an electronic device and a storage medium. The method comprises: obtaining a video stream to be processed; performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence; performing m-level upsampling processing on the first intermediate feature sequence to obtain a first temporal boundary probability sequence of a first action; the first temporal boundary probability sequence includes the probability that an image in the video stream to be processed is a start frame of the first action, and the probability that an image in the video stream to be processed is an end frame of the first action.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video processing method and device, an electronic device, and a storage medium. Background Art

[0002] With the explosive growth of the number of various videos, how to select temporal action nominations containing target actions from massive videos is of great significance.

[0003] By applying the temporal nomination detection technology to process the video, the temporal boundary probability sequence of the video can be obtained, where the temporal boundary probability sequence includes the probability that each frame image in the video is the starting frame of the temporal action nomination, and the probability that each frame image is the ending frame of the temporal action nomination. Then, the temporal action nomination containing the target action can be found from the massive video based on the temporal boundary probability sequence. However, the accuracy of the temporal boundary probability sequence obtained by the current temporal nomination detection technology is low. Summary of the invention

[0004] The present application provides a video processing method and device, an electronic device and a storage medium.

[0005] In a first aspect, a video processing method is provided, the method comprising:

[0006] Get the video stream to be processed;

[0007] Performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence;

[0008] Performing m-level upsampling processing on the first intermediate feature sequence to obtain a first temporal boundary probability sequence of the first action; the first temporal boundary probability sequence includes a probability that the image in the to-be-processed video stream is a start frame of the first action, and a probability that the image in the to-be-processed video stream is an end frame of the first action;

[0009] The input data of the i+1th level upsampling process in the m-level upsampling process is the fusion data of the output data of the i-th level upsampling process in the m-level upsampling process and the input data of the j-th level feature extraction process in the n-level feature extraction process; the resolution of the output data of the i-th level upsampling process and the input data of the j-th level feature extraction process are the same; i is a positive integer less than m, and j is a positive integer not greater than n.

[0010] In combination with any implementation manner of the present application, before performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence, the method further includes:

[0011] Performing feature extraction processing on the video stream to be processed to obtain a first feature sequence; the first feature sequence corresponds to semantic information of the video stream to be processed and optical flow feature information of a background in the video stream to be processed; the background includes content in the video stream to be processed except the first action;

[0012] The step of performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence includes:

[0013] The n-level feature extraction process is performed on the first feature sequence to obtain the first intermediate feature sequence.

[0014] In combination with any embodiment of the present application, the method further includes:

[0015] Performing time sequence reversal processing on the first feature sequence to obtain a second feature sequence whose playing order is opposite to the first feature sequence;

[0016] Performing p-level feature extraction processing on the second feature sequence to obtain a second intermediate feature sequence;

[0017] Performing q-level upsampling processing on the second intermediate feature sequence to obtain a second temporal boundary probability sequence of the first action; the second temporal boundary probability sequence includes a probability that an image in the video stream to be processed is a start frame of the first action, and a probability that an image in the video stream to be processed is an end frame of the first action;

[0018] The input data of the r+1th level upsampling process in the qth level upsampling process is the fusion data of the output data of the rth level upsampling process in the qth level upsampling process and the input data of the sth level feature extraction process in the pth level feature extraction process; the output data of the rth level upsampling process and the input data of the sth level feature extraction process have the same resolution; r is a positive integer less than q, and s is a positive integer not greater than p;

[0019] The at least one first temporal action nomination is obtained according to the first temporal boundary probability sequence and the second temporal boundary probability sequence.

[0020] In combination with any embodiment of the present application, the input data of the j-th level feature extraction processing includes shallow feature information of the video stream to be processed; the s-th level feature extraction processing includes shallow feature information of the first feature sequence.

[0021] In combination with any implementation manner of the present application, obtaining at least one first time sequence action nomination for the first action according to the time sequence boundary probability sequence of the first action and the second time sequence boundary probability sequence includes:

[0022] fusing the first time series boundary probability sequence and the second time series boundary probability sequence to obtain a third time series boundary probability sequence of the first action;

[0023] According to the third temporal boundary probability sequence, the at least one first temporal action nomination is obtained.

[0024] In combination with any implementation manner of the present application, the at least one first time-sequence action nomination includes a second time-sequence action nomination, and the method further includes:

[0025] Extracting a feature sequence nominated by the second time-series action from the first feature sequence as a third feature sequence; the third feature sequence includes: a first channel feature sequence and a second channel feature sequence;

[0026] Determine the similarity between the first channel feature sequence and the second channel feature sequence to obtain a first similarity;

[0027] The confidence of the second time-series action nomination is obtained according to the first similarity and the third feature sequence.

[0028] In combination with any embodiment of the present application, the determining the similarity between the first channel feature data and the second channel feature data to obtain a first similarity includes:

[0029] Determine the matrix product of the first channel feature sequence and the second channel feature sequence to obtain a fourth feature sequence as the first similarity.

[0030] In combination with any implementation manner of the present application, the at least one first time-sequence action nomination further includes a third time-sequence action nomination different from the second time-sequence action nomination;

[0031] Before obtaining the confidence of the second time-series action nomination based on the first similarity and the third feature sequence, the method further includes:

[0032] Determine the similarity between the second time-sequence action nomination and the third time-sequence action nomination to obtain a second similarity;

[0033] The obtaining, based on the first similarity and the third feature sequence, the confidence of the second time-sequence action nomination includes:

[0034] The confidence of the second time-series action nomination is obtained according to the first similarity, the second similarity and the third feature sequence.

[0035] In combination with any embodiment of the present application, extracting the feature sequence of the second time-series action nomination from the first feature sequence as the third feature sequence includes:

[0036] Extracting the third feature sequence and the feature sequence nominated by the third sequential action from the first feature sequence to obtain a fifth feature sequence;

[0037] The determining the similarity between the first time-sequence action nomination and the second time-sequence action nomination to obtain a second similarity includes:

[0038] The matrix product of the fifth characteristic sequence and the sixth characteristic sequence is determined to obtain a seventh characteristic sequence as the second similarity; the sixth characteristic sequence is the transpose of the fifth characteristic sequence.

[0039] In combination with any implementation manner of the present application, obtaining the confidence of the second time-series action nomination according to the first similarity, the second similarity and the third feature sequence includes:

[0040] Performing a first feature extraction process on the first fused feature sequence to obtain a first confidence level of the second time-series action nomination; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence;

[0041] Performing a second feature extraction process on the second fused feature sequence to obtain a second confidence level of the second time-series action nomination; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence;

[0042] An average of the first confidence and the second confidence is determined as the confidence of the second time-series action nomination.

[0043] In combination with any embodiment of the present application, the first feature extraction process is implemented by u-level convolution processing, and the second feature extraction process is implemented by v-level convolution processing; both u and v are integers greater than 1;

[0044] Before determining the average of the first confidence level and the second confidence level as the confidence level of the first time-series action nomination, the method further includes:

[0045] The first feature map and the second feature map are merged to obtain a third feature map; the first feature map is output data of the c-th level convolution process in the u-level convolution process, and the second feature map is output data of the d-th level convolution process in the v-level convolution process; the size of the first feature map is the same as the size of the second feature map; c is a positive integer less than u, and d is a positive integer less than v;

[0046] Performing a third feature extraction process on the third feature graph to obtain a third confidence level of the second time-series action nomination;

[0047] The step of determining the average of the first confidence level and the second confidence level as the confidence level of the first sequential action nomination includes:

[0048] An average of the first confidence, the second confidence, and the third confidence is determined as the confidence of the first sequential action nomination.

[0049] In combination with any implementation manner of the present application, obtaining the confidence of the second time-series action nomination according to the first similarity, the second similarity and the third feature sequence includes:

[0050] Performing feature extraction processing on the first fused feature sequence to obtain a first reference feature sequence; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence;

[0051] Performing feature extraction processing on the second fused feature sequence to obtain a second reference feature sequence; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence;

[0052] fusing the first reference feature sequence and the second reference feature sequence to obtain a third reference feature sequence;

[0053] Perform feature extraction processing on the third reference feature sequence to obtain the confidence level of the second time-series action nomination.

[0054] In combination with any embodiment of the present application, the method further includes:

[0055] Sorting the at least one first time-sequence action nomination in descending order of quality score to obtain a time-sequence action nomination order of the first action; the quality score is obtained according to the time-sequence boundary probability sequence of the first time-sequence action nomination and the confidence of the first time-sequence action nomination;

[0056] The at least one first time-sequence action nomination is displayed according to the time-sequence action nomination order of the first action.

[0057] In combination with any embodiment of the present application, the video processing method is applied to a temporal action nomination network, and the training process of the temporal action nomination network includes:

[0058] Acquire a first training video stream and a neural network to be trained; the neural network to be trained includes a first feature extraction module and a first upsampling module;

[0059] Using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence;

[0060] Using the first upsampling module to perform b-level upsampling processing on the third intermediate feature sequence to obtain a fourth temporal boundary probability sequence of the first action; the fourth temporal boundary probability sequence includes a probability that the image in the first training video stream is a start frame of the first action, and a probability that the image in the first training video stream is an end frame of the first action;

[0061] The input data of the k+1th level upsampling process in the b-level feature extraction process is the fusion data of the output data of the kth level upsampling process in the b-level upsampling process and the input data of the f-th level feature extraction process in the a-level feature extraction process; the output data of the kth level upsampling process and the input data of the f-th level feature extraction process have the same resolution; k is a positive integer less than b, and f is a positive integer not greater than a;

[0062] Obtaining the loss of the neural network to be trained according to a difference between the fourth temporal boundary probability sequence and a label of the first training video stream; the label of the first training video stream includes a temporal action nomination of the first action in the first training video stream;

[0063] The parameters of the neural network to be trained are updated based on the loss of the neural network to be trained to obtain the temporal action nomination network.

[0064] In combination with any embodiment of the present application, the neural network to be trained further includes a basic feature extraction module;

[0065] Before using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence, the training process further includes:

[0066] Using the basic feature extraction module to perform feature extraction processing on the first training video stream, obtaining an eighth feature sequence; the eighth feature sequence corresponds to the semantic information of the first training video stream and the optical flow feature information of the background in the first training video stream; the background includes the content in the first training video stream except the first action;

[0067] The step of performing level A feature extraction processing on the first training video stream using the first feature extraction module to obtain a third intermediate feature sequence includes:

[0068] The first feature extraction module is used to perform a-level feature extraction processing on the seven feature sequences to obtain the third intermediate feature sequence.

[0069] In combination with any embodiment of the present application, the neural network to be trained further includes a second feature extraction module and a second upsampling module;

[0070] Before obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream, the training process further includes:

[0071] Performing time sequence reversal processing on the eighth characteristic sequence to obtain a ninth characteristic sequence whose playing order is opposite to that of the eighth characteristic sequence;

[0072] Using the second feature extraction module to perform x-level feature extraction processing on the ninth feature sequence to obtain a fourth intermediate feature sequence;

[0073] Using the second upsampling module to perform y-level upsampling processing on the second intermediate feature sequence, to obtain a fifth temporal boundary probability sequence of the first action; the fifth temporal boundary probability sequence includes a probability that an image in the first training video stream is a start frame of the first action, and a probability that an image in the first training video stream is an end frame of the first action; the playback order of the first training video stream is opposite to the playback order of the first training video stream;

[0074] The input data of the w+1th level upsampling process in the x-level upsampling process is the fusion data of the output data of the wth level upsampling process in the x-level upsampling process and the input data of the eth level feature extraction process in the y-level feature extraction process; the output data of the w-level upsampling process and the input data of the e-level feature extraction process have the same resolution; e is a positive integer less than y, and w is a positive integer not greater than x;

[0075] The neural network to be trained fuses the fourth time series boundary probability sequence and the fifth time series boundary probability sequence to obtain a sixth time series boundary probability sequence of the first action;

[0076] The step of obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream includes:

[0077] The loss of the neural network to be trained is obtained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream.

[0078] In combination with any implementation manner of the present application, the neural network to be trained further includes a nomination relationship module;

[0079] Before obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream, the training process further includes:

[0080] The neural network to be trained extracts a feature sequence of a fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence; the fourth temporal action nomination is a temporal action nomination of the first action obtained according to the sixth temporal boundary probability sequence; the tenth feature sequence includes: a third channel feature sequence and a fourth channel feature sequence;

[0081] Determine the similarity between the third channel feature sequence and the fourth channel feature sequence using the nominated relationship module to obtain a third similarity;

[0082] The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence;

[0083] The step of obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream includes:

[0084] Obtaining a first loss according to a difference between the sixth temporal boundary probability sequence and the label of the first training video stream;

[0085] Obtaining a second loss according to a difference between a confidence of the fourth temporal action nomination and a label of the first training video stream;

[0086] The sum of the first loss and the second loss is used as the loss of the neural network to be trained.

[0087] In combination with any embodiment of the present application, the using the nominated relationship module to determine the similarity between the third channel feature sequence and the fourth channel feature sequence to obtain a third similarity includes:

[0088] The nominated relationship module is used to determine the matrix product of the third channel feature sequence and the fourth channel feature sequence to obtain an eleventh feature sequence as the third similarity.

[0089] In combination with any implementation manner of the present application, before the nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence, the training process further includes:

[0090] The nomination relationship module determines the similarity between the fourth time-series action nomination and the fifth time-series action nomination to obtain a fourth similarity; the fifth time-series action nomination is a time-series action nomination of the first action obtained according to the sixth time-series boundary probability sequence, and the fifth time-series action nomination is different from the fourth time-series action nomination;

[0091] The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence, including:

[0092] The confidence of the fourth time-series action nomination is obtained according to the third similarity, the fourth similarity and the tenth feature sequence.

[0093] In combination with any embodiment of the present application, the neural network to be trained extracts the feature sequence of the fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence, including:

[0094] The neural network to be trained extracts the tenth feature sequence and the feature sequence nominated by the fifth time-series action from the eighth feature sequence to obtain a twelfth feature sequence;

[0095] The nomination relationship module determines the similarity between the fourth time-sequence action nomination and the fifth time-sequence action nomination to obtain a fourth similarity, including:

[0096] The nomination relationship module determines the matrix product of the twelfth feature sequence and the thirteenth feature sequence to obtain a fourteenth feature sequence as the fourth similarity; the thirteenth feature sequence is the transpose of the twelfth feature sequence.

[0097] In combination with any implementation manner of the present application, obtaining the confidence of the fourth time-series action nomination based on the third similarity, the fourth similarity and the tenth feature sequence includes:

[0098] Performing a fourth feature extraction process on the third fused feature sequence to obtain a fourth confidence level of the fourth time-series action nomination; the third fused feature sequence, the tenth feature sequence and the eleventh feature sequence are obtained;

[0099] Performing a fifth feature extraction process on the fourth fused feature sequence to obtain a fifth confidence level of the fourth time-series action nomination; the tenth feature sequence and the fourteenth feature sequence of the fourth fused feature sequence are obtained;

[0100] An average of the fourth confidence and the fifth confidence is determined as the confidence of the fourth sequential action nomination.

[0101] In combination with any embodiment of the present application, the fourth feature extraction process is implemented by g-level convolution processing, and the fifth feature extraction process is implemented by h-level convolution processing; both g and h are integers greater than 1;

[0102] Before determining the average of the fourth confidence and the fifth confidence as the confidence of the fourth sequential action nomination, the training process further includes:

[0103] The fourth feature map and the fifth feature map are merged to obtain a sixth feature map; the fourth feature map is output data of the z-th level convolution process in the g-level convolution process, and the fifth feature map is output data of the t-th level convolution process in the h-level convolution process; the size of the fourth feature map is the same as the size of the fifth feature map; z is a positive integer less than g, and t is a positive integer less than h;

[0104] Performing a sixth feature extraction process on the sixth feature graph to obtain a sixth confidence level of the fourth sequential action nomination;

[0105] The step of determining an average of the fourth confidence level and the fifth confidence level as the confidence level of the first time sequence action nomination includes:

[0106] An average of the fourth confidence, the fifth confidence, and the sixth confidence is determined as the confidence of the first sequential action nomination.

[0107] In combination with any implementation manner of the present application, the obtaining of the first training video stream includes:

[0108] Obtain a first positive sample video stream set and a first negative sample video stream set; the video streams in the first positive sample video stream set all contain the temporal action nomination of the first action; the video streams in the first negative sample video stream set all contain the temporal action nomination of the first action;

[0109] The first positive sample video stream set includes a first scale video stream set and a second scale video stream set; the scale of the video stream in the first scale video stream set is in a first scale interval, the scale of the video stream in the second scale video stream set is in a second scale interval, and the first scale interval has no intersection with the second scale interval; wherein the scale of the positive sample video stream is a ratio of a reference resolution to a resolution of the positive sample video stream; the positive sample video stream is a video stream in the first positive sample video stream set; the reference resolution is a resolution nominated by a sequential action of the first action;

[0110] When the number of video streams included in the first scaled video stream set is less than the number of video streams included in the second video stream set, sampling the first scaled video stream set at a first sampling rate to obtain a first training video stream subset, and sampling the first scaled video stream set at a second sampling rate to obtain a second training video stream subset; the first sampling rate is greater than the second sampling rate;

[0111] A video stream is selected from the first training video stream subset and / or the second training video stream subset and / or the first negative sample video stream set as the first training video stream.

[0112] In combination with any implementation manner of the present application, the obtaining of the first positive sample video stream set and the first negative sample video stream set includes:

[0113] Acquire a training video stream set; the training video stream set includes a second positive sample video stream set and a second negative sample video stream set; the video streams in the second positive sample video stream set all include the temporal action nomination of the first action; the video streams in the second negative sample all include the temporal action nomination of the first action;

[0114] In a case where the number of video streams included in the second positive sample video stream set is greater than the number of video streams included in the second negative sample video stream set, sampling from the second positive sample video stream set at a third sampling rate to obtain the second positive sample video stream set, and sampling from the second negative sample video stream set at a fourth sampling rate to obtain the second negative sample video stream set; the third sampling rate is less than the fourth sampling rate;

[0115] When the number of video streams included in the second positive sample video stream set is less than the number of video streams included in the second negative sample video stream set, the second positive sample video stream set is sampled from the second positive sample video stream set at a fifth sampling rate, and the second negative sample video stream set is sampled from the second negative sample video stream set at a sixth sampling rate; the fifth sampling rate is greater than the sixth sampling rate.

[0116] In a second aspect, a video processing device is provided, the device comprising:

[0117] An acquisition unit, used for acquiring a video stream to be processed;

[0118] A feature extraction processing unit, used for performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence;

[0119] an upsampling processing unit, configured to perform m-level upsampling processing on the first intermediate feature sequence to obtain a first temporal boundary probability sequence of the first action; the first temporal boundary probability sequence includes a probability that the image in the video stream to be processed is a start frame of the first action, and a probability that the image in the video stream to be processed is an end frame of the first action;

[0120] The input data of the i+1th level upsampling process in the m-level upsampling process is the fusion data of the output data of the i-th level upsampling process in the m-level upsampling process and the input data of the j-th level feature extraction process in the n-level feature extraction process; the resolution of the output data of the i-th level upsampling process and the input data of the j-th level feature extraction process are the same; i is a positive integer less than m, and j is a positive integer not greater than n.

[0121] In combination with any embodiment of the present application, the feature extraction processing unit is further used to perform feature extraction processing on the video stream to be processed to obtain a first feature sequence before performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence; semantic information of the video stream to be processed and optical flow feature information of the background in the video stream to be processed corresponding to the first feature sequence; the background includes content in the video stream to be processed except the first action;

[0122] The feature extraction processing unit is further used to perform the n-level feature extraction processing on the first feature sequence to obtain the first intermediate feature sequence.

[0123] In combination with any embodiment of the present application, the video processing device further includes: a flip unit and a first processing unit;

[0124] The flip unit is used to perform time-sequence flipping processing on the first feature sequence to obtain a second feature sequence whose playing order is opposite to that of the first feature sequence;

[0125] The feature extraction processing unit is used to perform p-level feature extraction processing on the second feature sequence to obtain a second intermediate feature sequence;

[0126] The upsampling processing unit is used to perform q-level upsampling processing on the second intermediate feature sequence to obtain a second temporal boundary probability sequence of the first action; the second temporal boundary probability sequence includes a probability that the image in the video stream to be processed is a start frame of the first action, and a probability that the image in the video stream to be processed is an end frame of the first action;

[0127] The input data of the r+1th level upsampling process in the qth level upsampling process is the fusion data of the output data of the rth level upsampling process in the qth level upsampling process and the input data of the sth level feature extraction process in the pth level feature extraction process; the output data of the rth level upsampling process and the input data of the sth level feature extraction process have the same resolution; r is a positive integer less than q, and s is a positive integer not greater than p;

[0128] The first processing unit is configured to obtain the at least one first time sequence action nomination according to the first time sequence boundary probability sequence and the second time sequence boundary probability sequence.

[0129] In combination with any embodiment of the present application, the input data of the j-th level feature extraction processing includes shallow feature information of the video stream to be processed; the s-th level feature extraction processing includes shallow feature information of the first feature sequence.

[0130] In combination with any implementation manner of the present application, the first processing unit is configured to:

[0131] fusing the first time series boundary probability sequence and the second time series boundary probability sequence to obtain a third time series boundary probability sequence of the first action;

[0132] According to the third temporal boundary probability sequence, the at least one first temporal action nomination is obtained.

[0133] In combination with any implementation manner of the present application, the at least one first time-sequence action nomination includes a second time-sequence action nomination, and the video processing device further includes a second processing unit;

[0134] The second processing unit is used for:

[0135] Extracting a feature sequence nominated by the second time-series action from the first feature sequence as a third feature sequence; the third feature sequence includes: a first channel feature sequence and a second channel feature sequence;

[0136] Determine the similarity between the first channel feature sequence and the second channel feature sequence to obtain a first similarity;

[0137] The confidence of the second time-series action nomination is obtained according to the first similarity and the third feature sequence.

[0138] In combination with any implementation manner of the present application, the second processing unit is used to:

[0139] Determine the matrix product of the first channel feature sequence and the second channel feature sequence to obtain a fourth feature sequence as the first similarity.

[0140] In combination with any implementation manner of the present application, the at least one first time-sequence action nomination further includes a third time-sequence action nomination different from the second time-sequence action nomination;

[0141] The second processing unit is configured to determine the similarity between the second time-sequence action nomination and the third time-sequence action nomination to obtain a second similarity before obtaining the confidence of the second time-sequence action nomination based on the first similarity and the third feature sequence;

[0142] The first processing unit is used to obtain the confidence of the second time-series action nomination according to the first similarity, the second similarity and the third feature sequence.

[0143] In combination with any implementation manner of the present application, the second processing unit is used to:

[0144] Extracting the third feature sequence and the feature sequence nominated by the third sequential action from the first feature sequence to obtain a fifth feature sequence;

[0145] The matrix product of the fifth characteristic sequence and the sixth characteristic sequence is determined to obtain a seventh characteristic sequence as the second similarity; the sixth characteristic sequence is the transpose of the fifth characteristic sequence.

[0146] In combination with any implementation manner of the present application, the first processing unit is configured to:

[0147] Performing a first feature extraction process on the first fused feature sequence to obtain a first confidence level of the second time-series action nomination; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence;

[0148] Performing a second feature extraction process on the second fused feature sequence to obtain a second confidence level of the second time-series action nomination; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence;

[0149] An average of the first confidence and the second confidence is determined as the confidence of the second time-series action nomination.

[0150] In combination with any embodiment of the present application, the first feature extraction process is implemented by u-level convolution processing, and the second feature extraction process is implemented by v-level convolution processing; both u and v are integers greater than 1;

[0151] The video processing device further includes a third processing unit; the third processing unit is used to fuse the first feature map and the second feature map to obtain a third feature map before determining the average of the first confidence level and the second confidence level as the confidence level of the first time-series action nomination; the first feature map is the output data of the c-th level convolution processing in the u-level convolution processing, and the second feature map is the output data of the d-th level convolution processing in the v-level convolution processing; the size of the first feature map is the same as the size of the second feature map; c is a positive integer less than u, and d is a positive integer less than v;

[0152] The feature extraction processing unit is used to perform a third feature extraction process on the third feature graph to obtain a third confidence level of the second time-series action nomination;

[0153] The third processing unit is used to determine an average of the first confidence level, the second confidence level and the third confidence level as the confidence level of the first sequential action nomination.

[0154] In combination with any implementation manner of the present application, the first processing unit is configured to:

[0155] Performing feature extraction processing on the first fused feature sequence to obtain a first reference feature sequence; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence;

[0156] Performing feature extraction processing on the second fused feature sequence to obtain a second reference feature sequence; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence;

[0157] fusing the first reference feature sequence and the second reference feature sequence to obtain a third reference feature sequence;

[0158] Perform feature extraction processing on the third reference feature sequence to obtain the confidence level of the second time-series action nomination.

[0159] In combination with any embodiment of the present application, the device further includes:

[0160] A sorting unit, configured to sort the at least one first time-sequence action nomination in descending order of quality score to obtain a time-sequence action nomination order of the first action; the quality score is obtained according to a time-sequence boundary probability sequence of the first time-sequence action nomination and a confidence level of the first time-sequence action nomination;

[0161] The display unit is used to display the at least one first time-sequence action nomination according to the time-sequence action nomination order of the first action.

[0162] In combination with any embodiment of the present application, the video processing method performed by the video processing device is applied to a temporal action nomination network; the video processing device further includes a training unit; the training unit is used to perform a training process of the temporal action nomination network;

[0163] The training process of the temporal action nomination network includes:

[0164] Acquire a first training video stream and a neural network to be trained; the neural network to be trained includes a first feature extraction module and a first upsampling module;

[0165] Using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence;

[0166] Using the first upsampling module to perform b-level upsampling processing on the third intermediate feature sequence to obtain a fourth temporal boundary probability sequence of the first action; the fourth temporal boundary probability sequence includes a probability that the image in the first training video stream is a start frame of the first action, and a probability that the image in the first training video stream is an end frame of the first action;

[0167] The input data of the k+1th level upsampling process in the b-level feature extraction process is the fusion data of the output data of the kth level upsampling process in the b-level upsampling process and the input data of the f-th level feature extraction process in the a-level feature extraction process; the output data of the kth level upsampling process and the input data of the f-th level feature extraction process have the same resolution; k is a positive integer less than b, and f is a positive integer not greater than a;

[0168] Obtaining the loss of the neural network to be trained according to a difference between the fourth temporal boundary probability sequence and a label of the first training video stream; the label of the first training video stream includes a temporal action nomination of the first action in the first training video stream;

[0169] The parameters of the neural network to be trained are updated based on the loss of the neural network to be trained to obtain the temporal action nomination network.

[0170] In combination with any embodiment of the present application, the neural network to be trained further includes a basic feature extraction module;

[0171] Before using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence, the training process further includes:

[0172] Using the basic feature extraction module to perform feature extraction processing on the first training video stream, obtaining an eighth feature sequence; the eighth feature sequence corresponds to the semantic information of the first training video stream and the optical flow feature information of the background in the first training video stream; the background includes the content in the first training video stream except the first action;

[0173] The step of performing level A feature extraction processing on the first training video stream using the first feature extraction module to obtain a third intermediate feature sequence includes:

[0174] The first feature extraction module is used to perform a-level feature extraction processing on the seven feature sequences to obtain the third intermediate feature sequence.

[0175] In combination with any embodiment of the present application, the neural network to be trained further includes a second feature extraction module and a second upsampling module;

[0176] Before obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream, the training process further includes:

[0177] Performing time sequence reversal processing on the eighth characteristic sequence to obtain a ninth characteristic sequence whose playing order is opposite to that of the eighth characteristic sequence;

[0178] Using the second feature extraction module to perform x-level feature extraction processing on the ninth feature sequence to obtain a fourth intermediate feature sequence;

[0179] Using the second upsampling module to perform y-level upsampling processing on the second intermediate feature sequence, to obtain a fifth temporal boundary probability sequence of the first action; the fifth temporal boundary probability sequence includes a probability that an image in the first training video stream is a start frame of the first action, and a probability that an image in the first training video stream is an end frame of the first action; the playback order of the first training video stream is opposite to the playback order of the first training video stream;

[0180] The input data of the w+1th level upsampling process in the x-level upsampling process is the fusion data of the output data of the wth level upsampling process in the x-level upsampling process and the input data of the eth level feature extraction process in the y-level feature extraction process; the output data of the w-level upsampling process and the input data of the e-level feature extraction process have the same resolution; e is a positive integer less than y, and w is a positive integer not greater than x;

[0181] The neural network to be trained fuses the fourth time series boundary probability sequence and the fifth time series boundary probability sequence to obtain a sixth time series boundary probability sequence of the first action;

[0182] The step of obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream includes:

[0183] The loss of the neural network to be trained is obtained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream.

[0184] In combination with any implementation manner of the present application, the neural network to be trained further includes a nomination relationship module;

[0185] Before obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream, the training process further includes:

[0186] The neural network to be trained extracts a feature sequence of a fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence; the fourth temporal action nomination is a temporal action nomination of the first action obtained according to the sixth temporal boundary probability sequence; the tenth feature sequence includes: a third channel feature sequence and a fourth channel feature sequence;

[0187] Determine the similarity between the third channel feature sequence and the fourth channel feature sequence using the nominated relationship module to obtain a third similarity;

[0188] The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence;

[0189] The step of obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream includes:

[0190] Obtaining a first loss according to a difference between the sixth temporal boundary probability sequence and the label of the first training video stream;

[0191] Obtaining a second loss according to a difference between a confidence of the fourth temporal action nomination and a label of the first training video stream;

[0192] The sum of the first loss and the second loss is used as the loss of the neural network to be trained.

[0193] In combination with any embodiment of the present application, the using the nominated relationship module to determine the similarity between the third channel feature sequence and the fourth channel feature sequence to obtain a third similarity includes:

[0194] The nominated relationship module is used to determine the matrix product of the third channel feature sequence and the fourth channel feature sequence to obtain an eleventh feature sequence as the third similarity.

[0195] In combination with any implementation manner of the present application, before the nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence, the training process further includes:

[0196] The nomination relationship module determines the similarity between the fourth time-series action nomination and the fifth time-series action nomination to obtain a fourth similarity; the fifth time-series action nomination is a time-series action nomination of the first action obtained according to the sixth time-series boundary probability sequence, and the fifth time-series action nomination is different from the fourth time-series action nomination;

[0197] The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence, including:

[0198] The confidence of the fourth time-series action nomination is obtained according to the third similarity, the fourth similarity and the tenth feature sequence.

[0199] In combination with any embodiment of the present application, the neural network to be trained extracts the feature sequence of the fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence, including:

[0200] The neural network to be trained extracts the tenth feature sequence and the feature sequence nominated by the fifth time-series action from the eighth feature sequence to obtain a twelfth feature sequence;

[0201] The nomination relationship module determines the similarity between the fourth time-sequence action nomination and the fifth time-sequence action nomination to obtain a fourth similarity, including:

[0202] The nomination relationship module determines the matrix product of the twelfth feature sequence and the thirteenth feature sequence to obtain a fourteenth feature sequence as the fourth similarity; the thirteenth feature sequence is the transpose of the twelfth feature sequence.

[0203] In combination with any implementation manner of the present application, obtaining the confidence of the fourth time-series action nomination based on the third similarity, the fourth similarity and the tenth feature sequence includes:

[0204] Performing a fourth feature extraction process on the third fused feature sequence to obtain a fourth confidence level of the fourth time-series action nomination; the third fused feature sequence, the tenth feature sequence and the eleventh feature sequence are obtained;

[0205] Performing a fifth feature extraction process on the fourth fused feature sequence to obtain a fifth confidence level of the fourth time-series action nomination; the tenth feature sequence and the fourteenth feature sequence of the fourth fused feature sequence are obtained;

[0206] An average of the fourth confidence and the fifth confidence is determined as the confidence of the fourth sequential action nomination.

[0207] In combination with any embodiment of the present application, the fourth feature extraction process is implemented by g-level convolution processing, and the fifth feature extraction process is implemented by h-level convolution processing; both g and h are integers greater than 1;

[0208] Before determining the average of the fourth confidence and the fifth confidence as the confidence of the fourth sequential action nomination, the training process further includes:

[0209] The fourth feature map and the fifth feature map are merged to obtain a sixth feature map; the fourth feature map is output data of the z-th level convolution process in the g-level convolution process, and the fifth feature map is output data of the t-th level convolution process in the h-level convolution process; the size of the fourth feature map is the same as the size of the fifth feature map; z is a positive integer less than g, and t is a positive integer less than h;

[0210] Performing a sixth feature extraction process on the sixth feature graph to obtain a sixth confidence level of the fourth sequential action nomination;

[0211] The step of determining an average of the fourth confidence level and the fifth confidence level as the confidence level of the first time sequence action nomination includes:

[0212] An average of the fourth confidence, the fifth confidence, and the sixth confidence is determined as the confidence of the first sequential action nomination.

[0213] In combination with any implementation manner of the present application, the obtaining of the first training video stream includes:

[0214] Obtain a first positive sample video stream set and a first negative sample video stream set; the video streams in the first positive sample video stream set all contain the temporal action nomination of the first action; the video streams in the first negative sample video stream set all contain the temporal action nomination of the first action;

[0215] The first positive sample video stream set includes a first scale video stream set and a second scale video stream set; the scale of the video stream in the first scale video stream set is in a first scale interval, the scale of the video stream in the second scale video stream set is in a second scale interval, and the first scale interval has no intersection with the second scale interval; wherein the scale of the positive sample video stream is a ratio of a reference resolution to a resolution of the positive sample video stream; the positive sample video stream is a video stream in the first positive sample video stream set; the reference resolution is a resolution nominated by a sequential action of the first action;

[0216] When the number of video streams included in the first scaled video stream set is less than the number of video streams included in the second video stream set, sampling the first scaled video stream set at a first sampling rate to obtain a first training video stream subset, and sampling the first scaled video stream set at a second sampling rate to obtain a second training video stream subset; the first sampling rate is greater than the second sampling rate;

[0217] A video stream is selected from the first training video stream subset and / or the second training video stream subset and / or the first negative sample video stream set as the first training video stream.

[0218] In combination with any implementation manner of the present application, the obtaining of the first positive sample video stream set and the first negative sample video stream set includes:

[0219] Acquire a training video stream set; the training video stream set includes a second positive sample video stream set and a second negative sample video stream set; the video streams in the second positive sample video stream set all include the temporal action nomination of the first action; the video streams in the second negative sample all include the temporal action nomination of the first action;

[0220] In a case where the number of video streams included in the second positive sample video stream set is greater than the number of video streams included in the second negative sample video stream set, sampling from the second positive sample video stream set at a third sampling rate to obtain the second positive sample video stream set, and sampling from the second negative sample video stream set at a fourth sampling rate to obtain the second negative sample video stream set; the third sampling rate is less than the fourth sampling rate;

[0221] When the number of video streams included in the second positive sample video stream set is less than the number of video streams included in the second negative sample video stream set, the second positive sample video stream set is sampled from the second positive sample video stream set at a fifth sampling rate, and the second negative sample video stream set is sampled from the second negative sample video stream set at a sixth sampling rate; the fifth sampling rate is greater than the sixth sampling rate.

[0222] In a third aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation thereof.

[0223] In a fourth aspect, an electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0224] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any possible implementation method thereof.

[0225] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.

[0226] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0227] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0228] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0229] Figure 1 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0230] Figure 2 A schematic diagram of the structure of a sequential action nomination network provided in an embodiment of the present application;

[0231] Figure 3 A flowchart of a method for training a sequential action nomination network provided in an embodiment of the present application;

[0232] Figure 4 A schematic diagram of the structure of a feature extraction module and an upsampling module provided in an embodiment of the present application;

[0233] Figure 5 A schematic diagram of the structure of a nomination relationship module provided in an embodiment of the present application;

[0234] Figure 6 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0235] Figure 7 A schematic diagram of the hardware structure of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0236] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0237] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0238] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0239] It should be understood that in the present application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three and more than three, and "and / or" is used to describe the association relationship of associated objects, indicating that there may be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " can indicate that the objects before and after the association are in an "or" relationship, which means any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. The character " / " can also represent the division sign in mathematical operations, for example, a / b = a divided by b; 6 / 3 = 2. "At least one of the following" or similar expressions.

[0240] With the explosive growth of the number of various videos, how to select temporal action nominations containing target actions from massive videos is of great significance.

[0241] By applying the temporal nomination detection technology to process the video, a temporal boundary probability sequence of the video can be obtained, wherein the temporal boundary probability sequence includes the probability that each frame image in the video is the starting frame nominated for a temporal action, and the probability that each frame image is the ending frame nominated for a temporal action. Then, based on the temporal boundary probability sequence, the temporal action nomination containing the target action can be found from a large amount of video. However, the accuracy of the temporal boundary probability sequence obtained by the current temporal nomination detection technology is low. The embodiment of the present application provides a video processing method to improve the accuracy of the temporal boundary probability sequence.

[0242] In the embodiment of the present application, the temporal action nomination refers to a video containing the target action. For example, if the target action is kicking a ball, the temporal action nomination is a video containing the action of kicking a ball.

[0243] In the embodiment of the present application, the start frame refers to the first frame of image played in the time sequence action nomination; the end frame refers to the last frame of image played in the time sequence action nomination. Among them, all images in the target video stream include the target image.

[0244] The execution subject of the method embodiment of the present application is a video processing device. Optionally, the video processing device can be one of the following: a mobile phone, a computer, a server, and a tablet computer. The present application embodiment is described below in conjunction with the drawings in the present application embodiment.

[0245] See also Figure 1 , Figure 1 It is a flowchart of a video processing method provided in an embodiment of the present application.

[0246] 101. Obtain the video stream to be processed.

[0247] In the embodiment of the present application, the video stream to be processed includes at least two frames of images. The video stream to be processed can be a video stream collected in real time by the video processing device through the camera component; the first video stream can also be a video stream collected non-real time by the video processing device.

[0248] In a possible implementation manner, the video processing device obtains the video stream to be processed by downloading from the Internet.

[0249] In another possible implementation, the video processing device uses the video stream input by the user through an input component as the video stream to be processed. The above input component includes: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0250] In another possible implementation, the video processing device acquires the video stream to be processed by collecting it through a camera assembly. The camera assembly includes a camera.

[0251] In another possible implementation, a communication connection is provided between the video processing device and the first data terminal, and the video processing device receives a video stream sent by the first data terminal through the communication connection as the video stream to be processed.

[0252] 102. Perform n-level feature extraction processing on the above-mentioned video stream to be processed to obtain a first intermediate feature sequence.

[0253] In an embodiment of the present application, the n-level feature extraction process is implemented by n layers of feature extraction layers, where n is a positive integer. Each layer of feature extraction layer performs a first-level feature extraction process, and the n layers of feature extraction layers are connected in series in sequence, that is, the output data of the first layer of feature extraction layer is the input data of the second layer of feature extraction layer, the output data of the second layer of feature extraction layer is the input data of the third layer of feature extraction layer, ..., the output data of the n-1th layer of feature extraction layer is the input data of the nth layer of feature extraction layer, and the output data of the nth layer of feature extraction layer is an intermediate feature sequence. Wherein, n is a positive integer. Exemplarily, n is 3. Each layer of feature extraction layer includes at least one of the following processes: convolution processing, pooling processing.

[0254] In one possible implementation, the feature extraction layer includes at least one convolution layer, each convolution layer is used to perform temporal convolution processing, and when the number of convolution layers exceeds 1, all convolution layers are connected in series, and the semantic information extracted by each convolution layer is different.

[0255] Take the first feature extraction layer as an example. The video processing device performs layer-by-layer convolution processing on the video stream to be processed through at least one convolution layer in the first feature extraction layer, gradually extracts the semantic information in the video stream to be processed, and discards some shallow feature information (such as geometric relationship information between pixels, position information of pixels, and edge information). Therefore, the resolution of the feature data extracted later is smaller, and the semantic information in the feature data is more concentrated. By performing layer-by-layer convolution processing on the video stream to be processed through at least one convolution layer, the resolution of the video stream to be processed can be reduced while obtaining the semantic information in the video stream to be processed, and the output data of the first feature extraction layer can be obtained. In this way, in the subsequent processing process, the video processing device can reduce the data processing amount of the video processing device and improve the processing speed of the video processing device by processing the feature data output by the feature extraction layer.

[0256] It should be understood that in the embodiments of the present application, the convolution processing performed on the video stream is all time series convolution processing. And the time series convolution processing can be implemented by a time series convolution kernel. Specifically, the size of the time series convolution kernel is h*w*t, that is, the height of the time series convolution kernel is h, the width is w, and the span of the time series dimension is t.

[0257] In the embodiment of the present application, the resolution of a video stream refers to the duration of the video stream. For example, the duration of video stream a is 30 seconds, and the duration of video stream b is 20 seconds. At this time, the resolution of video stream a is lower than the resolution of video stream b.

[0258] The video processing device extracts semantic information of the video stream to be processed by performing n-level feature extraction processing on the video stream to be processed, and obtains a first intermediate feature sequence.

[0259] 103. Perform m-level upsampling processing on the first intermediate feature sequence to obtain a first time-series boundary probability sequence of the first action.

[0260] In the embodiment of the present application, the first action can be any action, for example, the first action is singing; the first action is dancing; the first action is fighting; the first action is falling; the first action is kicking a ball.

[0261] In the embodiment of the present application, the first temporal boundary probability sequence includes the probability that the image in the video stream to be processed is the start frame of the first action, and the probability that the image in the video stream to be processed is the end frame of the first action.

[0262] For example, the video stream to be processed includes: a first frame image, a second frame image and a third frame image, wherein the order of playing the video stream to be processed is: the first frame image, the second frame image, and the third frame image. The first temporal boundary probability sequence includes: the probability that the first frame image is the start frame of the first action, the probability that the second frame image is the start frame of the first action, the probability that the third frame image is the start frame of the first action, the probability that the first frame image is the end frame of the first action, the probability that the second frame image is the end frame of the first action, and the probability that the third frame image is the end frame of the first action.

[0263] In the embodiment of the present application, the video processing device can improve the resolution of the first intermediate feature sequence by performing upsampling processing on the first intermediate feature sequence. The video stream processing device can improve the resolution of the first intermediate feature sequence to the same as the resolution of the video stream to be processed by performing m-level upsampling processing on the first intermediate feature sequence, and then determine the probability that each frame image in the video stream to be processed is the boundary of the first action, where m is a positive integer.

[0264] The m-level upsampling process is implemented by m upsampling layers. Each upsampling layer performs one-level upsampling process, and the m upsampling layers are connected in series, that is, the output data of the 1st upsampling layer is the input data of the 2nd upsampling layer, the output data of the 2nd upsampling layer is the input data of the 3rd upsampling layer, ..., the output data of the m-1th upsampling layer is the input data of the mth upsampling layer, and the output data of the mth upsampling layer is the intermediate feature sequence. Wherein, m is a positive integer. Exemplarily, m is 3. Each upsampling layer includes at least one of the following processes: bilinear interpolation process, neighboring point interpolation process, and deconvolution process.

[0265] In one possible implementation, the upsampling layer includes at least one deconvolution layer, each deconvolution layer is used to amplify the resolution of the input data, and when the number of deconvolution layers exceeds 1, all deconvolution layers are connected in series.

[0266] Take the first upsampling layer as an example. The video processing device performs layer-by-layer convolution processing on the first intermediate feature sequence through at least one deconvolution layer in the first upsampling layer, and gradually amplifies the resolution of the first intermediate feature sequence. Therefore, the resolution of the feature data obtained later is larger. By performing layer-by-layer deconvolution processing on the first intermediate feature sequence through at least one deconvolution layer, a first temporal boundary probability sequence having the same resolution as the resolution of the video stream to be processed can be obtained.

[0267] Since shallow feature information will be discarded during the feature extraction process of the video processing device on the processed video stream, the video processing device will fuse the output data of the feature extraction layer with the data of the same resolution output by the upsampling layer during the upsampling process on the first intermediate feature sequence to enrich the information in the first time series boundary probability sequence, thereby improving the accuracy of the first time series boundary probability sequence.

[0268] For example, when the size of the first action in the image is small, the shallow feature information can greatly improve the accuracy of identifying whether the first action exists in the image. The video processing device can improve the accuracy of the temporal boundary probability sequence of the first action by fusing the output data of the feature extraction layer with the data of the same resolution output by the upsampling layer.

[0269] In a possible implementation, the input data of the i+1th level upsampling process in the m-level upsampling process is the fusion data between the output data of the i-th level upsampling process in the m-level upsampling process and the input data of the j-th level feature extraction process in the n-level feature extraction process. The output data of the i-th level upsampling process and the input data of the j-th level feature extraction process have the same resolution, i is a positive integer less than m, and j is a positive integer not greater than n.

[0270] For example, n=m=2, i=1, j=2. At this time, the video processing device sets the resolution of the input data of the second-level feature extraction processing (i.e., the output data of the first-level feature extraction processing) to be the same as the resolution of the output data of the first-level upsampling processing. The video processing device fuses these two data to obtain the input data of the second-level upsampling processing.

[0271] In an embodiment of the present application, the video processing device, during the process of upsampling the first intermediate feature sequence, fuses the feature sequence obtained by upsampling with the feature sequence obtained by feature extraction to enrich the shallow feature information in the feature sequence, thereby improving the accuracy of the boundary probability sequence.

[0272] As an optional implementation manner, before executing step 102, the video processing device further executes the following steps:

[0273] 1. Perform feature extraction processing on the above-mentioned video stream to be processed to obtain a first feature sequence.

[0274] In an embodiment of the present application, the first feature sequence corresponds to semantic information of the video stream to be processed and optical flow information of the background in the video stream to be processed, wherein the background includes content in the video stream to be processed except the first action.

[0275] For example, the video stream to be processed includes image a and image b. The first feature sequence corresponds to the following information: semantic information of image a and image b, optical flow feature information of the background in image a, and optical flow feature information of the background in image b.

[0276] The video processing device can obtain a first feature sequence by performing feature extraction processing on the video stream to be processed, wherein the feature extraction processing can be implemented by a combination of at least one of the following processing: convolution processing and pooling processing.

[0277] It should be understood that the first feature sequence includes semantic information of each frame of the video stream to be processed and optical flow information of the background in each frame of the video stream to be processed, and the play order of the first feature sequence is the same as the play order of the video stream to be processed.

[0278] For example, the playback order of the video stream to be processed is image a, image b, and image c. The first feature sequence includes feature image A, feature image B, and feature image C, wherein feature image A corresponds to the semantic information of image a and the optical flow feature information of the background in image a, feature image B corresponds to the semantic information of image b and the optical flow feature information of the background in image b, and feature image C corresponds to the semantic information of image c and the optical flow feature information of the background in image c. At this time, the playback order of the first feature sequence is feature image A, feature image B, and feature image C.

[0279] After obtaining the first feature sequence, the video processing device performs the following steps in the process of executing step 102:

[0280] 2. Perform the n-level feature extraction process on the first feature sequence to obtain the first intermediate feature sequence.

[0281] The video processing device obtains a first intermediate feature sequence by performing n-level feature extraction processing on the first feature sequence, which not only corresponds to the semantic information of the video stream to be processed, but also corresponds to the optical flow feature information of the background in the video stream to be processed. In this way, the first intermediate feature sequence is subsequently subjected to m-level upsampling processing, which not only utilizes the semantic information of the video stream to be processed, but also utilizes the optical flow feature information of the background in the video stream to be processed, to obtain a first temporal boundary probability sequence.

[0282] Since the video processing device can determine the change of the motion state of the background through the optical flow feature information, and then determine the change of the motion state of the first action based on the change of the motion state of the background, the start probability and end probability of the image in the video stream to be processed can be determined.

[0283] Therefore, the video processing device can improve the accuracy of the first time series boundary probability sequence by processing the first feature sequence to obtain the first time series boundary probability sequence.

[0284] As an optional implementation manner, the video processing device further performs the following steps:

[0285] 3. Perform time-sequence reversal processing on the first feature sequence to obtain a second feature sequence whose playback order is opposite to that of the first feature sequence.

[0286] 4. Perform p-level feature extraction processing on the second feature sequence to obtain a second intermediate feature sequence.

[0287] The feature extraction process in this step can refer to the feature extraction process in step 102, and will not be described in detail here. Optional p=n.

[0288] 5. Perform q-level upsampling processing on the above-mentioned second intermediate feature sequence to obtain the second time series boundary probability sequence of the above-mentioned first action.

[0289] In the embodiment of the present application, the second temporal boundary probability sequence includes the probability that the image in the to-be-processed video stream is the start frame of the first action, and the probability that the image in the second to-be-processed video stream is the end frame of the first action.

[0290] For example, the video stream to be processed includes: a first frame image, a second frame image and a third frame image, wherein the order of playing the video stream to be processed is: the first frame image, the second frame image, and the third frame image. The second temporal boundary probability sequence includes: the probability that the first frame image is the starting frame of the first action, the probability that the second frame image is the starting frame of the first action, the probability that the third frame image is the starting frame of the first action, the probability that the first frame image is the ending frame of the first action, the probability that the second frame image is the ending frame of the first action, and the probability that the third frame image is the ending frame of the first action.

[0291] The upsampling process in this step can refer to the upsampling process in step 103 and will not be described here. Optional q = m. As in step 102, in order to enrich and enhance the information in the second temporal boundary probability sequence, the video processing device fuses the output data of the feature extraction layer with the data of the same resolution output by the upsampling layer during the upsampling process of the second intermediate feature sequence.

[0292] In a possible implementation, the input data of the r+1th level upsampling process in the qth level upsampling process is the fusion data between the output data of the rth level upsampling process in the qth level upsampling process and the input data of the sth level feature extraction process in the pth level feature extraction process. The output data of the rth level upsampling process and the input data of the sth level feature extraction process have the same resolution, r is a positive integer less than q, and s is a positive integer not greater than p.

[0293] For example, p=q=2, r=1, s=2. At this time, the video processing device sets the resolution of the input data of the second-level feature extraction processing (i.e., the output data of the first-level feature extraction processing) to be the same as the resolution of the output data of the first-level upsampling processing. The video processing device fuses these two data to obtain the input data of the second-level upsampling processing.

[0294] 6. Based on the first time sequence boundary probability sequence and the second time sequence boundary probability sequence, obtain the at least one first time sequence action nomination.

[0295] After obtaining the first temporal boundary probability sequence and the second temporal boundary probability sequence, the video processing apparatus may determine at least one temporal action nomination of the first action in the video stream to be processed.

[0296] For the convenience of description, the probability that the image is the start frame of the first action is called the start probability, and the probability that the image is the end frame of the first action is called the end probability.

[0297] In a possible implementation, the video processing device uses an image whose start frame probability exceeds a first start probability threshold as a start frame of a temporal action nomination for a first action, and uses an image whose end frame probability exceeds a first end probability threshold as an end frame of a temporal action nomination for the first action. The video processing device uses an image between the start frame and the end frame as a temporal action nomination for the first action.

[0298] For example, the video stream to be processed includes the first frame image, the second frame image, and the third frame image, wherein the order of playing the video stream to be processed is: the first frame image, the second frame image, and the third frame image. The first temporal boundary probability sequence includes: the starting probability of the first frame image is 0.83, the ending probability of the first frame image is 0.2, the starting probability of the second frame image is 0.5, the ending probability of the second frame image is 0.9, the starting probability of the third frame image is 0.3, and the ending probability of the third frame image is 0.6. Assume that the first starting probability threshold is 0.8 and the first ending probability threshold is 0.8. Since the starting probability of the first frame image is greater than the first starting probability threshold, the ending probability of the second frame image exceeds the first ending probability threshold, the video stream composed of the first frame image and the second frame image, and the playing order is the first frame image and the second frame image, is the temporal action nomination of the first action.

[0299] In this way, the video processing device can obtain at least one first temporal action nomination according to the first temporal boundary probability sequence, and can obtain at least one first temporal action nomination according to the second temporal boundary probability sequence.

[0300] In another possible implementation, the video processing device uses the image with the highest probability of the start frame as the start frame of the time sequence action nomination for the first action, and uses the image with the highest probability of the end frame as the end frame of the time sequence action nomination for the first action. The video processing device uses the image between the start frame and the end frame as the time sequence action nomination for the first action.

[0301] For example, the video stream to be processed includes the first frame image, the second frame image, and the third frame image, wherein the order of playing the video stream to be processed is: the first frame image, the second frame image, and the third frame image. The first temporal boundary probability sequence includes: the start probability of the first frame image is 0.3, the end probability of the first frame image is 0.2, the start probability of the second frame image is 0.7, the end probability of the second frame image is 0.4, the start probability of the third frame image is 0.4, and the end probability of the third frame image is 0.6. Since the image with the highest start probability is the second frame image and the image with the highest end probability is the third frame image, the video stream composed of the second frame image and the third frame image, and the play order is the second frame image and the third frame image, is a first temporal action nomination.

[0302] In this way, the video processing device can obtain at least one first temporal action nomination according to the first temporal boundary probability sequence, and can obtain at least one first temporal action nomination according to the second temporal boundary probability sequence.

[0303] In another possible implementation, the video processing device fuses the first temporal boundary probability sequence and the second temporal boundary probability sequence to obtain a third temporal boundary probability sequence of the first action, and then obtains at least one first temporal action nomination based on the third temporal boundary probability sequence.

[0304] Optionally, fusing the first temporal boundary probability sequence and the second temporal boundary probability sequence may be to calculate the mean of the first temporal boundary probability sequence and the second temporal boundary probability sequence. For example, the video stream to be processed includes a first frame image and a second frame image. The first temporal boundary probability sequence includes: the starting probability of the first frame image is 0.8, the ending probability of the first frame image is 0.5, the starting probability of the second frame image is 0.3, and the ending probability of the second frame image is 0.9. The second temporal boundary probability sequence includes: the starting probability of the first frame image is 0.7, the ending probability of the first frame image is 0.4, the starting probability of the second frame image is 0.4, and the ending probability of the second frame image is 0.9.

[0305] The video processing device calculates the average of the first temporal boundary probability sequence and the second temporal boundary probability sequence to obtain: the starting probability of the first frame image is (0.8+0.7) / 2=0.75, the ending probability of the first frame image is (0.5+0.4) / 2=0.45, the starting probability of the second frame image is (0.3+0.4) / 2=0.35, and the ending probability of the second frame image is (0.9+0.9) / 2=0.9.

[0306] Optionally, assume that the starting probability of the image in the first temporal boundary probability sequence is H s , the ending probability is H e , the starting probability of the image in the second temporal boundary probability sequence is h s , the ending probability is h e , the starting probability of the image in the fused third boundary probability sequence is L s , the ending probability is L e Then H s , H e 、h s 、h e , L s , L e Satisfy the following formula:

[0307]

[0308] For example, the video stream to be processed includes a first frame image and a second frame image. The first temporal boundary probability sequence includes: the start probability of the first frame image is 0.8, the end probability of the first frame image is 0.5, the start probability of the second frame image is 0.3, and the end probability of the second frame image is 0.9. The second temporal boundary probability sequence includes: the start probability of the first frame image is 0.7, the end probability of the first frame image is 0.4, the start probability of the second frame image is 0.4, and the end probability of the second frame image is 0.9.

[0309] The video processing device fuses the first temporal boundary probability sequence and the second temporal boundary probability sequence to obtain: the starting probability of the first frame image is The ending probability of the first frame image is The starting probability of the second frame image is The ending probability of the second frame image is

[0310] The video processing device can improve the accuracy of the boundary probability (including the start probability and the end probability) of the image by fusing the first temporal boundary probability sequence and the second temporal boundary probability sequence to obtain a third temporal boundary probability sequence, thereby improving the accuracy of at least one first temporal action nomination.

[0311] In another possible implementation, the video processing device uses the maximum value of the boundary probability of the image in the first temporal boundary probability sequence and the boundary probability of the image in the second temporal boundary probability sequence as the boundary probability of the image to obtain a first intermediate temporal boundary probability sequence. Further, at least one first temporal action nomination can be obtained based on the first intermediate temporal boundary probability sequence.

[0312] For example, the video stream to be processed includes a first frame image and a second frame image. The first temporal boundary probability sequence includes: the start probability of the first frame image is 0.8, the end probability of the first frame image is 0.5, the start probability of the second frame image is 0.3, and the end probability of the second frame image is 0.9. The second temporal boundary probability sequence includes: the start probability of the first frame image is 0.7, the end probability of the first frame image is 0.4, the start probability of the second frame image is 0.4, and the end probability of the second frame image is 0.9.

[0313] Since the starting probability of the first frame image in the first temporal boundary probability sequence is greater than the starting probability of the first frame image in the second temporal boundary probability sequence, in the first intermediate boundary probability sequence, the starting probability of the first frame image is 0.8. Since the ending probability of the first frame image in the first temporal boundary probability sequence is greater than the ending probability of the first frame image in the second temporal boundary probability sequence, in the first intermediate boundary probability sequence, the ending probability of the first frame image is 0.5. Since the starting probability of the second frame image in the first temporal boundary probability sequence is less than the starting probability of the second frame image in the second temporal boundary probability sequence, in the first intermediate boundary probability sequence, the starting probability of the second frame image is 0.4. Since the ending probability of the second frame image in the first temporal boundary probability sequence is equal to the ending probability of the second frame image in the second temporal boundary probability sequence, in the first intermediate boundary probability sequence, the ending probability of the second frame image is 0.9.

[0314] The video processing device obtains a third temporal boundary probability sequence based on the first temporal boundary probability sequence and the second temporal boundary probability sequence, which can improve the accuracy of the boundary probability (including the start probability and the end probability) of the image, and further improve the accuracy of at least one first temporal action nomination.

[0315] As an optional implementation, at least one first time-sequence action nomination includes a second time-sequence action nomination. The video processing device further performs the following steps based on the above steps:

[0316] 7. Extract the feature sequence of the second time-series action nomination from the first feature sequence as the third feature sequence.

[0317] In the embodiment of the present application, the third feature sequence is the feature sequence nominated by the second time-sequence action. After obtaining the second time-sequence action nomination, the video processing device can extract the third feature sequence from the first feature sequence.

[0318] For example, the playback order of the video stream to be processed is: image a, image b, image c, image d. The first feature sequence includes: feature image A, feature image B, feature image C, feature image D, wherein feature image A corresponds to the semantic information of image a and the optical flow feature information of the background in image a, feature image B corresponds to the semantic information of image b and the optical flow feature information of the background in image b, feature image C corresponds to the semantic information of image c and the optical flow feature information of the background in image c, and feature image D corresponds to the semantic information of image d and the optical flow feature information of the background in image d.

[0319] Assume that the second time-series action nomination includes: image a, image b, image c. The video processing device extracts feature image A, feature image B, and feature image C from the first feature sequence as a third feature sequence.

[0320] In this step, the third feature sequence includes at least two channel feature sequences, and the at least two channel feature sequences include: a first channel feature sequence and a second channel feature sequence, wherein information corresponding to the first channel feature sequence is different from information corresponding to the second channel feature sequence.

[0321] For example, the information corresponding to the first channel feature sequence includes color information, and the information corresponding to the second channel feature sequence includes edge information. For another example, the information corresponding to the first channel feature sequence includes action category information, and the information corresponding to the second channel feature sequence includes action position information.

[0322] 8. Determine the similarity between the first channel feature sequence and the second channel feature sequence to obtain a first similarity.

[0323] Since the first action has at least one dimension of information, the first action can be identified by the at least one dimension of information. For example, suppose the first action is kicking a ball. At this time, if there is a football around a person's feet, it is likely that the person is kicking the ball; if a person is wearing a football uniform, it is likely that the person is kicking the ball; if a person is in a football field, it is likely that the person is kicking the ball; if a person lifts his leg, it is likely that the person is kicking the ball.

[0324] As mentioned above, the information corresponding to the feature sequences of different channels is different, that is, the information corresponding to different channels belongs to information of different dimensions. Based on the information of any dimension, the probability of whether the image in the first video to be processed contains the first action can be obtained. Then, if an image contains the first action, the feature sequence of any channel corresponds to the information related to the first action.

[0325] Therefore, the video processing device fully utilizes the correlation between the information corresponding to different channel feature sequences by determining the similarity between the first channel feature sequence and the second channel feature sequence, so as to subsequently determine the confidence that the second temporal action nomination includes the first action.

[0326] In a possible implementation, the video processing device determines a matrix product of the first channel feature sequence and the second channel feature sequence to obtain a fourth feature sequence as the first similarity.

[0327] 9. Based on the first similarity and the third feature sequence, obtain the confidence of the second temporal action nomination.

[0328] In the embodiment of the present application, the confidence of the second time-sequence action nomination is positively correlated with the completeness of the second time-sequence action nomination. If the number of images containing the first action in the video stream to be processed is α, the number of images in the second time-sequence action nomination is β, and the number of images containing the first action in the second time-sequence action nomination is γ, the completeness of the second time-sequence action nomination is:

[0329] For example, assuming that the video stream to be processed includes image a, image b, image c, image d, and image e, the playback order of the video stream to be processed is: image a, image b, image c, image d, image e. Among them, image a, image b, image d, and image e all contain the first action. If the video processing device obtains a first sequential action nomination for the first action by processing the video stream to be processed, it is recorded as the first sequential action nomination A, wherein the first sequential action nomination A includes image b and image c. At this time, α=4, β=2, γ=1, that is, the completeness of the first sequential action nomination A is:

[0330] In one possible implementation, the video processing device fuses the first similarity (i.e., the fourth feature sequence) with the third feature sequence to enrich the information corresponding to the third feature sequence and obtain the confidence of the second temporal action nomination. For example, the video processing device can obtain the confidence of the second temporal action nomination by adding the fourth feature sequence to the third feature sequence.

[0331] In another possible implementation, the video processing device obtains the confidence of the second time-sequence action nomination based on the first similarity, which is recorded as the first intermediate confidence. The video processing device obtains the confidence of the second time-sequence action nomination based on the third feature sequence, which is recorded as the second intermediate confidence. The video processing device uses the maximum value of the first intermediate confidence and the second intermediate confidence as the confidence of the second time-sequence action.

[0332] It should be understood that the embodiment of the present application describes a process in which a video processing device determines the similarity between a first channel feature sequence and a second channel feature sequence to obtain a first similarity, and obtains the confidence of a second sequential action based on the first similarity and a third feature sequence. In actual applications, if the third feature sequence contains at least three channels, the video processing device may determine the similarity between any two channel feature sequences to obtain a first similarity set. The video processing device may then obtain the confidence of the second sequential action based on the similarity in the first similarity set and the third feature sequence.

[0333] Due to the correlation between different channels in the third feature sequence corresponding to the first similarity, the video processing device obtains the confidence of the second temporal action nomination based on the first similarity and the third feature sequence, which can improve the accuracy of the confidence of the second temporal action nomination.

[0334] As an optional implementation manner, the at least one first time-sequence action nomination further includes a third time-sequence action nomination, and the video processing device further performs the following steps before performing step 9:

[0335] 10. Determine the similarity between the second time-sequence action nomination and the third time-sequence action nomination to obtain a second similarity.

[0336] In the embodiment of the present application, the second temporal action nomination is different from the third temporal action nomination. In a possible implementation, the video processing device determines the matrix product of the second temporal action nomination and the third temporal action nomination to obtain a seventh feature sequence as the second similarity.

[0337] After obtaining the second similarity, the video processing device performs the following steps during the process of executing step 9:

[0338] 91. Obtain the confidence of the second temporal action nomination based on the first similarity, the second similarity and the third feature sequence.

[0339] In a possible implementation, the video processing device fuses the first similarity (i.e., the fourth feature sequence) with the third feature sequence to enrich the information corresponding to the third feature sequence and obtain a fifth intermediate confidence (for example, the sum of the fourth feature sequence and the seventh feature sequence is used as the fifth intermediate confidence). The video processing device fuses the second similarity (i.e., the seventh feature sequence) with the third feature sequence to enrich the information corresponding to the third feature sequence and obtain a sixth intermediate confidence (for example, the sum of the fourth feature sequence and the seventh feature sequence is used as the sixth intermediate confidence).

[0340] The video processing device obtains the confidence of the second temporal action nomination by determining the average of the fifth intermediate confidence and the sixth intermediate confidence.

[0341] As an optional implementation, in the case where at least one first time-sequence action nomination includes a second time-sequence action nomination and a third time-sequence action nomination, the video processing device performs the following steps during the execution of step 7:

[0342] 11. Extract the third feature sequence and the feature sequence nominated by the third sequential action from the first feature sequence to obtain a fifth feature sequence.

[0343] After obtaining the fifth feature sequence, the video processing device obtains the second similarity by performing the following steps:

[0344] 12. Determine the matrix product of the fifth characteristic sequence and the sixth characteristic sequence to obtain a seventh characteristic sequence, wherein the sixth characteristic sequence is the transpose of the fifth characteristic sequence.

[0345] After obtaining the second similarity, the video processing device performs the following steps in the process of executing step 9: 13. Obtain the confidence of the second temporal action nomination based on the first similarity and the second similarity.

[0346] In one possible implementation, the video processing device fuses the first similarity (i.e., the fourth feature sequence) and the third feature sequence to obtain the confidence of the second temporal action nomination, which is recorded as the third intermediate confidence. The video processing device fuses the first similarity (i.e., the seventh feature sequence) and the third feature sequence to obtain the confidence of the second temporal action nomination, which is recorded as the fourth intermediate confidence. For example, the video processing device obtains the third intermediate confidence by adding the fourth feature sequence and the third feature sequence. The video processing device obtains the fourth intermediate confidence by adding the seventh feature sequence and the third feature sequence. The video processing device obtains the confidence of the second temporal action nomination by determining the average of the third intermediate confidence and the fourth intermediate confidence.

[0347] In another possible implementation, the video processing device fuses the first similarity (i.e., the fourth feature sequence), the second similarity (i.e., the seventh feature sequence), and the third feature sequence to enrich the information corresponding to the third feature sequence and obtain the confidence of the second temporal action nomination. For example, the video processing device can obtain the confidence of the second temporal action nomination by adding the fourth feature sequence, the seventh feature sequence, and the third feature sequence.

[0348] Since there is a correlation between different time-sequence action nominations, the video processing device obtains the second similarity of the correlation information between the second time-sequence action nomination and the third time-sequence action nomination by determining the similarity between the second time-sequence action nomination and the third time-sequence action nomination. The video processing device further determines the confidence of the second time-sequence action nomination based on the first similarity, the second similarity and the third feature sequence, which can improve the accuracy of the confidence of the second time-sequence action nomination.

[0349] As an optional implementation manner, the video processing device performs the following steps during the process of executing step 13:

[0350] 14. Perform a first feature extraction process on the first fused feature sequence to obtain a first confidence level of the second temporal action nomination.

[0351] In the embodiment of the present application, the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence. Optionally, the video processing device obtains the first fused feature sequence by adding the third feature sequence and the fourth feature sequence.

[0352] In the embodiment of the present application, the first feature extraction process includes at least one of the following processes: convolution process, pooling process. Optionally, the convolution process in this step can be implemented by a two-dimensional (ie, height and width) convolution kernel.

[0353] The video processing device performs a first feature extraction process on the first fused feature sequence to obtain a confidence level of the second time-series action nomination, which is the first confidence level.

[0354] 15. Perform a second feature extraction process on the second fused feature sequence to obtain a second confidence level of the second temporal action nomination.

[0355] In the embodiment of the present application, the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence. Optionally, the video processing device obtains the second fused feature sequence by adding the third feature sequence and the seventh feature sequence.

[0356] In the embodiment of the present application, the second feature extraction process includes at least one of the following processes: convolution process, pooling process. Optionally, the convolution process in this step can be implemented by a two-dimensional (ie, height and width) convolution kernel.

[0357] The video processing device performs a second feature extraction process on the second fused feature sequence to obtain a confidence level of the second time-series action nomination, which is the second confidence level.

[0358] 16. Determine the average of the first confidence and the second confidence as the confidence of the second sequential action nomination.

[0359] As an optional implementation, the first feature extraction process is implemented by u-level convolution processing, and the second feature extraction process is implemented by v-level convolution processing, where u and v are both integers greater than 1. The u-level convolution processing and the v-level convolution processing are both in the form of series connection, that is, the output data of the previous level convolution processing is the input data of the next level convolution processing.

[0360] Before executing step 16, the video processing device further executes the following steps:

[0361] 17. The first feature map and the second feature map are combined to obtain a third feature map.

[0362] In an embodiment of the present application, the first feature map is the output data of the c-th level convolution processing in the u-level convolution processing, the second feature map is the output data of the d-th level convolution processing in the v-level convolution processing, and the size of the first feature map is the same as the size of the second feature map, where c is a positive integer less than u, and d is a positive integer less than v.

[0363] It should be understood that the video processing device can reduce the dimension of the fourth feature sequence by performing convolution processing on the fourth feature sequence to obtain the first feature map. Specifically, the video processing device removes the timing dimension in the fourth feature sequence by performing convolution processing on the fourth feature sequence to obtain the first feature map. Similarly, the video processing device removes the timing dimension in the seventh feature sequence by performing convolution processing on the seventh feature sequence to obtain the second feature map. That is, both the first feature map and the second feature map are two-dimensional images. Therefore, in the embodiment of the present application, the size of the first feature map is: the height of the first feature map * the width of the first feature map, and the size of the second feature map is: the height of the second feature map * the width of the second feature map.

[0364] In a possible implementation manner, the video processing device obtains a third feature map by adding the first feature map and the second feature map.

[0365] 18. Perform a third feature extraction process on the third feature graph to obtain a third confidence level of the second sequential action nomination.

[0366] In the embodiment of the present application, the third feature extraction process includes at least one of the following processes: convolution process, pooling process. Optionally, the convolution process in this step can be implemented by a two-dimensional (ie, height and width) convolution kernel.

[0367] The video processing device obtains the confidence level of the second sequential action nomination by performing a third feature extraction process on the third feature map, which is the third confidence level.

[0368] After obtaining the third confidence level, the video processing device performs the following steps in the process of executing step 16: 19. Determine the average of the first confidence level, the second confidence level and the third confidence level as the confidence level of the first sequential action nomination.

[0369] Since the third feature map corresponds to both the correlation information between different channels in the second time-series nominated action and the correlation information between the second time-series nominated action and the third time-series nominated action, and the third confidence is obtained based on the third feature map, the video processing device can improve the accuracy of the confidence of the second time-series nominated action by executing step 19.

[0370] As an optional implementation manner, the video processing device performs the following steps during the process of executing step 13:

[0371] 20. Perform feature extraction processing on the first fused feature sequence to obtain a first reference feature sequence.

[0372] In the embodiment of the present application, the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence. Optionally, the video processing device obtains the first fused feature sequence by adding the third feature sequence and the fourth feature sequence.

[0373] The feature extraction process in this step includes at least one of the following processes: convolution process, pooling process. Optionally, the convolution process in this step can be implemented by a two-dimensional (ie, height and width) convolution kernel.

[0374] In this step, the video processing device obtains a first reference feature sequence by performing feature extraction processing on the first fused feature sequence.

[0375] 21. Perform feature extraction processing on the second fused feature sequence to obtain a second reference feature sequence.

[0376] In the embodiment of the present application, the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence. Optionally, the video processing device obtains the second fused feature sequence by adding the third feature sequence and the seventh feature sequence.

[0377] The feature extraction process in this step includes at least one of the following processes: convolution process, pooling process. Optionally, the convolution process in this step can be implemented by a two-dimensional (ie, height and width) convolution kernel.

[0378] In this step, the video processing device obtains a second reference feature sequence by performing a second feature extraction process on the second fused feature sequence.

[0379] 22. Fusing the first reference feature sequence and the second reference feature sequence to obtain a third reference feature sequence.

[0380] Optionally, the fusion processing in this step may be to add the first reference feature sequence and the second reference feature sequence.

[0381] 23. Perform feature extraction processing on the third reference feature sequence to obtain the confidence level of the second time-series action nomination.

[0382] The feature extraction process in this step includes at least one of the following processes: convolution process, pooling process. Optionally, the convolution process in this step can be implemented by a two-dimensional (ie, height and width) convolution kernel.

[0383] The video processing device obtains the confidence level of the second time-sequence action nomination by performing feature extraction processing on the third reference feature sequence.

[0384] By executing steps 20 to 23, the video processing device can use the information contained in the first fused feature sequence and the information contained in the second fused feature sequence to obtain the confidence of the second temporal action nomination, thereby improving the accuracy of the confidence of the second temporal action nomination.

[0385] The present application also provides a time sequence action nomination network, which can be used to implement the technical solution disclosed above. Figure 2 , Figure 2 This is a schematic diagram of the structure of a sequential action nomination network provided in an embodiment of the present application. Figure 2 As shown, the temporal action nomination network includes: a basic feature extraction module, a boundary probability sequence generation module, and a nomination relationship module.

[0386] The basic feature extraction module processes the video stream to be processed to obtain a first feature sequence. The boundary probability sequence generation module processes the video stream to be processed to obtain a first temporal boundary probability sequence of the first action; or the boundary probability sequence generation module processes the first feature sequence to obtain a first temporal boundary probability sequence of the first action. The nomination relationship module processes the first feature sequence to obtain the confidence of the first temporal action nomination.

[0387] Before using the temporal action nomination network to process the video stream to be processed, the temporal action nomination network needs to be trained. Therefore, the embodiment of the present application also provides a method for training the temporal action nomination network.

[0388] The execution subject of the training method may be a video processing device or may not be a video processing device, that is, the execution subject of the training method of the time-series action nomination neural network and the execution subject of processing the video stream to be processed using the time-series action nomination network may be the same or different, and the embodiment of the present application does not limit the execution subject of this embodiment. For the convenience of expression, the execution subject of the training method is referred to as a training device below, and optionally, the training device can be any of the following: a mobile phone, a computer, a tablet computer, a server, and a processor.

[0389] See also Figure 3 , Figure 3 A flowchart of a method for training a sequential action nomination network is provided for an embodiment of the present application.

[0390] 301. Obtain a first training video stream and a neural network to be trained.

[0391] In the embodiment of the present application, the first training video stream includes at least two frames of images. The first training video stream can be a video stream collected in real time by the training device through the camera component; the first video stream can also be a video stream collected non-real time by the training device.

[0392] In a possible implementation manner, the training device obtains the first training video stream by downloading from the Internet.

[0393] In another possible implementation, the training device uses the video stream input by the user through an input component as the first training video stream. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, an audio input device, and the like.

[0394] In another possible implementation, the training device acquires the first training video stream through a camera assembly, wherein the camera assembly includes a camera.

[0395] In another possible implementation, a communication connection is provided between the training device and the second data terminal, and the training device receives a video stream sent by the second data terminal through the communication connection as the first training video stream.

[0396] In an embodiment of the present application, the neural network to be trained is a neural network including a first feature extraction module and a first upsampling module. The first feature extraction module includes at least one of the following network layers: a convolution layer, a pooling layer, a normalization layer, and an activation layer. The first upsampling module includes at least one of the following network layers: a deconvolution layer, an upsampling layer, a normalization layer, and an activation layer.

[0397] In one implementation of obtaining a neural network to be trained, a training device receives a neural network input by a user through an input component as the neural network to be trained. The input component includes: a keyboard, a mouse, a touch screen, a touch pad, an audio input device, etc.

[0398] In another possible implementation, a communication connection is provided between the training device and the third data terminal, and the training device receives the neural network sent by the third data terminal through the communication connection as the neural network to be trained.

[0399] 302. Use the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence.

[0400] The implementation process of the a-level feature extraction process in this step can refer to the implementation process of the n-level feature extraction process in step 102, and this step will not be repeated. Optionally, a=n.

[0401] The training device extracts semantic information of the first training video stream by performing a-level feature extraction processing on the first training video stream, thereby obtaining a third intermediate feature sequence.

[0402] 303. Use the first upsampling module to perform level b upsampling processing on the third intermediate feature sequence to obtain a fourth time series boundary probability sequence of the first action.

[0403] In the embodiment of the present application, the fourth temporal boundary probability sequence includes the probability that the image in the first training video stream is the start frame of the first action, and the probability that the image in the first training video stream is the end frame of the first action.

[0404] The implementation process of the b-level upsampling process in this step can refer to the implementation process of the m-level upsampling process in step 103, and this step will not be repeated. Optionally, b=m.

[0405] In an embodiment of the present application, the input data of the k+1th level upsampling process in the b-level upsampling process is the fusion data between the output data of the kth level upsampling process in the above-mentioned b-level upsampling process and the input data of the f-th level feature extraction process in the above-mentioned a-level feature extraction process; the resolution of the output data of the k-th level upsampling process and the input data of the f-th level feature extraction process are the same; the above-mentioned k is a positive integer less than the above-mentioned b, and the above-mentioned f is a positive integer not greater than the above-mentioned a.

[0406] As an optional implementation, the structural diagram of the first feature extraction module and the first up-sampling module is as follows: Figure 4 shown.

[0407] 304. Obtain the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream.

[0408] In an embodiment of the present application, the label of the first training video stream includes a temporal action nomination of the first action in the first training video stream. For example, the first training video stream includes a first frame image, a second frame image, a third frame image, and a fourth frame image, and the playback order of the first training video stream is the first frame image, the second frame image, the third frame image, and the fourth frame image. If the first frame image and the fourth frame image do not contain the first action, and the second frame image and the third frame image both contain the first action. Then the label of the first training video stream includes: the second frame image to the third frame image are temporal action nominations for the first action.

[0409] The training device can obtain a first intermediate loss based on the difference between the starting probability sequence in the fourth temporal boundary probability sequence and the label of the first training video stream. The second intermediate loss can be obtained based on the difference between the ending probability sequence in the fourth temporal boundary probability sequence and the label of the first training video stream. After obtaining the first intermediate loss and the second intermediate loss, the training device can obtain the loss of the neural network to be trained by determining the sum of the first intermediate loss and the second intermediate loss.

[0410] 305. Update the parameters of the neural network to be trained based on the loss of the neural network to be trained to obtain the time sequence action nomination network.

[0411] As an optional implementation, the training device further performs the following steps before performing step 302:

[0412] 24. Use the basic feature extraction module to perform feature extraction processing on the first training video stream to obtain a seven-feature sequence.

[0413] In the embodiment of the present application, the eighth feature sequence corresponds to the semantic information of the first training video stream and the optical flow information of the background in the first training video stream, wherein the background includes the content in the first training video stream except the first action.

[0414] In an embodiment of the present application, the basic feature extraction module may be a two-stream convolutional network, which is used to extract semantic information and optical flow information.

[0415] After obtaining the seven-feature sequence, the training device performs the following steps in the process of executing step 302:

[0416] 25. Use the first feature extraction module to perform level A feature extraction processing on the seven feature sequences to obtain the third intermediate feature sequence.

[0417] As an optional implementation, after executing step 24 and before executing step 304, the training device further executes the following steps:

[0418] 26. Perform time-series reversal processing on the eighth characteristic sequence to obtain a ninth characteristic sequence whose playback order is opposite to that of the eighth characteristic sequence.

[0419] 27. Use the second feature extraction module to perform x-level feature extraction processing on the ninth feature sequence to obtain a fourth intermediate feature sequence.

[0420] The feature extraction process in this step can refer to the feature extraction process in step 302, and will not be described in detail here. Optional x=a.

[0421] 28. Use the second upsampling module to perform y-level upsampling processing on the second intermediate feature sequence to obtain the fifth time series boundary probability sequence of the first action.

[0422] In this step, the fifth temporal boundary probability sequence includes the probability that the image in the first training video stream is the start frame of the first action, and the probability that the image in the first training video stream is the end frame of the first action.

[0423] The up-sampling process in this step may refer to the up-sampling process in step 303, and will not be described in detail here. Optional y=b.

[0424] In a possible implementation, the input data of the w+1th level upsampling process in the x-level upsampling process is the fusion data between the output data of the w-th level upsampling process in the x-level upsampling process and the input data of the e-th level feature extraction process in the y-level feature extraction process. The output data of the w-th level upsampling process and the input data of the e-th level feature extraction process have the same resolution, where e is a positive integer less than y, and w is a positive integer not greater than x.

[0425] Optionally, the second feature extraction module has the same structure as the first feature extraction module, and the parameters of the second feature extraction module are the same as the parameters of the first feature extraction module. The second upsampling module has the same structure as the first upsampling module, and the parameters of the second upsampling module are the same as the parameters of the first upsampling module.

[0426] 29. The neural network to be trained fuses the fourth time series boundary probability sequence and the fifth time series boundary probability sequence to obtain the sixth time series boundary probability sequence of the first action.

[0427] The implementation method of this step can be referred to the implementation method of the video processing device fusing the first temporal boundary probability sequence and the second temporal boundary probability sequence in step 6, which will not be repeated here.

[0428] After obtaining the fifth time-series boundary probability sequence, the training device performs the following steps in the process of executing step 304:

[0429] 30. According to the difference between the sixth temporal boundary probability sequence and the label of the first training video stream, the loss of the network to be trained is obtained.

[0430] The training device can obtain a third intermediate loss based on the difference between the starting probability sequence in the sixth temporal boundary probability sequence and the label of the first training video stream. The fourth intermediate loss can be obtained based on the difference between the ending probability sequence in the sixth temporal boundary probability sequence and the label of the first training video stream. After obtaining the third intermediate loss and the fourth intermediate loss, the training device can obtain the loss of the neural network to be trained by determining the sum of the third intermediate loss and the fourth intermediate loss.

[0431] As an optional implementation, the neural network to be trained further includes a nomination relationship module. Before executing step 30, the training device further performs the following steps:

[0432] 31. The neural network to be trained extracts a feature sequence of the fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence.

[0433] In the embodiment of the present application, the fourth time sequence action nomination is a time sequence action nomination of the first action obtained according to the sixth time sequence boundary probability sequence. The tenth feature sequence includes: a third channel feature sequence and a fourth channel feature sequence, wherein the information corresponding to the third channel feature sequence is different from the information corresponding to the fourth channel feature sequence.

[0434] 32. Use the nomination relationship module to determine the similarity between the third channel feature sequence and the fourth channel feature sequence to obtain a third similarity.

[0435] The implementation method of this step can refer to the implementation method of obtaining the first similarity in step 8, which will not be repeated here.

[0436] In a possible implementation, the training device uses the nominated relationship module to determine the matrix product of the third channel feature sequence and the fourth channel feature sequence to obtain an eleventh feature sequence as the third similarity.

[0437] 33. The nomination relationship module obtains the confidence of the fourth time-series action nomination based on the third similarity and the tenth feature sequence.

[0438] The implementation of this step can refer to the implementation of step 9, which will not be repeated here. After obtaining the confidence of the fourth time sequence action nomination, the training device performs the following steps during the execution of step 30:

[0439] 34. Obtain a first loss based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream.

[0440] The implementation of this step can refer to the implementation of step 30. It should be understood that in this step, the training device obtains not the loss of the neural network to be trained, but the first loss based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream.

[0441] 35. Obtain a second loss based on the difference between the confidence of the fourth temporal action nomination and the label of the first training video stream.

[0442] Optionally, the training device substitutes the confidence of the fourth temporal action nomination into the regression loss function (smooth L1) to obtain the second loss.

[0443] 36. The sum of the first loss and the second loss is used as the loss of the neural network to be trained.

[0444] As an optional implementation, the training device further performs the following steps before performing step 33:

[0445] 37. The nomination relationship module determines the similarity between the fourth time-sequence action nomination and the fifth time-sequence action nomination to obtain a fourth similarity.

[0446] In the embodiment of the present application, the fifth time sequence action nomination is a time sequence action nomination of the first action obtained according to the sixth time sequence boundary probability sequence, and the fifth time sequence action nomination is different from the fourth time sequence action nomination. The implementation method of this step can refer to the implementation method of step 10, which will not be repeated here.

[0447] After obtaining the fourth similarity, the training device performs the following steps during the execution of step 33:

[0448] 38. Based on the third similarity, the fourth similarity and the tenth feature sequence, obtain the confidence of the fourth temporal action nomination.

[0449] The implementation method of this step can refer to step 91 and will not be repeated here.

[0450] As an optional implementation, the training device performs the following steps during the execution of step 31:

[0451] 39. The neural network to be trained extracts the tenth feature sequence and the feature sequence nominated by the fifth sequential action from the eighth feature sequence to obtain a twelfth feature sequence.

[0452] After obtaining the twelfth feature sequence, the training device performs the following steps during the execution of step 37:

[0453] 40. The nomination relationship module determines the matrix product of the twelfth feature sequence and the thirteenth feature sequence to obtain a fourteenth feature sequence as the fourth similarity, wherein the thirteenth feature sequence is the transpose of the twelfth feature sequence.

[0454] As an optional implementation, the training device performs the following steps during the execution of step 38:

[0455] 41. Perform a fourth feature extraction process on the third fused feature sequence to obtain a fourth confidence level of the fourth temporal action nomination.

[0456] In the embodiment of the present application, the third fused feature sequence is obtained by fusing the tenth feature sequence and the eleventh feature sequence. Optionally, the video processing device obtains the third fused feature sequence by adding the tenth feature sequence and the eleventh feature sequence.

[0457] The implementation method of this step can refer to the implementation method of step 14, wherein the third fused feature sequence corresponds to the first fused feature sequence, the fourth feature extraction process corresponds to the first feature extraction process, and the fourth confidence level of the fourth time-series action nomination corresponds to the first confidence level of the second time-series action nomination.

[0458] 42. Perform a fifth feature extraction process on the fourth fused feature sequence to obtain a fifth confidence level of the fourth temporal action nomination.

[0459] In the embodiment of the present application, the fourth fused feature sequence is obtained by fusing the tenth feature sequence and the fourteenth feature sequence. Optionally, the video processing device obtains the fourth fused feature sequence by adding the tenth feature sequence and the fourteenth feature sequence.

[0460] The implementation method of this step can refer to the implementation method of step 15, wherein the fourth fused feature sequence corresponds to the second fused feature sequence, the fifth feature extraction process corresponds to the second feature extraction process, and the fifth confidence level of the fourth time-series action nomination corresponds to the second confidence level of the second time-series action nomination.

[0461] 43. Determine an average of the fourth confidence level and the fifth confidence level as the confidence level of the fourth sequential action nomination.

[0462] The implementation method of this step can refer to the implementation method of step 16, which will not be repeated here.

[0463] As an optional implementation, the fourth feature extraction process is implemented by g-level convolution processing, and the fifth feature extraction process is implemented by h-level convolution processing, wherein g and h are both integers greater than 1. The g-level convolution processing and the h-level convolution processing are both in the form of series connection, that is, the output data of the previous level convolution processing is the input data of the next level convolution processing.

[0464] Before executing step 43, the training device further executes the following steps:

[0465] 44. The fourth feature map and the fifth feature map are combined to obtain the sixth feature map.

[0466] In an embodiment of the present application, the fourth feature map is the output data of the z-th level convolution processing in the g-level convolution processing, and the fifth feature map is the output data of the t-th level convolution processing in the above-mentioned h-level convolution processing. The size of the fourth feature map is the same as the size of the fifth feature map, wherein z is a positive integer less than g, and t is a positive integer less than h.

[0467] The implementation method of this step may refer to the implementation method of step 17, wherein the fourth feature map corresponds to the first feature map, the fifth feature map corresponds to the second feature map, and the sixth feature map corresponds to the third feature map.

[0468] 45. Perform sixth feature extraction processing on the sixth feature map to obtain a sixth confidence level of the fourth sequential action nomination.

[0469] The implementation method of this step can refer to the implementation method of step 18, wherein the third feature map corresponds to the sixth feature map, the fourth feature extraction process corresponds to the sixth feature extraction process, and the third confidence level of the second time sequence action nomination corresponds to the sixth confidence level of the fourth time sequence action nomination.

[0470] After obtaining the sixth confidence level, the training device performs the following steps during the execution of step 43:

[0471] 46. ​​Determine an average of the fourth confidence level, the fifth confidence level, and the sixth confidence level as the confidence level of the first sequential action nomination.

[0472] As an optional implementation, the structural diagram of the nomination relationship module in the embodiment of the present application is as follows: Figure 5 As shown. Figure 5 As shown, the nomination relationship module includes: a feature sequence generation submodule, an inter-nomination correlation submodule, an inter-channel correlation submodule and a confidence generation submodule. The feature sequence generation submodule is used to execute step 24. The inter-channel correlation submodule is used to execute steps 28 and 37. The inter-nomination correlation submodule is used to execute steps 37, 38 and 42. The confidence generation submodule is used to execute steps 33, 37, 43, 44, 45 and 47.

[0473] As an optional implementation manner, the training device obtains the first training video stream by performing the following steps:

[0474] 47. Obtain a first positive sample video stream set and a first negative sample video stream set.

[0475] In the embodiment of the present application, the video streams in the first positive sample video stream set all include the temporal action nomination of the first action, and the video streams in the first negative sample video stream set all do not include the temporal action nomination of the first action.

[0476] In the embodiment of the present application, the first positive sample video stream set includes a first scale video stream set and a second scale video stream set, wherein the scale of the video stream in the first scale video stream set is in a first scale interval, the scale of the video stream in the second scale video stream set is in a second scale interval, and the first scale interval has no intersection with the second scale interval.

[0477] The video stream in the first positive sample video stream set is called a positive sample video stream. In the embodiment of the present application, the scale of the positive sample video stream is the ratio of the reference resolution to the resolution of the positive sample video stream, and the positive sample video stream is the video stream in the first positive sample video stream set.

[0478] For example, the first positive sample video stream set includes a positive sample video stream A. Positive sample video stream A includes image a, image b, and image c, where image a and image b both contain the first action, and image c does not contain the first action. At this time, the reference resolution of the positive sample video stream A is the playback time of 2 frames of images, and the resolution of the positive sample video stream A is the playback time of 3 frames of images. If the playback time of any image is equal, the scale of the positive sample video stream A is: 2 / 3.

[0479] 48. When the number of video streams included in the first-scale video stream set is less than the number of video streams included in the second video stream set, a first training video stream subset is obtained by sampling from the first-scale video stream set at a first sampling rate, and a second training video stream subset is obtained by sampling from the first-scale video stream set at a second sampling rate, wherein the first sampling rate is greater than the second sampling rate.

[0480] 49. Select a video stream from the first training video stream subset and / or the second training video stream subset and / or the first negative sample video stream set as the first training video stream.

[0481] That is, in the embodiment of the present application, the training data for the neural network to be trained includes a first training video stream subset, a second training video stream subset, and the above-mentioned first negative sample video stream set.

[0482] By executing steps 47 to 49, the training device can balance the number of positive sample video streams of different scales in the training data of the neural network to be trained, thereby improving the training effect of the neural network to be trained.

[0483] As an optional implementation, the training device performs the following steps during the execution of step 47:

[0484] 50. Obtain a training video stream set.

[0485] In an embodiment of the present application, the training video stream set includes a second positive sample video stream set and a second negative sample video stream set, wherein the video streams in the second positive sample video stream set all include temporal action nominations for the first action, and the video streams in the second negative sample all do not include temporal action nominations for the first action.

[0486] 51. When the number of video streams included in the second positive sample video stream set is greater than the number of video streams included in the second negative sample video stream set, the second positive sample video stream set is obtained by sampling from the second positive sample video stream set at a third sampling rate, and the second negative sample video stream set is obtained by sampling from the second negative sample video stream set at a fourth sampling rate, wherein the third sampling rate is less than the fourth sampling rate.

[0487] 52. When the number of video streams included in the second positive sample video stream set is less than the number of video streams included in the second negative sample video stream set, the second positive sample video stream set is obtained by sampling from the second positive sample video stream set at a fifth sampling rate, and the second negative sample video stream set is obtained by sampling from the second negative sample video stream set at a sixth sampling rate, wherein the fifth sampling rate is greater than the sixth sampling rate.

[0488] By executing steps 50 to 52, the training device can balance the number of positive sample video streams and the number of negative sample video streams in the training data of the neural network to be trained, thereby improving the training effect of the neural network to be trained.

[0489] As an optional implementation, the training device performs the following steps during the execution of step 36:

[0490] 53. Determine the sum of the first loss and the second loss to obtain the third loss.

[0491] 54. Determine the regularization function value of the neural network to be trained to obtain the fourth loss.

[0492] In the embodiment of the present application, the regularization function value may be one of the following: an L1 regularization function value, an L2 regularization function value. Optionally, the regularization function value is an L2 regularization function value.

[0493] 55. The sum of the third loss and the fourth loss is used as the loss of the neural network to be trained.

[0494] The training device obtains the loss of the neural network to be trained by adding the third loss on the basis of the first loss and the second loss, and then updates the parameters of the neural network to be trained according to the loss of the neural network to be trained to obtain the sequential action nomination network, thereby reducing the probability of overfitting of the sequential action nomination network.

[0495] Based on the technical solution provided by the embodiments of the present application and / or the sequential action nomination network provided by the embodiments of the present application, the embodiments of the present application also provide several possible application scenarios.

[0496] Scenario 1: In the case of a large flow of people, if someone falls at the escalator, it will bring great safety hazards. Therefore, the managers of the smart park hope to monitor whether there are pedestrians falling at the escalator through monitoring software to reduce the occurrence of safety accidents. At this time, the first action is falling, and the video processing device can extract the video stream containing the fall from the monitoring video stream of the smart park by running the monitoring software, executing the method described above, or running the time-series nomination network described above. Optionally, the above-mentioned video processing device can be a server.

[0497] Scenario 2: With the increase in online videos, it is very important to find the videos that users want to watch from the massive amount of videos.

[0498] When receiving the video information of interest input by the user, the video processing device can find the videos of interest to the user from the massive network videos by executing the method described above or running the temporal nomination network described above, and recommend these videos to the user.

[0499] For example, Zhang San is a basketball fan and wants to watch basketball videos on his mobile phone, so he inputs a command into the mobile phone to select basketball videos as videos of interest. In this case, the first action is playing basketball, and the video processing device is the mobile phone. By executing the method described above or running the temporal action nomination network described above, the mobile phone can find videos related to basketball from a large number of online videos and recommend these videos related to basketball to Zhang San.

[0500] Optionally, when the number of first action sequence nominations is relatively large, the video processing device may obtain the boundary probability sequence of each first action sequence nomination and the confidence of each first action sequence nomination according to the method described above. The video processing device may further obtain the quality score of the first action sequence nomination according to the sequence boundary probability sequence of the first action sequence nomination and the confidence of the first action sequence nomination. The video processing device sorts at least one first action sequence nomination in descending order of quality score to obtain the sequence nomination order of the first action, and displays at least one first action sequence nomination according to the sequence nomination order of the first action.

[0501] For example, the start frame of the first temporal action nomination A is image a and the end frame is image b, wherein the start probability of image a is p1 and the end probability of image b is p2. Assume that the confidence of the first temporal action nomination A is p3. The video processing device obtains the quality score of the first temporal action nomination A by determining the product of p1, p2 and p3.

[0502] The video processing device displays at least one first time-sequence action nomination according to the time-sequence action nomination order of the first action, which can improve the accuracy of recommending time-sequence action nominations to users.

[0503] For example, in scenario 2, the video processing device first determines the quality scores of all basketball-related videos, and then recommends the 10 videos with the highest quality scores to Zhang San.

[0504] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0505] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0506] See also Figure 6 , Figure 6 The schematic diagram of the structure of a video processing device provided in an embodiment of the present application includes: an acquisition unit 11, a feature extraction processing unit 12, and an upsampling processing unit 13, wherein:

[0507] An acquisition unit 11 is used to acquire a video stream to be processed;

[0508] A feature extraction processing unit 12 is used to perform n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence;

[0509] The upsampling processing unit 13 is used to perform m-level upsampling processing on the first intermediate feature sequence to obtain a first temporal boundary probability sequence of the first action; the first temporal boundary probability sequence includes the probability that the image in the video stream to be processed is the start frame of the first action, and the probability that the image in the video stream to be processed is the end frame of the first action;

[0510] The input data of the i+1th level upsampling process in the m-level upsampling process is the fusion data of the output data of the i-th level upsampling process in the m-level upsampling process and the input data of the j-th level feature extraction process in the n-level feature extraction process; the resolution of the output data of the i-th level upsampling process and the input data of the j-th level feature extraction process are the same; i is a positive integer less than m, and j is a positive integer not greater than n.

[0511] In combination with any embodiment of the present application, the feature extraction processing unit 12 is further used for performing feature extraction processing on the video stream to be processed to obtain a first feature sequence before performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence; semantic information of the video stream to be processed and optical flow feature information of the background in the video stream to be processed corresponding to the first feature sequence; the background includes content in the video stream to be processed except the first action;

[0512] The feature extraction processing unit 12 is further used to perform the n-level feature extraction processing on the first feature sequence to obtain the first intermediate feature sequence.

[0513] In combination with any embodiment of the present application, the video processing device 1 further includes: a flip unit 14 and a first processing unit 15;

[0514] The flip unit 14 is used to perform time-sequence flipping processing on the first feature sequence to obtain a second feature sequence whose playing order is opposite to the first feature sequence;

[0515] The feature extraction processing unit 12 is used to perform p-level feature extraction processing on the second feature sequence to obtain a second intermediate feature sequence;

[0516] The upsampling processing unit 13 is used to perform q-level upsampling processing on the second intermediate feature sequence to obtain a second temporal boundary probability sequence of the first action; the second temporal boundary probability sequence includes the probability that the image in the video stream to be processed is the start frame of the first action, and the probability that the image in the video stream to be processed is the end frame of the first action;

[0517] The input data of the r+1th level upsampling process in the qth level upsampling process is the fusion data of the output data of the rth level upsampling process in the qth level upsampling process and the input data of the sth level feature extraction process in the pth level feature extraction process; the output data of the rth level upsampling process and the input data of the sth level feature extraction process have the same resolution; r is a positive integer less than q, and s is a positive integer not greater than p;

[0518] The first processing unit 15 is configured to obtain the at least one first time sequence action nomination according to the first time sequence boundary probability sequence and the second time sequence boundary probability sequence.

[0519] In combination with any embodiment of the present application, the input data of the j-th level feature extraction processing includes shallow feature information of the video stream to be processed; the s-th level feature extraction processing includes shallow feature information of the first feature sequence.

[0520] In combination with any embodiment of the present application, the first processing unit 15 is used to:

[0521] fusing the first time series boundary probability sequence and the second time series boundary probability sequence to obtain a third time series boundary probability sequence of the first action;

[0522] According to the third temporal boundary probability sequence, the at least one first temporal action nomination is obtained.

[0523] In combination with any embodiment of the present application, the at least one first time-sequence action nomination includes a second time-sequence action nomination, and the video processing device further includes a second processing unit 16;

[0524] The second processing unit 16 is used for:

[0525] Extracting a feature sequence nominated by the second time-series action from the first feature sequence as a third feature sequence; the third feature sequence includes: a first channel feature sequence and a second channel feature sequence;

[0526] Determine the similarity between the first channel feature sequence and the second channel feature sequence to obtain a first similarity;

[0527] The confidence of the second time-series action nomination is obtained according to the first similarity and the third feature sequence.

[0528] In combination with any embodiment of the present application, the second processing unit 16 is used to:

[0529] Determine the matrix product of the first channel feature sequence and the second channel feature sequence to obtain a fourth feature sequence as the first similarity.

[0530] In combination with any implementation manner of the present application, the at least one first time-sequence action nomination further includes a third time-sequence action nomination different from the second time-sequence action nomination;

[0531] The second processing unit 16 is configured to determine the similarity between the second time-sequence action nomination and the third time-sequence action nomination to obtain a second similarity before obtaining the confidence of the second time-sequence action nomination based on the first similarity and the third feature sequence;

[0532] The first processing unit 15 is configured to obtain the confidence of the second time-sequence action nomination according to the first similarity, the second similarity and the third feature sequence.

[0533] In combination with any embodiment of the present application, the second processing unit 16 is used to:

[0534] Extracting the third feature sequence and the feature sequence nominated by the third sequential action from the first feature sequence to obtain a fifth feature sequence;

[0535] The matrix product of the fifth characteristic sequence and the sixth characteristic sequence is determined to obtain a seventh characteristic sequence as the second similarity; the sixth characteristic sequence is the transpose of the fifth characteristic sequence.

[0536] In combination with any embodiment of the present application, the first processing unit 15 is used to:

[0537] Performing a first feature extraction process on the first fused feature sequence to obtain a first confidence level of the second time-series action nomination; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence;

[0538] Performing a second feature extraction process on the second fused feature sequence to obtain a second confidence level of the second time-series action nomination; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence;

[0539] An average of the first confidence and the second confidence is determined as the confidence of the second time-series action nomination.

[0540] In combination with any embodiment of the present application, the first feature extraction process is implemented by u-level convolution processing, and the second feature extraction process is implemented by v-level convolution processing; both u and v are integers greater than 1;

[0541] The video processing device further includes a third processing unit 17; the third processing unit 17 is used to fuse the first feature map and the second feature map to obtain a third feature map before determining the average of the first confidence level and the second confidence level as the confidence level of the first time-series action nomination; the first feature map is the output data of the c-th level convolution processing in the u-level convolution processing, and the second feature map is the output data of the d-th level convolution processing in the v-level convolution processing; the size of the first feature map is the same as the size of the second feature map; c is a positive integer less than u, and d is a positive integer less than v;

[0542] The feature extraction processing unit 12 is used to perform a third feature extraction process on the third feature graph to obtain a third confidence level of the second time-series action nomination;

[0543] The third processing unit is used to determine an average of the first confidence level, the second confidence level and the third confidence level as the confidence level of the first sequential action nomination.

[0544] In combination with any embodiment of the present application, the first processing unit 15 is used to:

[0545] Performing feature extraction processing on the first fused feature sequence to obtain a first reference feature sequence; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence;

[0546] Performing feature extraction processing on the second fused feature sequence to obtain a second reference feature sequence; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence;

[0547] fusing the first reference feature sequence and the second reference feature sequence to obtain a third reference feature sequence;

[0548] Perform feature extraction processing on the third reference feature sequence to obtain the confidence level of the second time-series action nomination.

[0549] In combination with any embodiment of the present application, the device 1 further includes:

[0550] A sorting unit 18 is used to sort the at least one first time-sequence action nomination in descending order of quality score to obtain a time-sequence action nomination order of the first action; the quality score is obtained according to the time-sequence boundary probability sequence of the first time-sequence action nomination and the confidence of the first time-sequence action nomination;

[0551] The display unit 19 is configured to display the at least one first time-sequence action nomination according to the time-sequence action nomination order of the first action.

[0552] In combination with any embodiment of the present application, the video processing method performed by the video processing device is applied to a temporal action nomination network; the video processing device further includes a training unit 20; the training unit is used to perform a training process of the temporal action nomination network;

[0553] The training process of the temporal action nomination network includes:

[0554] Acquire a first training video stream and a neural network to be trained; the neural network to be trained includes a first feature extraction module and a first upsampling module;

[0555] Using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence;

[0556] Using the first upsampling module to perform b-level upsampling processing on the third intermediate feature sequence to obtain a fourth temporal boundary probability sequence of the first action; the fourth temporal boundary probability sequence includes a probability that the image in the first training video stream is a start frame of the first action, and a probability that the image in the first training video stream is an end frame of the first action;

[0557] The input data of the k+1th level upsampling process in the b-level feature extraction process is the fusion data of the output data of the kth level upsampling process in the b-level upsampling process and the input data of the f-th level feature extraction process in the a-level feature extraction process; the output data of the kth level upsampling process and the input data of the f-th level feature extraction process have the same resolution; k is a positive integer less than b, and f is a positive integer not greater than a;

[0558] Obtaining the loss of the neural network to be trained according to a difference between the fourth temporal boundary probability sequence and a label of the first training video stream; the label of the first training video stream includes a temporal action nomination of the first action in the first training video stream;

[0559] The parameters of the neural network to be trained are updated based on the loss of the neural network to be trained to obtain the temporal action nomination network.

[0560] In combination with any embodiment of the present application, the neural network to be trained further includes a basic feature extraction module;

[0561] Before using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence, the training process further includes:

[0562] Using the basic feature extraction module to perform feature extraction processing on the first training video stream, obtaining an eighth feature sequence; the eighth feature sequence corresponds to the semantic information of the first training video stream and the optical flow feature information of the background in the first training video stream; the background includes the content in the first training video stream except the first action;

[0563] The step of performing level A feature extraction processing on the first training video stream using the first feature extraction module to obtain a third intermediate feature sequence includes:

[0564] The first feature extraction module is used to perform a-level feature extraction processing on the seven feature sequences to obtain the third intermediate feature sequence.

[0565] In combination with any embodiment of the present application, the neural network to be trained further includes a second feature extraction module and a second upsampling module;

[0566] Before obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream, the training process further includes:

[0567] Performing time sequence reversal processing on the eighth characteristic sequence to obtain a ninth characteristic sequence whose playing order is opposite to that of the eighth characteristic sequence;

[0568] Using the second feature extraction module to perform x-level feature extraction processing on the ninth feature sequence to obtain a fourth intermediate feature sequence;

[0569] Using the second upsampling module to perform y-level upsampling processing on the second intermediate feature sequence, to obtain a fifth temporal boundary probability sequence of the first action; the fifth temporal boundary probability sequence includes a probability that an image in the first training video stream is a start frame of the first action, and a probability that an image in the first training video stream is an end frame of the first action; the playback order of the first training video stream is opposite to the playback order of the first training video stream;

[0570] The input data of the w+1th level upsampling process in the x-level upsampling process is the fusion data of the output data of the wth level upsampling process in the x-level upsampling process and the input data of the eth level feature extraction process in the y-level feature extraction process; the output data of the w-level upsampling process and the input data of the e-level feature extraction process have the same resolution; e is a positive integer less than y, and w is a positive integer not greater than x;

[0571] The neural network to be trained fuses the fourth time series boundary probability sequence and the fifth time series boundary probability sequence to obtain a sixth time series boundary probability sequence of the first action;

[0572] The step of obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream includes:

[0573] The loss of the neural network to be trained is obtained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream.

[0574] In combination with any implementation manner of the present application, the neural network to be trained further includes a nomination relationship module;

[0575] Before obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream, the training process further includes:

[0576] The neural network to be trained extracts a feature sequence of a fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence; the fourth temporal action nomination is a temporal action nomination of the first action obtained according to the sixth temporal boundary probability sequence; the tenth feature sequence includes: a third channel feature sequence and a fourth channel feature sequence;

[0577] Determine the similarity between the third channel feature sequence and the fourth channel feature sequence using the nominated relationship module to obtain a third similarity;

[0578] The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence;

[0579] The step of obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream includes:

[0580] Obtaining a first loss according to a difference between the sixth temporal boundary probability sequence and the label of the first training video stream;

[0581] Obtaining a second loss according to a difference between a confidence of the fourth temporal action nomination and a label of the first training video stream;

[0582] The sum of the first loss and the second loss is used as the loss of the neural network to be trained.

[0583] In combination with any embodiment of the present application, the using the nominated relationship module to determine the similarity between the third channel feature sequence and the fourth channel feature sequence to obtain a third similarity includes:

[0584] The nominated relationship module is used to determine the matrix product of the third channel feature sequence and the fourth channel feature sequence to obtain an eleventh feature sequence as the third similarity.

[0585] In combination with any implementation manner of the present application, before the nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence, the training process further includes:

[0586] The nomination relationship module determines the similarity between the fourth time-series action nomination and the fifth time-series action nomination to obtain a fourth similarity; the fifth time-series action nomination is a time-series action nomination of the first action obtained according to the sixth time-series boundary probability sequence, and the fifth time-series action nomination is different from the fourth time-series action nomination;

[0587] The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence, including:

[0588] The confidence of the fourth time-series action nomination is obtained according to the third similarity, the fourth similarity and the tenth feature sequence.

[0589] In combination with any embodiment of the present application, the neural network to be trained extracts the feature sequence of the fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence, including:

[0590] The neural network to be trained extracts the tenth feature sequence and the feature sequence nominated by the fifth time-series action from the eighth feature sequence to obtain a twelfth feature sequence;

[0591] The nomination relationship module determines the similarity between the fourth time-sequence action nomination and the fifth time-sequence action nomination to obtain a fourth similarity, including:

[0592] The nomination relationship module determines the matrix product of the twelfth feature sequence and the thirteenth feature sequence to obtain a fourteenth feature sequence as the fourth similarity; the thirteenth feature sequence is the transpose of the twelfth feature sequence.

[0593] In combination with any implementation manner of the present application, obtaining the confidence of the fourth time-series action nomination based on the third similarity, the fourth similarity and the tenth feature sequence includes:

[0594] Performing a fourth feature extraction process on the third fused feature sequence to obtain a fourth confidence level of the fourth time-series action nomination; the third fused feature sequence, the tenth feature sequence and the eleventh feature sequence are obtained;

[0595] Performing a fifth feature extraction process on the fourth fused feature sequence to obtain a fifth confidence level of the fourth time-series action nomination; the tenth feature sequence and the fourteenth feature sequence of the fourth fused feature sequence are obtained;

[0596] An average of the fourth confidence and the fifth confidence is determined as the confidence of the fourth sequential action nomination.

[0597] In combination with any embodiment of the present application, the fourth feature extraction process is implemented by g-level convolution processing, and the fifth feature extraction process is implemented by h-level convolution processing; both g and h are integers greater than 1;

[0598] Before determining the average of the fourth confidence and the fifth confidence as the confidence of the fourth sequential action nomination, the training process further includes:

[0599] The fourth feature map and the fifth feature map are merged to obtain a sixth feature map; the fourth feature map is output data of the z-th level convolution process in the g-level convolution process, and the fifth feature map is output data of the t-th level convolution process in the h-level convolution process; the size of the fourth feature map is the same as the size of the fifth feature map; z is a positive integer less than g, and t is a positive integer less than h;

[0600] Performing a sixth feature extraction process on the sixth feature graph to obtain a sixth confidence level of the fourth sequential action nomination;

[0601] The step of determining an average of the fourth confidence level and the fifth confidence level as the confidence level of the first time sequence action nomination includes:

[0602] An average of the fourth confidence, the fifth confidence, and the sixth confidence is determined as the confidence of the first sequential action nomination.

[0603] In combination with any implementation manner of the present application, the obtaining of the first training video stream includes:

[0604] Obtain a first positive sample video stream set and a first negative sample video stream set; the video streams in the first positive sample video stream set all contain the temporal action nomination of the first action; the video streams in the first negative sample video stream set all contain the temporal action nomination of the first action;

[0605] The first positive sample video stream set includes a first scale video stream set and a second scale video stream set; the scale of the video stream in the first scale video stream set is in a first scale interval, the scale of the video stream in the second scale video stream set is in a second scale interval, and the first scale interval has no intersection with the second scale interval; wherein the scale of the positive sample video stream is a ratio of a reference resolution to a resolution of the positive sample video stream; the positive sample video stream is a video stream in the first positive sample video stream set; the reference resolution is a resolution nominated by a sequential action of the first action;

[0606] When the number of video streams included in the first scaled video stream set is less than the number of video streams included in the second video stream set, sampling the first scaled video stream set at a first sampling rate to obtain a first training video stream subset, and sampling the first scaled video stream set at a second sampling rate to obtain a second training video stream subset; the first sampling rate is greater than the second sampling rate;

[0607] A video stream is selected from the first training video stream subset and / or the second training video stream subset and / or the first negative sample video stream set as the first training video stream.

[0608] In combination with any implementation manner of the present application, the obtaining of the first positive sample video stream set and the first negative sample video stream set includes:

[0609] Acquire a training video stream set; the training video stream set includes a second positive sample video stream set and a second negative sample video stream set; the video streams in the second positive sample video stream set all include the temporal action nomination of the first action; the video streams in the second negative sample all include the temporal action nomination of the first action;

[0610] In a case where the number of video streams included in the second positive sample video stream set is greater than the number of video streams included in the second negative sample video stream set, sampling from the second positive sample video stream set at a third sampling rate to obtain the second positive sample video stream set, and sampling from the second negative sample video stream set at a fourth sampling rate to obtain the second negative sample video stream set; the third sampling rate is less than the fourth sampling rate;

[0611] When the number of video streams included in the second positive sample video stream set is less than the number of video streams included in the second negative sample video stream set, the second positive sample video stream set is sampled from the second positive sample video stream set at a fifth sampling rate, and the second negative sample video stream set is sampled from the second negative sample video stream set at a sixth sampling rate; the fifth sampling rate is greater than the sixth sampling rate.

[0612] In this embodiment, the video processing device enriches the shallow feature information in the feature sequence by fusing the feature sequence obtained by upsampling the first intermediate feature sequence with the feature sequence obtained by feature extraction during upsampling of the first intermediate feature sequence, thereby improving the accuracy of the boundary probability sequence.

[0613] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0614] Figure 7 A schematic diagram of the hardware structure of a video processing device provided in an embodiment of the present application. The video processing device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines, or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0615] The processor 21 may be one or more graphics processing units (GPUs). When the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 21 may be a processor group consisting of multiple GPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0616] The memory 22 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0617] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0618] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 22 can be used to store the video stream to be processed obtained through the input device 23, or the memory 22 can also be used to store the first time boundary probability sequence obtained by the processor 21, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0619] Understandably, Figure 7 Only a simplified design of a video processing device is shown. In practical applications, the video processing device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all video processing devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0620] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0621] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0622] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0623] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0624] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0625] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0626] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A video processing method, characterized in that: The method comprises: Get the video stream to be processed; Performing n-level feature extraction processing on the video stream to be processed through n-layer feature extraction layers to obtain a first intermediate feature sequence, wherein the n-layer feature extraction layers are sequentially connected in series, and the feature extraction layers include at least one of the following processing: convolution processing and pooling processing; Performing m-level upsampling processing on the first intermediate feature sequence through m-layer upsampling layers to obtain a first temporal boundary probability sequence of the first action; the first temporal boundary probability sequence includes the probability that the image in the video stream to be processed is the start frame of the first action, and the probability that the image in the video stream to be processed is the end frame of the first action, and the upsampling layer includes at least one of the following processing: bilinear interpolation processing, neighboring point interpolation processing, and deconvolution processing; The input data of the i+1th level upsampling process in the m-level upsampling process is the fusion data of the output data of the i-th level upsampling process in the m-level upsampling process and the input data of the j-th level feature extraction process in the n-level feature extraction process; the resolution of the output data of the i-th level upsampling process and the input data of the j-th level feature extraction process are the same; i is a positive integer less than m, and j is a positive integer not greater than n.

2. The method according to claim 1, characterized in that Before performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence, the method further includes: Performing feature extraction processing on the video stream to be processed to obtain a first feature sequence; the first feature sequence corresponds to semantic information of the video stream to be processed and optical flow feature information of a background in the video stream to be processed; the background includes content in the video stream to be processed except the first action; The step of performing n-level feature extraction processing on the video stream to be processed to obtain a first intermediate feature sequence includes: The n-level feature extraction process is performed on the first feature sequence to obtain the first intermediate feature sequence.

3. The method according to claim 2, characterized in that The method further comprises: Performing time sequence reversal processing on the first feature sequence to obtain a second feature sequence whose playing order is opposite to the first feature sequence; Performing p-level feature extraction processing on the second feature sequence to obtain a second intermediate feature sequence; Performing q-level upsampling processing on the second intermediate feature sequence to obtain a second temporal boundary probability sequence of the first action; the second temporal boundary probability sequence includes a probability that an image in the to-be-processed video stream is a start frame of the first action, and a probability that an image in the to-be-processed video stream is an end frame of the first action; The input data of the r+1th level upsampling process in the qth level upsampling process is the fusion data of the output data of the rth level upsampling process in the qth level upsampling process and the input data of the sth level feature extraction process in the pth level feature extraction process; the output data of the rth level upsampling process and the input data of the sth level feature extraction process have the same resolution; r is a positive integer less than q, and s is a positive integer not greater than p; The at least one first temporal action nomination is obtained according to the first temporal boundary probability sequence and the second temporal boundary probability sequence.

4. The method according to claim 3, characterized in that The input data of the j-th level feature extraction process includes the shallow feature information of the video stream to be processed; the s-th level feature extraction process includes the shallow feature information of the first feature sequence.

5. The method according to claim 3 or 4, characterized in that: The obtaining, according to the time sequence boundary probability sequence of the first action and the second time sequence boundary probability sequence, at least one first time sequence action nomination of the first action comprises: fusing the first time series boundary probability sequence and the second time series boundary probability sequence to obtain a third time series boundary probability sequence of the first action; According to the third temporal boundary probability sequence, the at least one first temporal action nomination is obtained.

6. The method according to any one of claims 3 to 5, characterized in that The at least one first time-sequence action nomination includes a second time-sequence action nomination, and the method further includes: Extracting a feature sequence nominated by the second time-series action from the first feature sequence as a third feature sequence; the third feature sequence includes: a first channel feature sequence and a second channel feature sequence; Determine the similarity between the first channel feature sequence and the second channel feature sequence to obtain a first similarity; The confidence of the second time-series action nomination is obtained according to the first similarity and the third feature sequence.

7. The method according to claim 6, characterized in that The determining the similarity between the first channel feature data and the second channel feature data to obtain a first similarity includes: Determine the matrix product of the first channel feature sequence and the second channel feature sequence to obtain a fourth feature sequence as the first similarity.

8. The method according to claim 6 or 7, characterized in that: The at least one first temporal action nomination further includes a third temporal action nomination different from the second temporal action nomination; Before obtaining the confidence of the second time-series action nomination based on the first similarity and the third feature sequence, the method further includes: Determine the similarity between the second time-sequence action nomination and the third time-sequence action nomination to obtain a second similarity; The obtaining, based on the first similarity and the third feature sequence, the confidence of the second time-sequence action nomination includes: The confidence of the second time-series action nomination is obtained according to the first similarity, the second similarity and the third feature sequence.

9. The method according to claim 8, characterized in that The step of extracting the feature sequence nominated by the second time-series action from the first feature sequence as a third feature sequence includes: Extracting the third feature sequence and the feature sequence nominated by the third sequential action from the first feature sequence to obtain a fifth feature sequence; The determining the similarity between the first time-sequence action nomination and the second time-sequence action nomination to obtain a second similarity includes: The matrix product of the fifth characteristic sequence and the sixth characteristic sequence is determined to obtain a seventh characteristic sequence as the second similarity; the sixth characteristic sequence is the transpose of the fifth characteristic sequence.

10. The method according to claim 9, characterized in that The obtaining, according to the first similarity, the second similarity and the third feature sequence, the confidence of the second time-sequence action nomination includes: Performing a first feature extraction process on the first fused feature sequence to obtain a first confidence level of the second time-series action nomination; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence; Performing a second feature extraction process on the second fused feature sequence to obtain a second confidence level of the second time-series action nomination; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence; An average of the first confidence and the second confidence is determined as the confidence of the second time-series action nomination.

11. The method according to claim 10, characterized in that The first feature extraction process is implemented by u-level convolution process, and the second feature extraction process is implemented by v-level convolution process; both u and v are integers greater than 1; Before determining the average of the first confidence level and the second confidence level as the confidence level of the first time-series action nomination, the method further includes: The first feature map and the second feature map are merged to obtain a third feature map; the first feature map is output data of the c-th level convolution process in the u-level convolution process, and the second feature map is output data of the d-th level convolution process in the v-level convolution process; the size of the first feature map is the same as the size of the second feature map; c is a positive integer less than u, and d is a positive integer less than v; Performing a third feature extraction process on the third feature graph to obtain a third confidence level of the second time-series action nomination; The step of determining the average of the first confidence level and the second confidence level as the confidence level of the first sequential action nomination includes: An average of the first confidence, the second confidence, and the third confidence is determined as the confidence of the first sequential action nomination.

12. The method according to claim 9, characterized in that The obtaining, according to the first similarity, the second similarity and the third feature sequence, the confidence of the second time-sequence action nomination includes: Performing feature extraction processing on the first fused feature sequence to obtain a first reference feature sequence; the first fused feature sequence is obtained by fusing the third feature sequence and the fourth feature sequence; Performing feature extraction processing on the second fused feature sequence to obtain a second reference feature sequence; the second fused feature sequence is obtained by fusing the third feature sequence and the seventh feature sequence; fusing the first reference feature sequence and the second reference feature sequence to obtain a third reference feature sequence; Perform feature extraction processing on the third reference feature sequence to obtain the confidence level of the second time-series action nomination.

13. The method according to any one of claims 8 to 12, characterized in that The method further comprises: Sorting the at least one first time-sequence action nomination in descending order of quality score to obtain a time-sequence action nomination order of the first action; the quality score is obtained according to the time-sequence boundary probability sequence of the first time-sequence action nomination and the confidence of the first time-sequence action nomination; The at least one first time-sequence action nomination is displayed according to the time-sequence action nomination order of the first action.

14. The method according to any one of claims 1 to 13, characterized in that The video processing method is applied to a temporal action nomination network, and the training process of the temporal action nomination network includes: Acquire a first training video stream and a neural network to be trained; the neural network to be trained includes a first feature extraction module and a first upsampling module; Using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence; Using the first upsampling module to perform b-level upsampling processing on the third intermediate feature sequence to obtain a fourth temporal boundary probability sequence of the first action; the fourth temporal boundary probability sequence includes a probability that the image in the first training video stream is a start frame of the first action, and a probability that the image in the first training video stream is an end frame of the first action; The input data of the k+1th level upsampling process in the b-level feature extraction process is the fusion data of the output data of the kth level upsampling process in the b-level upsampling process and the input data of the f-th level feature extraction process in the a-level feature extraction process; the output data of the kth level upsampling process and the input data of the f-th level feature extraction process have the same resolution; k is a positive integer less than b, and f is a positive integer not greater than a; Obtaining the loss of the neural network to be trained according to a difference between the fourth temporal boundary probability sequence and a label of the first training video stream; the label of the first training video stream includes a temporal action nomination of the first action in the first training video stream; The parameters of the neural network to be trained are updated based on the loss of the neural network to be trained to obtain the temporal action nomination network.

15. The method according to claim 14, characterized in that The neural network to be trained also includes a basic feature extraction module; Before using the first feature extraction module to perform level A feature extraction processing on the first training video stream to obtain a third intermediate feature sequence, the training process further includes: Using the basic feature extraction module to perform feature extraction processing on the first training video stream, obtaining an eighth feature sequence; the eighth feature sequence corresponds to the semantic information of the first training video stream and the optical flow feature information of the background in the first training video stream; the background includes the content in the first training video stream except the first action; The step of performing level A feature extraction processing on the first training video stream using the first feature extraction module to obtain a third intermediate feature sequence includes: The first feature extraction module is used to perform a-level feature extraction processing on the seven feature sequences to obtain the third intermediate feature sequence.

16. The method according to claim 15, characterized in that The neural network to be trained also includes a second feature extraction module and a second upsampling module; Before obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream, the training process further includes: Performing time sequence reversal processing on the eighth characteristic sequence to obtain a ninth characteristic sequence whose playing order is opposite to that of the eighth characteristic sequence; Using the second feature extraction module to perform x-level feature extraction processing on the ninth feature sequence to obtain a fourth intermediate feature sequence; Using the second upsampling module to perform y-level upsampling processing on the second intermediate feature sequence, to obtain a fifth temporal boundary probability sequence of the first action; the fifth temporal boundary probability sequence includes a probability that an image in the first training video stream is a start frame of the first action, and a probability that an image in the first training video stream is an end frame of the first action; the playback order of the first training video stream is opposite to the playback order of the first training video stream; The input data of the w+1th level upsampling process in the x-level upsampling process is the fusion data of the output data of the wth level upsampling process in the x-level upsampling process and the input data of the eth level feature extraction process in the y-level feature extraction process; the output data of the w-level upsampling process and the input data of the e-level feature extraction process have the same resolution; e is a positive integer less than y, and w is a positive integer not greater than x; The neural network to be trained fuses the fourth time series boundary probability sequence and the fifth time series boundary probability sequence to obtain a sixth time series boundary probability sequence of the first action; The step of obtaining the loss of the neural network to be trained based on the difference between the fourth temporal boundary probability sequence and the label of the first training video stream includes: The loss of the neural network to be trained is obtained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream.

17. The method according to claim 16, characterized in that The neural network to be trained also includes a nomination relationship module; Before obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream, the training process further includes: The neural network to be trained extracts a feature sequence of a fourth temporal action nomination from the eighth feature sequence as the tenth feature sequence; the fourth temporal action nomination is a temporal action nomination of the first action obtained according to the sixth temporal boundary probability sequence; the tenth feature sequence includes: a third channel feature sequence and a fourth channel feature sequence; Determine the similarity between the third channel feature sequence and the fourth channel feature sequence using the nominated relationship module to obtain a third similarity; The nomination relationship module obtains the confidence of the fourth time-series action nomination according to the third similarity and the tenth feature sequence; The step of obtaining the loss of the neural network to be trained based on the difference between the sixth temporal boundary probability sequence and the label of the first training video stream includes: Obtaining a first loss according to a difference between the sixth temporal boundary probability sequence and the label of the first training video stream; Obtaining a second loss according to a difference between a confidence of the fourth temporal action nomination and a label of the first training video stream; The sum of the first loss and the second loss is used as the loss of the neural network to be trained.

18. The method according to any one of claims 14 to 17, characterized in that The step of obtaining a first training video stream includes: Obtain a first positive sample video stream set and a first negative sample video stream set; the video streams in the first positive sample video stream set all contain the temporal action nomination of the first action; the video streams in the first negative sample video stream set all contain the temporal action nomination of the first action; The first positive sample video stream set includes a first scale video stream set and a second scale video stream set; the scale of the video stream in the first scale video stream set is in a first scale interval, the scale of the video stream in the second scale video stream set is in a second scale interval, and the first scale interval has no intersection with the second scale interval; wherein the scale of the positive sample video stream is a ratio of a reference resolution to a resolution of the positive sample video stream; the positive sample video stream is a video stream in the first positive sample video stream set; the reference resolution is a resolution nominated by a sequential action of the first action; When the number of video streams included in the first scaled video stream set is less than the number of video streams included in the second video stream set, sampling the first scaled video stream set at a first sampling rate to obtain a first training video stream subset, and sampling the first scaled video stream set at a second sampling rate to obtain a second training video stream subset; the first sampling rate is greater than the second sampling rate; A video stream is selected from the first training video stream subset and / or the second training video stream subset and / or the first negative sample video stream set as the first training video stream.

19. The method according to claim 18, characterized in that The obtaining of the first positive sample video stream set and the first negative sample video stream set includes: Acquire a training video stream set; the training video stream set includes a second positive sample video stream set and a second negative sample video stream set; the video streams in the second positive sample video stream set all include the temporal action nomination of the first action; the video streams in the second negative sample all include the temporal action nomination of the first action; In a case where the number of video streams included in the second positive sample video stream set is greater than the number of video streams included in the second negative sample video stream set, sampling from the second positive sample video stream set at a third sampling rate to obtain the second positive sample video stream set, and sampling from the second negative sample video stream set at a fourth sampling rate to obtain the second negative sample video stream set; the third sampling rate is less than the fourth sampling rate; When the number of video streams included in the second positive sample video stream set is less than the number of video streams included in the second negative sample video stream set, the second positive sample video stream set is sampled from the second positive sample video stream set at a fifth sampling rate, and the second negative sample video stream set is sampled from the second negative sample video stream set at a sixth sampling rate; the fifth sampling rate is greater than the sixth sampling rate.

20. A video processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a video stream to be processed; A feature extraction processing unit, configured to perform n-level feature extraction processing on the video stream to be processed through n layers of feature extraction layers to obtain a first intermediate feature sequence, wherein the n layers of feature extraction layers are sequentially connected in series, and the feature extraction layers include at least one of the following processing: convolution processing and pooling processing; An upsampling processing unit, configured to perform m-level upsampling processing on the first intermediate feature sequence through m-layer upsampling layers to obtain a first temporal boundary probability sequence of a first action; the first temporal boundary probability sequence includes a probability that the image in the video stream to be processed is a start frame of the first action, and a probability that the image in the video stream to be processed is an end frame of the first action, and the upsampling layer includes at least one of the following processing: bilinear interpolation processing, neighboring point interpolation processing, and deconvolution processing; The input data of the i+1th level upsampling process in the m-level upsampling process is the fusion data of the output data of the i-th level upsampling process in the m-level upsampling process and the input data of the j-th level feature extraction process in the n-level feature extraction process; the resolution of the output data of the i-th level upsampling process and the input data of the j-th level feature extraction process are the same; i is a positive integer less than m, and j is a positive integer not greater than n.

21. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 19.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 19.