Method and system for video motion recognition

Abstract

The invention provides a method and system for video action recognition, and relates to the technical field of artificial intelligence, and the method comprises the steps: terminal equipment inputs first format data into an action importance classifier, generates a key action video clip, inputs the key action video clip into an action recognition network, and outputs the key action video clip; a first part of computing network of the action recognition network is arranged on the terminal equipment, a second part of computing network of the action recognition network is arranged on the edge server, the first part of computing network outputs a first network computing result, and the first network computing result is subjected to data simplification through a feature memory bank; and generating calculation unloading data and transmitting the calculation unloading data to the edge server at the calculation unloading point. And the edge server continuously executes action recognition through the second part of computing network based on the computing unloading data, generates an action recognition result and sends the action recognition result to the terminal equipment. Therefore, the real-time performance and the accuracy of video action recognition can be improved by calculating the combination of the unloading point and the edge server in a resource-constrained environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for video action recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, computer vision applications have gradually permeated people's daily lives. Video action recognition is a typical application of computer vision. By analyzing the spatiotemporal features of human movements in videos, video action recognition classifies and detects human behavior, showing broad application prospects in scenarios such as security monitoring and social behavior analysis. However, most current video action recognition algorithms rely on complex model structures and high-performance computing resources, making them difficult to deploy directly on resource-constrained, low-cost terminal devices, severely restricting their practical application.

[0003] To address this issue, existing technologies offer a two-stage edge computing task offloading method suitable for human motion recognition. This method acquires video data through terminal monitoring devices and employs a two-stage human motion recognition task offloading strategy to hierarchically offload the human motion recognition computation. However, this method relies on only a single computation offloading strategy, making it difficult to fully guarantee the real-time performance of motion recognition. Consequently, it becomes difficult to detect the situation on-site in a timely manner and respond quickly to abnormal behavior in personnel motion monitoring tasks.

[0004] Therefore, improving the real-time performance and accuracy of video action recognition in resource-constrained environments has become a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] To address the issues of low real-time performance and accuracy in video action recognition under resource-constrained environments in existing technologies, this invention provides a method and system for video action recognition. The technical solution is as follows:

[0006] On the one hand, a method for video action recognition is provided, the method comprising: a terminal device acquiring a video segment, converting the video segment into a format, and generating first format data that can be input into an action importance classifier; The terminal device inputs the first format data into the action importance classifier, and processes the first format data through the action importance classifier to generate key action video clips; The terminal device inputs the key action video clip into the action recognition network for action recognition. The action recognition network includes a first part of the computing network and a second part of the computing network. The first part of the computing network is set on the terminal device, and the second part of the computing network is set on the edge server. The terminal device inputs the key action video clip into the first part of the computing network to generate a first network calculation result. The terminal device simplifies the first network calculation result through the feature memory, generates calculation unloading data, and transmits the calculation unloading data to the edge server at the calculation unloading point; The edge server receives the computation offload data, continues to perform motion recognition through the second part of the computing network based on the computation offload data, generates motion recognition results, and sends the motion recognition results to the terminal device so that the terminal device can parse and output the motion recognition results as the motion recognition results of the key motion video segment.

[0007] Optionally, the terminal device transmits the computation offload data to the edge server at the computation offload point, including: The terminal device reads the feature maps in the computational unloading data in channel order, and stitches the read feature maps into video frames at the set video frame resolution, wherein the unfilled areas in the video frames are filled with zeros; The terminal device encodes the video frames using a hardware acceleration method based on GStreamer and transmits them to the edge server. The edge server receives the computation offloading data, including: After receiving the video frame, the edge server reconstructs the video frame according to a predefined splicing order to generate the computational offloading data.

[0008] Optionally, the first network calculation result includes multiple first data sets. The terminal device is equipped with a first feature memory, which stores the channel attention distribution corresponding to the transmitted first data and the first identifier corresponding to the transmitted first data, wherein the channel attention distribution corresponding to the transmitted first data and the first identifier corresponding to the transmitted first data are in one-to-one correspondence. The edge server is equipped with a second feature memory, which stores the received first data and the first identifier corresponding to the received first data, wherein the received first data and the first identifier corresponding to the received first data are in one-to-one correspondence. The step of simplifying the first network calculation result through the feature memory to generate computation offloading data includes: For each first data in the current first network calculation result, the channel attention distribution of the current first data is compared with the channel attention distribution stored in the first feature memory. If there is a first channel attention distribution in the first feature memory with a similarity higher than the second set threshold to the channel attention distribution of the current first data, the current first data is not transmitted, and the current first data is replaced with the first identifier corresponding to the first channel attention distribution. After completing the channel attention comparison of all first data in the current first network calculation result, the calculation unloading data is generated based on the first identifier replacement result; The edge server continues to perform action recognition based on the computation offload data through the second part of the computing network, generating action recognition results, including: The received first data corresponding to the first identifier in the computational unloading data is retrieved from the second feature memory. The received first data is used to replace the first identifier in the computational unloading data to generate data to be processed. The data to be processed is then input into the second part of the computing network to continue to perform action recognition and generate action recognition results.

[0009] Optionally, the terminal device is provided with a third feature memory, and the method further includes: At the calculation unloading point, the terminal device generates a second channel attention distribution based on the calculation result of the first network through a channel attention mechanism, and stores the action recognition result corresponding to the calculation result of the first network and the second channel attention distribution into a third feature memory.

[0010] Optionally, before storing the first network calculation result and the second channel attention distribution into the third feature memory, the method further includes: The similarity calculation is performed one by one between the second channel attention distribution and the channel attention distributions already stored in the third feature memory; If a third channel attention distribution exists in the third feature memory that has a similarity to the second channel attention distribution that is higher than a first set threshold, the calculation result of the first network is not simplified, and the action recognition result corresponding to the third channel attention distribution is used as the action recognition result of the key action video segment to be transmitted for parsing and output.

[0011] Optionally, the step of performing similarity calculations between the second channel attention distribution and the channel attention distributions already stored in the third feature memory includes: Sort the attention of each channel in the second channel attention distribution according to its weight from largest to smallest, and determine the top-ranked channels in the sorted distribution. The channel attention is the first channel attention group; Sort the attention of each channel in the second channel attention distribution according to its weight from largest to smallest, and determine the top-ranked channels in the sorted distribution. The channel attention is the first channel attention group; Extract the channel attention distributions stored in the third feature memory one by one, and sort each channel attention distribution extracted from the third feature memory in descending order of weight to determine the top-ranked channel attention distributions. The channel attention is the second channel attention group; A similarity calculation is performed on the first channel attention group and the second channel attention group. If the similarity calculation result is higher than a first set threshold, it is determined that there is a third channel attention distribution in the third feature memory that has a similarity to the second channel attention distribution that is higher than the first set threshold, and the extraction of the channel attention distribution stored in the third feature memory is stopped. If the similarity calculation result is lower than the first set threshold, the next stored channel attention distribution is extracted from the third feature memory.

[0012] Optionally, the action importance classifier is trained in the following manner: Obtain a video action recognition training set, and group the video action recognition training set according to the real action category labels; The video data from each group is fed into the SlowFast network to obtain the predicted probability of the action in each group of video data. For each set of video data, the first video data is labeled as ideally distributed video data, and the prediction probability of the first video data is higher than the prediction probability of other video data. For second video data whose predicted probability is less than that of the first video, calculate the Jensen-Shannon divergence value between the predicted probability of the second video data and the predicted probability of the first video data, and label the second video data with the Jensen-Shannon divergence value. The first video data and the second video data are used as training data, and an action importance classifier is trained based on the training data and the corresponding labels.

[0013] In another aspect, the present invention provides a system for video action recognition, the system being used to implement the method for video action recognition provided in the embodiments of the present invention, the system comprising: A terminal device is configured to acquire video clips, convert the video clips into a format that generates first format data that can be input into an action importance classifier; input the first format data into the action importance classifier, process the first format data through the action importance classifier, and generate key action video clips; input the key action video clips into an action recognition network for action recognition, the action recognition network including a first part of a computing network and a second part of a computing network, the first part of the computing network being set on the terminal device, and the second part of the computing network being set on an edge server; the terminal device inputs the key action video clips into the first part of the computing network to generate a first network calculation result; simplify the first network calculation result through a feature memory to generate computational offload data, and transmit the computational offload data to the edge server at the computational offload point; An edge server is used to receive the computational offload data, continue to perform motion recognition through the second part of the computing network based on the computational offload data, generate motion recognition results, and send the motion recognition results to the terminal device so that the terminal device can parse and output the motion recognition results as the motion recognition results of the key motion video segment.

[0014] On the other hand, a device for video action recognition is also provided, the device for video action recognition including: processor; The memory stores computer-readable instructions, which, when executed by a processor, implement the method provided in the embodiments of the present invention.

[0015] On the other hand, a computer-readable storage medium is also provided, which stores program code that can be called by a processor to execute the method provided in the embodiments of the present invention.

[0016] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention, in its embodiments, acquires video clips via a terminal device, converts the video clips to a format, generating first format data that can be input into an action importance classifier. The first format data is then input into the action importance classifier, which processes the data to generate key action video clips. These key action video clips are then input into an action recognition network for action recognition. The action recognition network includes a first part of a computational network and a second part of a computational network. The first part of the computational network is located on the terminal device, and the second part of the computational network is located on an edge server. The terminal device inputs the key action video clips into the first part of the computational network, generating a first network computation result. This first network computation result is simplified using a feature memory to generate computational offload data, which is then transmitted to the edge server at the computational offload point. The edge server receives the computational offload data and, based on this data, continues to perform action recognition through the second part of the computational network, generating an action recognition result. This action recognition result is then sent to the terminal device, allowing the terminal device to parse and output the action recognition result as the action recognition result of the key action video clip. This allows for improved real-time performance and accuracy of video action recognition in resource-constrained environments by combining computational offload points and edge servers. Furthermore, the feature memory simplifies the data that needs to be transmitted at computational offload points, reducing the data transmission load and improving the real-time performance of video action recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for video action recognition provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the framework of a method for video action recognition provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an action importance classification model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a feature memory mechanism provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a device for video action recognition provided in an embodiment of the present invention; Figure 6This is a schematic diagram of the structure of a device for video motion recognition provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0020] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0021] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0022] In this embodiment of the invention, sometimes a subscript such as W1 may be represented in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0023] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0024] To address the issues of low real-time performance and accuracy in video action recognition under resource-constrained environments in existing technologies, this invention provides a method and system for video action recognition. The technical solution is as follows:

[0025] like Figure 1 As shown, this embodiment of the invention provides a method for video action recognition, the method comprising: S1. The terminal device acquires a video segment, converts the video segment into a format, and generates first format data that can be input into the action importance classifier. S2. The terminal device inputs the first format data into the action importance classifier, processes the first format data through the action importance classifier, and generates key action video clips. S3. The terminal device inputs the key action video clip into the action recognition network to perform action recognition. The action recognition network includes a first part of the computing network and a second part of the computing network. The first part of the computing network is set on the terminal device, and the second part of the computing network is set on the edge server. The terminal device inputs the key action video clip into the first part of the computing network to generate a first network calculation result. S4. The terminal device simplifies the first network calculation result through the feature memory, generates calculation unloading data, and transmits the calculation unloading data to the edge server at the calculation unloading point. S5. The edge server receives the computational offload data, continues to perform motion recognition through the second part of the computing network based on the computational offload data, generates motion recognition results, and sends the motion recognition results to the terminal device so that the terminal device can parse and output the motion recognition results as the motion recognition results of the key motion video segment.

[0026] Optionally, the terminal device transmits the computation offload data to the edge server at the computation offload point, including: The terminal device reads the feature maps from the computational unloading data in channel order, and stitches the read feature maps into video frames at the set video frame resolution. Unfilled areas in the video frames are filled with zeros. The terminal device encodes the video frames using a hardware acceleration method based on GStreamer and transmits them to the edge server.

[0027] The edge server receives the computation offloading data, including: After receiving the video frame, the edge server reconstructs the video frame according to a predefined splicing order to generate the computational offloading data.

[0028] Optionally, the first network calculation result includes multiple first data sets. The terminal device is equipped with a first feature memory, which stores the channel attention distribution corresponding to the transmitted first data and the first identifier corresponding to the transmitted first data, wherein the channel attention distribution corresponding to the transmitted first data and the first identifier corresponding to the transmitted first data are in one-to-one correspondence. The edge server is equipped with a second feature memory, which stores the received first data and the first identifier corresponding to the received first data, wherein the received first data and the first identifier corresponding to the received first data are in one-to-one correspondence. The step of simplifying the first network calculation result through the feature memory to generate computation offloading data includes: For each first data in the current first network calculation result, the channel attention distribution of the current first data is compared with the channel attention distribution stored in the first feature memory. If there is a first channel attention distribution in the first feature memory with a similarity higher than the second set threshold to the channel attention distribution of the current first data, the current first data is not transmitted, and the current first data is replaced with the first identifier corresponding to the first channel attention distribution. After completing the channel attention comparison of all first data in the current first network calculation result, the calculation unloading data is generated based on the first identifier replacement result; The edge server continues to perform action recognition based on the computation offload data through the second part of the computing network, generating action recognition results, including: The received first data corresponding to the first identifier in the computational unloading data is retrieved from the second feature memory. The received first data replaces the first identifier in the computational unloading data, generating data to be processed. The data to be processed is then input into the second part of the computational network to continue performing action recognition and generate an action recognition result. In practical applications, a channel attention distribution can be generated based on the first data using a channel attention mechanism. The channel attention distribution of the first data is compared with the channel attention distributions stored in the first feature memory one by one for similarity calculation. If there is a first channel attention distribution in the first feature memory whose similarity to the current first data is higher than a second set threshold, the first data is not transmitted, and the first identifier corresponding to the first channel attention distribution replaces the first data in the first network calculation result.

[0029] Optionally, the step of performing similarity calculations on the channel attention distribution of the first data and the channel attention distributions already stored in the first feature memory includes: Sort each channel attention in the current first data's channel attention distribution according to its weight from largest to smallest, and determine the top-ranked channels in the sorted distribution. The channel attention is designated as the third channel attention group. The channel attention distributions stored in the first feature memory are extracted one by one. Each channel attention in the extracted distribution is sorted by weight from largest to smallest, and the top-ranked channels are determined. The channel attention is designated as the fourth channel attention group. Similarity calculation is performed on the third channel attention group and the fourth channel attention group. If the similarity calculation result is higher than the second set threshold, it is determined that there is a first channel attention distribution in the first feature memory that has a similarity to the channel attention distribution of the current first data that is higher than the second set threshold, and the extraction of the channel attention distribution stored in the first feature memory is stopped. If the similarity calculation result is lower than the second set threshold, the next stored channel attention distribution is extracted from the first feature memory.

[0030] Optionally, the terminal device is provided with a third feature memory, and the method further includes: At the calculation unloading point, the terminal device generates a second channel attention distribution based on the calculation result of the first network through a channel attention mechanism, and stores the action recognition result corresponding to the calculation result of the first network and the second channel attention distribution into a third feature memory.

[0031] Optionally, before storing the first network calculation result and the second channel attention distribution into the third feature memory, the method further includes: The similarity calculation is performed one by one between the second channel attention distribution and the channel attention distributions stored in the third feature memory. If there is a third channel attention distribution in the third feature memory that has a similarity to the second channel attention distribution that is higher than a first set threshold, the calculation result of the first network is not simplified, and the action recognition result corresponding to the third channel attention distribution is used as the action recognition result of the key action video segment to be transmitted for parsing and output.

[0032] Optionally, the step of performing similarity calculations between the second channel attention distribution and the channel attention distributions already stored in the third feature memory includes: Sort the attention of each channel in the second channel attention distribution according to its weight from largest to smallest, and determine the top-ranked channels in the sorted distribution. The channel attention is the first channel attention group. Extract the channel attention distributions stored in the third feature memory one by one, and sort each channel attention distribution extracted from the third feature memory in descending order of weight to determine the top-ranked channel attention distributions. The channel attention is the second channel attention group. A similarity calculation is performed on the first channel attention group and the second channel attention group. If the similarity calculation result is higher than the first set threshold, it is determined that there is a third channel attention distribution in the third feature memory that has a similarity to the second channel attention distribution that is higher than the first set threshold, and the extraction of the channel attention distribution stored in the third feature memory is stopped. If the similarity calculation result is lower than the first set threshold, the next stored channel attention distribution is extracted from the third feature memory.

[0033] Optionally, the action importance classifier is trained in the following manner: A video action recognition training set is obtained, and the training set is grouped according to the real action category label. The video data in each group is input into a SlowFast network to obtain the predicted probability of the action in each group of video data. For each group of video data, the first video data is labeled as ideally distributed video data, and the predicted probability of the first video data is higher than the predicted probability of other video data. For the second video data whose predicted probability is less than the predicted probability of the first video data, the Jensen-Shannon divergence value between the predicted probability of the second video data and the predicted probability of the first video data is calculated. The second video data is labeled with the Jensen-Shannon divergence value. The first video data and the second video data are used as training data, and the action importance classifier is trained based on the training data and the corresponding labels.

[0034] The execution subject of the video action recognition method provided in this embodiment of the invention is an embedded terminal device.

[0035] In some embodiments, the method for video action recognition provided by the present invention includes: Step (1) Use edge computing devices to collect real-time monitoring video stream data.

[0036] Step (2) The action importance classifier generates key action video clips.

[0037] Step (3) reduces the computational burden on terminal devices by adopting a computation offloading strategy.

[0038] Step (4) The feature memory mechanism optimizes the amount of data transmission during the computation unloading process.

[0039] Step (5) Real-time display of motion prediction results.

[0040] In practical applications, such as Figure 2As shown, step (1): Real-time acquisition of surveillance video stream data using edge computing devices. Real-time video streams can be acquired through surveillance cameras, and data acquisition and processing can be performed. In some implementations, this may include the following steps:

[0041] Step (1-1) Video Acquisition: Open the monitoring camera of the edge computing device, set the resolution and transmission frame rate, and acquire the video stream in real time.

[0042] For acquiring video streams from surveillance cameras, this invention employs the Real-Time Streaming Protocol (RTSP) to transmit the video stream and temporarily stores the received data in a buffer. The video stream in the buffer is organized as a frame sequence, with each frame stored according to a unified encoding format, such as H.264 or H.265 compression encoding, to facilitate subsequent data processing and analysis.

[0043] Step (1-2) Video Stream Data Segmentation: The acquired video stream is divided into equal-length video segments containing Z frames, where Z is a positive integer. These video segments are then transmitted to the terminal device. In some implementations, the terminal device is embedded. Upon receiving the video segments, the terminal device decodes them to obtain the original image frames (RGB format). These image frames are then further converted into tensor form, denoted as... ,in Represents a frame of RGB image, satisfying C represents the number of channels per frame, H represents the height of each frame, and W represents the width of each frame. This converts Z frame video segments into a sequence. ,remember This is the first format of data, to facilitate subsequent feature extraction and action recognition processing.

[0044] Step (2): Analyze the first-format data using an action importance classifier on the terminal device and output key action segments. Its model architecture is as follows: Figure 3 As shown, the action importance classifier uses a lightweight Transformer model to extract temporal information from video segments and combines it with a Multilayer Perceptron (MLP) to classify the importance of the extracted features, thereby achieving the identification and selection of key action segments. The action importance classifier is denoted as... The training and use of the action importance classifier mainly includes the following steps:

[0045] Step (2-1) Data Construction for Training the Action Importance Classifier: Using a publicly available video action recognition training set, uniformly and non-repeatingly sample each video segment in the dataset to generate M sets of video data. The video data consists of frame sequences, with each set comprising S frames, represented as follows: .

[0046]

[0047] in It is the probability of predicting the importance of a frame sequence, and its value range is... . For the classifier, in order to make the classifier To effectively extract keyframe sequence information from videos, it's first necessary to determine the relative importance of M sets of video data in each video segment within the action recognition model. In practical applications, the sampled M sets of video data can be input into the original SlowFast network to obtain the corresponding predicted probability vectors. ,in Let L represent the predicted probability distribution of a set of video data, and L represent the total number of action categories. For each set of video data, the first video data is labeled as ideally distributed video data, where the predicted probability of the first video data is higher than the predicted probabilities of other video data. For the second video data, whose predicted probability is lower than that of the first video data, the Jensen-Shannon divergence value between the predicted probabilities of the second video data and the first video data is calculated, and the second video data is labeled using the Jensen-Shannon divergence value. The first and second video data are used as training data, and an action importance classifier is trained based on the training data and its corresponding labels.

[0048] In practical applications, for each video, its corresponding real-world action category label is known, and the following criteria are selected:

[0049] That is, among the prediction results of M sets of frame sequences, select the subsequence with the highest prediction probability for the true class. .Will The corresponding first video data is labeled as ideally distributed video data.

[0050] For the second video data whose predicted probability is less than that of the first video data, calculate... Compared with the ideal distribution The Jensen-Shannon (JS) divergence between the two data points is used as a supervision signal to label the second video data.

[0051]

[0052] in For the first The intermediate distribution component of the class, when the JS divergence value is closer to 0, indicates a predicted distribution. Approaching the ideal distribution This demonstrates that the stronger the discriminative power of the video frame sequence in the action recognition task, the higher its importance. Here, i represents the action category identifier, and L represents the total number of action categories. Step (2-2) utilizes the action importance classifier trained in step (2-1). For the first formatted data obtained in step (1-2) Perform calculations, when the calculated When a preset threshold condition is met, the corresponding video segment is identified as a key action video segment, and the key action video segment is input into the action recognition network to complete subsequent feature calculation and action recognition processing. The preset threshold condition can be determined by those skilled in the art as needed, and this application is not limited to it.

[0053] Step (3): By introducing a computation offloading strategy, computational tasks are rationally allocated between terminal devices and edge computing nodes, thereby effectively reducing the computational burden on terminal devices and improving overall processing efficiency. This mainly includes the following steps:

[0054] Step (3-1) involves selecting a suitable computational offload point to segment the action recognition network, and then uploading the computational tasks after the offload point to an edge server with high-performance computing capabilities for execution. During data transmission, the first network computation results generated by the first part of the computational network in the terminal device need to be stored in a feature memory. The first network computation results are then simplified using the feature memory to generate computational offload data. This computational offload data is then transmitted to the edge server at the computational offload point to complete subsequent computations. The transmission of computational offload data involves the transmission of a large number of feature maps. To improve transmission efficiency, each feature map is read sequentially by channel in the feature map sequence. The size of each feature map is... Where H1 represents the number of rows and W1 represents the number of columns, and the video frame resolution is set. The feature maps are then stitched together to generate video frames. Indicates the row number. This indicates the column number. Specifically, the feature maps are arranged sequentially from left to right and top to bottom, with each frame accommodating a maximum of [number missing]. Each feature map is used. When there are unfilled areas in the stitched frame, zero-padding is used to maintain the structural integrity of the frame. After the feature maps are stitched together, the video frames are encoded and transmitted using a hardware acceleration method based on GStreamer to realize the transmission of video frames containing computational unloading data. After receiving the video frames containing computational unloading data, the edge server reconstructs the video frames containing computational unloading data according to a predefined stitching order to generate the original feature representation, i.e., the computational unloading data, and uses this feature representation for subsequent computational tasks.

[0055] Step (4): As Figure 4As shown, a feature memory mechanism is used to optimize the data transfer volume generated during computational unloading, thereby reducing the transmission overhead of duplicate features and improving data transfer efficiency and overall system performance. The main steps include:

[0056] Step (4-1) Extract the T-frame feature map from the first network calculation result at the calculation unloading point. Where C2, T, H2, and W2 are positive integers, T is the number of frames, C2 represents the number of channels in each frame's feature map, H2 represents the height of each frame's feature map, and W2 represents the width of each frame's feature map. This is achieved through a channel attention mechanism. Weight distribution of each channel , This represents the dimension of the flattened channel attention weight vector. In the terminal device's third feature memory ( Figure 4 In the terminal device memory bank 1), the most recent N channel attention distributions are cached. And the channel attention distribution of the most recent N times. The corresponding calculation results returned by the edge computing device That is, the action recognition result corresponding to the most recent Nth network calculation results.

[0057]

[0058] Experimental results show that the channel attention distribution has high consistency for different video segments of the same continuous action.

[0059] When the new channel attention distribution is obtained at the calculation unloading point At the same time, the distribution of attention in the historical channels in the third feature memory bank Perform a similarity comparison. If the current channel attention distribution... Attention distribution with a certain historical channel The similarity is higher than a first set threshold, where k is a positive integer less than N. For the second channel attention distribution, For the third channel attention distribution, then use replace It directly returns the corresponding historical results. There is no need to request inference computation from the edge server again. If If the similarity requirement is not met with the attention distribution of all historical channels, then it is written into... It then continues to perform subsequent calculations on the edge server and returns the prediction result. At the same time, write the results to middle.

[0060] To further reduce the computational cost of similarity comparison, only the top-ranked items in the attention distribution, sorted by weight from largest to smallest, are selected. The channel distributions are compared, and the similarity comparison uses formula (5), where i and j They represent and Center front Index of channel attention weights and express and Center front The set of indices for channel attention weights. and Attention distribution of the two channels being compared This represents the dimension of the flattened channel attention weight vector.

[0061]

[0062] After comprehensively considering the performance and efficiency of the model, this invention selects SlowFast as the backbone network of the action recognition network. Since SlowFast uses a dual-channel structure, the feature maps of the two channels that need to be compared in the slow channel are represented as follows: The feature maps of the two channels that need to be compared in the fast channel are represented as follows: To determine the channel attention similarity between two clips, the first two channels of the Slow and Fast channels can be calculated separately. The similarity of channel attention weights, i.e.

[0063]

[0064] In this case, the first set threshold includes a first fast channel threshold and a first slow channel threshold. The two channels being compared are considered to have a successful attention distribution similarity match only if the similarity of the fast channels meets the preset first fast channel threshold and the similarity of the slow channels meets the preset first slow channel threshold.

[0065] Step (4-2) continues to use the similarity matching strategy proposed in (4-1), calculating the channel attention weights for each frame of feature map data in the result of each first network. Similarity comparison is performed. The feature map data is the first set of data. Therefore, a second feature memory is constructed on the edge server. Figure 4 The edge server memory 1), and the first feature memory (built on the terminal device) Figure 4The terminal device memory 2): wherein, the first feature memory of the terminal device is used to cache the channel attention distribution of the most recently sent X frame feature map data (i.e., the first data, where X is a positive integer). The first identifier corresponds to the first data of the X frames sent, wherein the channel attention distribution corresponding to the first data sent is in one-to-one correspondence with the first identifier corresponding to the first data sent. The second feature memory of the edge server is used to cache the received feature map data of X frames (i.e., the first data, where X is a positive integer). And the first identifier corresponding to the first data of the received X frames, where C2 represents the number of channels in each frame feature map, H2 represents the height of each frame feature map, and W2 represents the width of each frame feature map. There is a one-to-one correspondence between the received first data and the corresponding first identifier. The transmitted first data is consistent with the received first data, and the correspondence between the first data, the first identifier, and the channel attention distribution is one-to-one.

[0066] When step (4-1) determines that there is no historical data with similarity matching in the third feature memory for the first network calculation result, before transmitting the data to the edge server for subsequent calculation, the similarity between the channel attention distribution of each first data in the current first network calculation result and the channel attention distribution stored in the first feature memory is calculated. In some implementations, this can be compared according to formulas (5) and (6). and The degree of similarity matching, among which, This represents the channel attention distribution of the new frame feature map (first data) that has not yet been sent. If the similarity is higher than a second preset threshold, it indicates that there is a first channel attention distribution in the first feature memory that is similar to the channel attention distribution of the current first data, and the second feature memory of the edge server has cached data with a similarity to the channel attention distribution of the current first data that is higher than the second preset threshold. The second preset threshold includes a second fast channel threshold and a second slow channel threshold. The channel attention similarity of the current first data is considered to be successfully matched only when the similarity of the fast channels meets the preset second fast channel threshold and the similarity of the slow channels meets the preset second slow channel threshold. At this time, only the first data in the current first network calculation result that fails to match similarity needs to be transmitted. For the first data that successfully matches similarity, the first data is replaced with the first identifier corresponding to the first channel attention distribution, generating computation unloading data, which is then sent to the edge server. After receiving the computation unloading data, the edge server retrieves the data stored in the second feature memory based on the first identifier, and combines it with the newly received computation unloading data for splicing and reconstruction, thereby effectively avoiding the repeated transmission of redundant features.

[0067] Step (5): Real-time display of action prediction results, used to realize the final output of the action recognition process, mainly includes the following steps: Step (5-1): After receiving the action recognition result returned by the edge server, the terminal device parses and processes the action recognition result and generates an action recognition response in real time, thereby realizing online reasoning and prediction of the monitoring video stream and ensuring that the system has immediacy and interactivity.

[0068] Another aspect of the present invention, such as Figure 5 As shown, a system for video action recognition is also provided. This system is used to implement the video action recognition method provided in the embodiments of the present invention. The system includes: Terminal device 501 is used to acquire video clips, convert the video clips into a format to generate first format data that can be input into an action importance classifier; input the first format data into the action importance classifier, process the first format data through the action importance classifier to generate key action video clips; input the key action video clips into an action recognition network for action recognition, the action recognition network including a first part computing network and a second part computing network, the first part computing network being set on the terminal device, and the second part computing network being set on an edge server; the terminal device inputs the key action video clips into the first part computing network to generate a first network calculation result; simplifies the first network calculation result through a feature memory to generate computational offload data, and transmits the computational offload data to the edge server at the computational offload point; Edge server 502 is used to receive the computation offload data, continue to perform action recognition through the second part of the computing network based on the computation offload data, generate action recognition results, and send the action recognition results to the terminal device, so that the terminal device 501 can parse and output the action recognition results as the action recognition results of the key action video segment.

[0069] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention, in its embodiments, acquires video clips via a terminal device, converts the video clips to a format, generating first format data that can be input into an action importance classifier. The first format data is then input into the action importance classifier, which processes the data to generate key action video clips. These key action video clips are then input into an action recognition network for action recognition. The action recognition network includes a first part of a computational network and a second part of a computational network. The first part of the computational network is located on the terminal device, and the second part of the computational network is located on an edge server. The terminal device inputs the key action video clips into the first part of the computational network, generating a first network computation result. This first network computation result is simplified using a feature memory to generate computational offload data, which is then transmitted to the edge server at the computational offload point. The edge server receives the computational offload data and, based on this data, continues to perform action recognition through the second part of the computational network, generating an action recognition result. This action recognition result is then sent to the terminal device, allowing the terminal device to parse and output the action recognition result as the action recognition result of the key action video clip. This allows for improved real-time performance and accuracy of video action recognition in resource-constrained environments by combining computational offload points and edge servers. Furthermore, the feature memory simplifies the data that needs to be transmitted at computational offload points, reducing the data transmission load and improving the real-time performance of video action recognition.

[0070] On the other hand, a device for video action recognition is provided, the device for video action recognition comprising: processor; The memory stores computer-readable instructions, which, when executed by a processor, implement the method provided in the embodiments of the present invention.

[0071] On the other hand, a computer-readable storage medium is provided, which stores program code that can be called by a processor to execute the method provided in the embodiments of the present invention.

[0072] Figure 6 This is a schematic diagram of the structure of a device for video action recognition provided in an embodiment of the present invention, as shown below. Figure 6 As shown, optionally, the device 610 for video motion recognition may include a first processor 2001.

[0073] Optionally, the device 610 for video motion recognition may also include a memory 2002 and a transceiver 2003.

[0074] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0075] The following is combined Figure 6 A detailed description of the various components of the device 610 used for video motion recognition is provided below: The first processor 2001 is the control center of the device 610 for video motion recognition. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0076] Optionally, the first processor 2001 can perform various functions of the device 610 for video motion recognition by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0077] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 are shown in the diagram.

[0078] In a specific implementation, as one example, the device 610 for video action recognition may also include multiple processors, for example... Figure 6 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0079] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0080] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the device 610 for video motion recognition (…). Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0081] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0082] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 6 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0083] Alternatively, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected via the interface circuit of the device 610 for video motion recognition. Figure 6 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0084] It should be noted that, Figure 6 The structure of the device 610 for video motion recognition shown in the figure does not constitute a limitation on the device for video motion recognition. Actual devices for video motion recognition may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0085] Furthermore, the technical effects of the device 610 for video action recognition can be referred to the technical effects of the method for video action recognition in the above method embodiments, and will not be repeated here.

[0086] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0087] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0088] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, motor drive, or data center to another website, computer, motor drive, or data center via infrared, microwave, or other means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a motor drive or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0089] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0090] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0091] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0094] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0097] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a motor driver, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for video action recognition, characterized in that, The method includes: The terminal device acquires a video clip, converts the video clip into a format, and generates first format data that can be input into the action importance classifier; The terminal device inputs the first format data into the action importance classifier, and processes the first format data through the action importance classifier to generate key action video clips; The terminal device inputs the key action video clip into the action recognition network for action recognition. The action recognition network includes a first part of the computing network and a second part of the computing network. The first part of the computing network is set on the terminal device, and the second part of the computing network is set on the edge server. The terminal device inputs the key action video clip into the first part of the computing network to generate a first network calculation result. The terminal device simplifies the first network calculation result through the feature memory, generates calculation unloading data, and transmits the calculation unloading data to the edge server at the calculation unloading point; The edge server receives the computation offload data, continues to perform motion recognition through the second part of the computing network based on the computation offload data, generates motion recognition results, and sends the motion recognition results to the terminal device so that the terminal device can parse and output the motion recognition results as the motion recognition results of the key motion video segment.

2. The method according to claim 1, characterized in that, The terminal device transmits the computation offloading data to the edge server at the computation offloading point, including: The terminal device reads the feature maps in the computational unloading data in channel order, and stitches the read feature maps into video frames at the set video frame resolution, wherein the unfilled areas in the video frames are filled with zeros; The terminal device encodes the video frames using a hardware acceleration method based on GStreamer and transmits them to the edge server. The edge server receives the computation offloading data, including: After receiving the video frame, the edge server reconstructs the video frame according to a predefined splicing order to generate the computational offloading data.

3. The method according to claim 1, characterized in that, The first network computation result includes multiple first data sets. The terminal device is equipped with a first feature memory, which stores the channel attention distribution corresponding to the transmitted first data and the first identifier corresponding to the transmitted first data, wherein the channel attention distribution corresponding to the transmitted first data and the first identifier corresponding to the transmitted first data are in one-to-one correspondence. The edge server is equipped with a second feature memory, which stores the received first data and the first identifier corresponding to the received first data, wherein the received first data and the first identifier corresponding to the received first data are in one-to-one correspondence. The step of simplifying the first network computation result through the feature memory to generate computation offloading data includes: For each first data in the current first network calculation result, the channel attention distribution of the current first data is compared with the channel attention distribution stored in the first feature memory. If there is a first channel attention distribution in the first feature memory with a similarity higher than the second set threshold to the channel attention distribution of the current first data, the current first data is not transmitted, and the current first data is replaced with the first identifier corresponding to the first channel attention distribution. After completing the channel attention comparison of all first data in the current first network calculation result, the calculation unloading data is generated based on the first identifier replacement result; The edge server continues to perform action recognition based on the computation offload data through the second part of the computing network, generating action recognition results, including: The received first data corresponding to the first identifier in the computational unloading data is retrieved from the second feature memory. The received first data is used to replace the first identifier in the computational unloading data to generate data to be processed. The data to be processed is then input into the second part of the computing network to continue to perform action recognition and generate action recognition results.

4. The method according to claim 1, characterized in that, The terminal device is equipped with a third feature memory, and the method further includes: At the calculation unloading point, the terminal device generates a second channel attention distribution based on the calculation result of the first network through a channel attention mechanism, and stores the action recognition result corresponding to the calculation result of the first network and the second channel attention distribution into a third feature memory.

5. The method according to claim 4, characterized in that, Before storing the first network calculation results and the second channel attention distribution into the third feature memory, the method further includes: The similarity calculation is performed one by one between the second channel attention distribution and the channel attention distributions already stored in the third feature memory; If a third channel attention distribution exists in the third feature memory that has a similarity to the second channel attention distribution that is higher than a first set threshold, the calculation result of the first network is not simplified, and the action recognition result corresponding to the third channel attention distribution is used as the action recognition result of the key action video segment to be transmitted for parsing and output.

6. The method according to claim 5, characterized in that, The step of performing similarity calculations between the second channel attention distribution and the channel attention distributions already stored in the third feature memory includes: Sort the attention of each channel in the second channel attention distribution according to its weight from largest to smallest, and determine the top-ranked channels in the sorted distribution. The channel attention is the first channel attention group; Extract the channel attention distributions stored in the third feature memory one by one, and sort each channel attention distribution extracted from the third feature memory in descending order of weight to determine the top-ranked channel attention distributions. The channel attention is the second channel attention group; A similarity calculation is performed on the first channel attention group and the second channel attention group. If the similarity calculation result is higher than a first set threshold, it is determined that there is a third channel attention distribution in the third feature memory that has a similarity to the second channel attention distribution that is higher than the first set threshold, and the extraction of the channel attention distribution stored in the third feature memory is stopped. If the similarity calculation result is lower than the first set threshold, the next stored channel attention distribution is extracted from the third feature memory.

7. The method according to claim 1, characterized in that, The action importance classifier is trained in the following manner: Obtain a video action recognition training set, and group the video action recognition training set according to the real action category labels; The video data from each group is fed into the SlowFast network to obtain the predicted probability of the action in each group of video data. For each set of video data, the first video data is labeled as ideally distributed video data, and the prediction probability of the first video data is higher than the prediction probability of other video data. For second video data whose predicted probability is less than that of the first video, calculate the Jensen-Shannon divergence value between the predicted probability of the second video data and the predicted probability of the first video data, and label the second video data with the Jensen-Shannon divergence value. The first video data and the second video data are used as training data, and an action importance classifier is trained based on the training data and the corresponding labels.

8. A system for video action recognition, said system for video action recognition being used to implement the method for video action recognition as described in any one of claims 1-7, characterized in that, The system includes: A terminal device is configured to acquire video clips, convert the video clips into a format that generates first format data that can be input into an action importance classifier; input the first format data into the action importance classifier, process the first format data through the action importance classifier, and generate key action video clips; input the key action video clips into an action recognition network for action recognition, the action recognition network including a first part of a computing network and a second part of a computing network, the first part of the computing network being set on the terminal device, and the second part of the computing network being set on an edge server; the terminal device inputs the key action video clips into the first part of the computing network to generate a first network calculation result; simplify the first network calculation result through a feature memory to generate computational offload data, and transmit the computational offload data to the edge server at the computational offload point; An edge server is used to receive the computational offload data, continue to perform motion recognition through the second part of the computing network based on the computational offload data, generate motion recognition results, and send the motion recognition results to the terminal device so that the terminal device can parse and output the motion recognition results as the motion recognition results of the key motion video segment.

9. A device for video action recognition, characterized in that, The device for video motion recognition includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images