Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and a computer readable storage medium. The method comprises: acquiring video data; performing image division on the video data in a color RGB channel and an optical flow channel respectively to generate a plurality of image frames and a plurality of optical flows respectively; performing convolution processing on the plurality of image frames and the optical flow in the color RGB channel and the optical flow channel respectively to generate a first convolution result and a second convolution result of time and space respectively; performing color RGB classification processing and optical flow classification processing respectively to obtain a first action classification result in the image frame and a second action classification result in the optical flow; and performing fusion processing to generate a first fusion action extraction result of the video data. The embodiment of the application greatly reduces the workload of manual annotation required by machine learning training, realizes automatic action category annotation of a long video, increases the samples of model training, and improves the training effect.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] With the development of video technology, more and more industries are using cameras to capture video for anomaly detection and identification of target areas or tasks. In such applications, machine learning techniques have been proposed to automatically identify objects in videos. However, with the advancement of artificial intelligence, people are no longer satisfied with simply identifying target objects in videos; they hope to further identify the actions of objects in videos, thereby achieving automatic judgment of scenes or events occurring in videos.

[0003] Therefore, a technical solution is needed that can identify actions from videos. Summary of the Invention

[0004] This application provides a data processing method and apparatus, an electronic device, and a computer-readable storage medium to address the shortcomings of existing technologies that cannot determine scenes or events occurring in videos.

[0005] To achieve the above objectives, embodiments of this application provide a data processing method, including:

[0006] Acquire video data;

[0007] The video data is divided into image segments in the RGB color channel and the optical flow channel to generate multiple image frames and multiple optical flows respectively;

[0008] The plurality of image frames and the optical flow are convolved in the RGB color channel and the optical flow channel respectively to generate a first convolution result and a second convolution result in time and space respectively, wherein the first convolution result includes a first motion feature extracted from the image frame, and the second convolution result includes a second motion feature extracted from the optical flow;

[0009] The first action feature and the second action feature are subjected to RGB color classification processing and optical flow classification processing, respectively, to obtain the first action classification result and the second action classification result in the optical flow of the image frame;

[0010] The first action feature and the second action feature are fused together to generate the first fused action extraction result of the video data.

[0011] This application also provides a data processing apparatus, including:

[0012] The acquisition module is used to acquire video data;

[0013] The segmentation module is used to segment the video data in the RGB color channel and the optical flow channel respectively to generate multiple image frames and multiple optical flows respectively;

[0014] The first convolution module is used to perform convolution processing on the multiple image frames in the RGB color channels to generate a first convolution result in time and space, wherein the first convolution result includes a first action feature extracted from the image frames.

[0015] The second convolution module is used to perform convolution processing on the optical flow channel to generate a second convolution result in time and space, wherein the second convolution result includes a second motion feature extracted from the optical flow.

[0016] A first classifier is used to perform RGB color classification processing on the first action feature to obtain the first action classification result in the image frame;

[0017] A second classifier is used to perform optical flow classification processing on the second action feature to obtain the second action classification result in the optical flow.

[0018] The first fusion module is used to fuse the first action feature and the second action feature to generate a first fused action extraction result of the video data.

[0019] This application also provides an electronic device, including:

[0020] Memory, used to store programs;

[0021] A processor is configured to run the program stored in the memory, wherein the program executes the data processing method provided in the embodiments of this application.

[0022] This application also provides a computer-readable storage medium storing a computer program executable by a processor, wherein the program, when executed by the processor, implements the data processing method provided in this application.

[0023] The data processing methods, apparatus, electronic devices, and computer-readable storage media provided in this application improve the recognition rate of actions in long videos by fusing features extracted using two different feature extraction methods, RGB color stream (RGB) and optical flow, during the training of the action recognition model. This significantly reduces the amount of manual annotation required for machine learning training, achieves automatic action category annotation for long videos, and increases the training samples by automatically annotating multiple short videos divided from a long video, thus improving the training effect. Therefore, when the model is actually applied to action recognition, the accuracy of action recognition can be improved based on the above training results.

[0024] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0026] Figure 1 A schematic diagram illustrating the principle of the data processing scheme provided in the embodiments of this application;

[0027] Figure 2 A flowchart of one embodiment of the data processing method provided in this application;

[0028] Figure 3 A flowchart of another embodiment of the data processing method provided in this application;

[0029] Figure 4 This is a schematic diagram of the structure of an embodiment of the data processing apparatus provided in this application;

[0030] Figure 5 A schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0031] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0032] Example 1

[0033] The solution provided in this application can be applied to any system with image data processing capabilities, such as an image computing server, etc. Figure 1 This is a schematic diagram illustrating the principle of the data processing scheme provided in the embodiments of this application. Figure 1 The scenario shown is merely one example of the principle of the technical solution of this application.

[0034] With the development of video technology, more and more industries are using cameras to capture video for anomaly detection and identification of target areas or tasks. In such applications, machine learning techniques have been proposed to automatically identify objects in videos. However, with the advancement of artificial intelligence, people are no longer satisfied with simply identifying target objects in videos; they want to further identify the actions of objects in videos, thereby achieving automatic judgment of scenes or events occurring in videos. Therefore, a technical solution capable of recognizing actions from videos is needed.

[0035] Existing technologies have proposed using machine learning to enable computer systems to automatically identify human actions from videos. However, such AI recognition technology requires a large number of labeled video samples as training samples for machine learning. In particular, existing AI recognition technologies can only identify the actions of objects in each frame of a video. Therefore, when preparing machine learning samples for such AI recognition processing, it is necessary to prepare videos with each frame labeled as video samples. Such preparation undoubtedly leads to high costs. Furthermore, since a video usually contains a large number of video frames, labeling each frame also takes a lot of time, making the speed of machine learning very slow, especially unable to keep up with the current project development schedule.

[0036] Furthermore, in existing technologies, when identifying the action of a moving object from each frame of an image, the action is usually identified based on the difference between the identified object and the background image. Specifically, this requires extracting these differences as action features, and then identifying different actions based on these extracted features. However, such difference features do not consider the consistency of actions—that is, identical action features often have similarities. Therefore, existing action feature extraction schemes that only consider the differences between the object and background images are prone to errors, such as mistaking two consecutive actions with little variation for background actions, leading to inaccurate determination of the action's occurrence time.

[0037] In particular, to enhance the accuracy of models in recognizing foreground actions, existing technologies have proposed introducing a background class—samples without action—during training. However, since background samples are not always solid colors and often contain various targets or objects, it is difficult for the model to correctly identify them as background during training without annotation. Therefore, existing technologies typically involve manually annotating these background samples, for example, labeling them as 1 or truth, to enable the model to correctly identify the background class. However, such annotation can lead the model to consider these background samples as positive samples, and the correctly identified action class is also a positive sample 1. Therefore, this approach can easily result in a high probability of the background being identified as action.

[0038] To address this, this application proposes a video action detection scheme. By introducing a novel machine learning framework for action detection in videos, this framework enables post-fusion processing of signals from both the RGB color stream and optical flow channels. For example, as... Figure 1 As shown, Figure 1 This diagram illustrates the principle of training common long video action recognition models.

[0039] exist Figure 1 In this process, a long video segment 1 can be selected as a training sample and input into the model. As mentioned above, in existing technologies, feature extraction is performed on each frame of the input long video 1 separately, that is, features are extracted separately through two channels (RGB color stream and optical flow). The extracted features are then filtered by a filtering module to generate an attention component that considers the influence of the preceding and following frames on the frame. Furthermore, the input long video 1 is divided into multiple short videos 11-15, and each short video is input into the optical flow extraction module and the RGB color stream (RGB) extraction module for feature extraction to generate RGB feature extraction results and optical flow feature extraction results.

[0040] For example, such as Figure 1As shown, after passing through the feature extraction module, action features X, attention signal A, and weights W of attention signal A can be generated respectively. Attention signal A reflects the correlation between the current frame and other frames, and weights W reflect the weight of attention signal A. Therefore, after obtaining the extraction results, existing technologies typically perform weighted fusion of the attention signals from the two channels to obtain a fused attention signal Afuse. This fused attention signal Afuse is then processed by, for example, matrix multiplication, with suppression signals used to suppress the influence of positive samples on background samples. The resulting flow suppression signal and RGB suppression signal are then input together with the flow action feature Xflow and the RGB action feature XRGB into the classification modules of their respective channels to calculate the classification probability, thereby obtaining the action recognition results for each frame. In existing technologies, the action features obtained from the optical flow and RGB color flow channels are trained separately, and only the two action features are ultimately averaged. However, such a simple averaging process can easily lead to actions in some frames being misidentified as background.

[0041] In this regard, this application makes the following... Figure 1 The post-fusion framework shown in the figure, based on the existing dual-channel I3D feature extraction technology, further fuses the flow motion feature Xflow and the RGB motion feature XRGB to generate fused motion feature data, which can then be used to identify the motion in the target video.

[0042] For example, image matching can typically be used to search for a specific object, such as a person or a particular item, in a video clip. In this embodiment, motion features from the flow channel can be further fused into image feature matching to achieve motion search in the video. For instance, a video clip can be specified, and optical flow and RGB features can be extracted from it. Motion fusion data can then be generated based on these extractions. This motion fusion data can be used as feature data for a specified motion to perform motion search in the target video. Specifically, the motion fusion data is fused with the classification results obtained by fusing the target video with the classification results output by a classification model or further with the classification results obtained after feature extraction from the target video. The similarity of the motion fusion data with a similarity greater than a threshold is determined as the motion in the target video that is similar to the specified motion. Thus, the video frame or video clip containing the motion can be output as the search result for the specified motion.

[0043] Therefore, compared with existing technologies, this application sets a fully connected layer after the two feature extraction modules of color image frame feature extraction and optical flow feature extraction. This layer calculates global action characteristics by comparing the outputs of the two sub-networks. The fused result is then multiplied by the color RGB flow feature extraction result and the suppression component extracted from the optical flow feature extraction, and then input into a classification model to obtain the classification result. Finally, the classification result of the suppression component is fused with the classification result of the feature extraction to obtain the action recognition result.

[0044] For example, in logistics service scenarios, such as the sorting process, staff need to pick up and put down packages. During this process, staff must prevent damage to users' packages caused by picking them up and putting them down. Therefore, usually only dedicated supervisors can manually check to confirm whether anyone has damaged the packages. This is not only labor-intensive, but supervisors may also experience visual fatigue from prolonged video viewing, leading to oversight and failure to identify damage. Therefore, according to the embodiments of this application, real-time or historically acquired video can be segmented into multiple short video clips. For each short video clip, image segmentation is performed in the RGB color channel and optical flow channel to generate multiple image frames and multiple optical flows. Based on previously stored image features of staff, the identity of the staff in each image frame can be identified, and the identified staff in the image frames can be labeled accordingly. Furthermore, feature extraction is performed on the multiple image frames and optical flows in the RGB color channel and optical flow channel, for example, convolution processing can be performed to generate temporal and spatial convolution results respectively. In this embodiment of the application, the convolution result can be used as the extraction result of motion features in each image frame and optical flow. Furthermore, the motion feature extraction results in the two channels can be subjected to color classification processing and optical flow classification processing respectively to obtain motion classification results in the image frame and motion classification results in the optical flow. By further fusing the motion features in these classification results, fused motion feature fusion data can be generated. Therefore, these motion feature fusion data can actually characterize the actions of the staff when performing operations. These actions can then be compared with the motion feature data of pre-specified actions that may damage the package to filter out possible damaging actions from these actions. For example, the similarity between the motion feature fusion data and the motion data of a specified motion can be further calculated, and motion data exceeding a threshold can be identified as destructive motions. The corresponding video frames for these destructive motions can be determined accordingly, and the staff members corresponding to these destructive motions in the frames can be identified based on the annotations made to the staff members in the video frames. This allows the identification of the staff members who made such destructive motions, and the final output can be provided to the management personnel by annotating the video frames. Therefore, the management personnel can directly manage the behavior of the staff members based on the video clips of these possible annotated staff members without having to review the collected continuous video for a long time. This not only saves manpower but also improves efficiency and identification accuracy.

[0045] Furthermore, in unmanned store scenarios, users only need to take the goods they want to buy, scan the code, and pay. Therefore, security monitoring is usually required to ensure the safety of goods and money within the store. Typically, dedicated supervisors manually review store videos to confirm whether goods have been taken without payment. However, since unmanned stores are open 24 hours a day, supervisors need to monitor the videos continuously, which is not only labor-intensive but also prone to visual fatigue and oversight due to prolonged video viewing. Therefore, according to the embodiments of this application, the real-time or historically captured video footage can be segmented into multiple short video clips. For each short video clip, image segmentation is performed in both the RGB color channel and the optical flow channel to generate multiple image frames and multiple optical flows. Furthermore, feature extraction is performed on the multiple image frames and optical flows in the RGB color channel and the optical flow channel, for example, by performing convolution processing to generate temporal and spatial convolution results. In this embodiment, the convolution result can be used as the extraction result of motion features in each image frame and optical flow. Furthermore, the motion feature extraction results in the two channels can be subjected to color classification and optical flow classification respectively to obtain motion classification results in the image frame and motion classification results in the optical flow. By further fusing the motion features in these classification results, fused motion feature data is generated. Therefore, this fused motion feature data can actually characterize the actions performed by users in the store. Specifically, since it is an unmanned store, any action occurring in the store is undoubtedly the user's act of viewing and purchasing goods. Therefore, the relationship between consecutive actions can be calculated based on the aforementioned fused motion feature data. For example, when checking out, users typically need to pick up an item and scan the barcode to pay. However, if a user picks up an item but the next action is to put it in a bag instead of scanning the barcode, this action is likely to pose a safety hazard to the store. Therefore, by calculating whether adjacent actions in the motion feature fusion data conform to a preset action sequence, it can be confirmed whether the continuous actions are safety-hazardous. Furthermore, when an action such as picking up an item without scanning the barcode is identified, the corresponding video frame can be labeled, and / or an alarm message can be sent to the management personnel or management server to trigger an alarm or alert sound in the store. Therefore, by accurately identifying actions in long videos and judging dangerous actions based on such identified actions and automatically triggering alarms, the safety supervision of unmanned stores does not require dedicated personnel to view the collected continuous videos for extended periods, saving manpower and improving efficiency and user experience.

[0046] Furthermore, the motion recognition scheme of this application can also be applied to scenarios such as sports motion training. In such sports teaching and training scenarios, users need to learn and imitate prescribed movements to complete teaching or training. Existing technology typically only allows specialized coaches to observe each student's practice movements and judge which movements are incorrect or which details of a movement are not in place based on their own experience. However, this method heavily relies on the experience and working time of the instructors, and in reality, each student hopes to receive one-on-one guidance from a professional coach. Therefore, according to the embodiments of this application, practice videos of each or multiple users can be collected. This collection can be conducted in a teaching venue or by the users themselves and uploaded to the teaching management server. The teaching management can then divide the acquired practice videos into multiple short video segments, and for each short video segment, image segmentation is performed in the RGB color channel and optical flow channel to generate multiple image frames and multiple optical flows. Furthermore, feature extraction is performed on the multiple image frames and optical flows in the RGB color channel and optical flow channel, respectively. For example, convolution processing can be performed to generate temporal and spatial convolution results respectively. In this embodiment, the convolution result can be used as the extraction result of motion features in each image frame and optical flow. Furthermore, the motion feature extraction results in the two extracted channels can be subjected to color classification and optical flow classification processing respectively to obtain motion classification results in the image frame and motion classification results in the optical flow. By further fusing the motion features in these classification results, fused motion feature data is generated. Therefore, this fused motion feature data can actually represent the practice movements of the learner user during imitation practice. Therefore, a video of a coach demonstrating standardized movements can be pre-collected and similarly processed to obtain motion feature fusion data of standardized movements as reference data. Thus, the similarity between the motion fusion data representing the practice movements and the motion features in the reference data can be calculated, and movements with similarity below a threshold can be identified as incorrect movements. Specifically, incorrect movements can be further classified based on the calculated similarity value, for example, into those that are not quite standard and those that are completely incorrect. Furthermore, the parts that differ from the reference movement can be further labeled in the video frame corresponding to the movement, or the calculated similarity can be used as a score for the movement. Therefore, this application accurately identifies the actions of trainees or participants in teaching or competitions, and outputs video frames recording the different actions based on the differences between the identified actions and reference actions, or outputs the calculated similarity value as a score along with the video frame. This not only saves a lot of teaching human resources and judging resources for competitions, but also improves the efficiency and accuracy of action standardization identification.

[0047] This application improves the action recognition rate in long videos by fusing features extracted using two different feature extraction methods, color (RGB) and optical flow, during the training of the action recognition model. This significantly reduces the amount of manual annotation required for machine learning training, enabling automatic action category labeling for long videos. Furthermore, the ability to automatically label multiple short videos segmented from a long video increases the number of training samples and improves training effectiveness. Consequently, when the model is applied to actual action recognition, the accuracy of action recognition can be improved based on the aforementioned training results.

[0048] Based on the model trained above, this application introduces an attention mechanism to identify similar actions when using the model for action recognition. Furthermore, based on the identified similar actions, similar features among the identified action features are merged to further enhance the difference between action features and the background, thereby improving the accuracy of action recognition.

[0049] The above embodiments illustrate the technical principles and exemplary application framework of the embodiments of this application. The specific technical solutions of the embodiments of this application will be further described in detail below through multiple embodiments.

[0050] Example 2

[0051] Figure 2 The flowchart illustrates one embodiment of the data processing method provided in this application. The execution entity of this method can be various terminal or server devices with image data processing capabilities, or it can be a device or chip integrated into these devices. Figure 2 As shown, the data processing method includes the following steps:

[0052] S201, acquire video data.

[0053] In the embodiments of this application, video data can be obtained from various data sources. For example, in a video capture scenario, real-time captured data can be obtained from a video capture device such as a camera, or unrecognized data or sample data that has been recognized and labeled can be obtained from a data server that stores such video capture data.

[0054] S202, the video data is divided into image segments in the RGB color channel and the optical flow channel respectively to generate multiple image frames and multiple optical flows respectively.

[0055] In step S202, the video data acquired in step S201 can be segmented into images. Specifically, in this embodiment, the video data acquired in step S201 can be segmented in the color (RGB) channel, i.e., divided into multiple image frames. That is, traditionally, video data is statically divided into frames to obtain a static image (color, RGB) reflecting each unit of time. Furthermore, in this embodiment, in step S202, optical flow segmentation can also be performed in the optical flow channel. This involves using the temporal changes of pixels in the image sequence and the correlation between adjacent frames to determine the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. Optical flow is then used to represent the changes of foreground objects relative to the background in the video.

[0056] S203, multiple image frames are convolved with the optical flow in the color RGB channel and the optical flow channel respectively to generate a first convolution result and a second convolution result in time and space respectively.

[0057] After generating image frames and optical flow in step S202, convolution processing can be further performed on the respective channels of the image frames and optical flow to generate two-dimensional spatial-temporal convolutions (D×T). For example, convolution processing of the image frame sequence can obtain a first convolution result including a first motion feature extracted from the image frames, and convolution processing of the optical flow can obtain a second convolution result including a second motion feature extracted from the optical flow. In other words, in this embodiment, by dividing the video data into static (RGB image frames) and dynamic (optical flow) segments and performing convolution processing in step S203, it is possible to extract results containing motion features from static images and motion features from dynamic optical flow, reflecting changes in motion, thereby preserving motion information in the video data to the greatest extent.

[0058] S204, perform color classification processing and optical flow classification processing on the first action feature and the second action feature respectively to obtain the first action classification result in the image frame and the second action classification result in the optical flow.

[0059] Therefore, after obtaining the first motion feature reflecting the motion characteristics in the static image frame and the motion feature reflecting the motion changes in optical flow, i.e., the changes of the foreground object or subject relative to the background, the obtained first motion feature and second motion feature can be further classified and calculated in step S204. Specifically, a classifier can be used to calculate the probability, i.e., the score, of the first motion feature and the second motion feature belonging to each pre-classification, and the classification result of the first motion feature and the second motion feature can be determined according to the score ranking.

[0060] S205, perform fusion processing on the first motion feature and the second motion feature to generate the first fused motion extraction result of the video data.

[0061] In step S205, the first motion feature of the RGB channel and the second motion feature of the optical flow channel are further fused to generate fused motion feature data, which can then be used to identify the motion in the target video.

[0062] Specifically, in existing technologies, motion features acquired from the optical flow and color (RGB) channels are trained separately, and then the two motion features are averaged at the end. However, this simple averaging process can easily lead to motion in some frames being misidentified as background.

[0063] Therefore, compared with the prior art, the embodiment of this application sets a fully connected layer after the two feature extraction modules of color RGB flow feature extraction and optical flow feature extraction. In step S205, global action characteristics are calculated by comparing the outputs of the two sub-networks. The fusion result after feature fusion is then multiplied by the suppression components extracted from the color RGB flow feature extraction and optical flow features, respectively, and then input into the classification model to obtain the classification result. Finally, the action recognition result is obtained by fusing the classification result of the suppression components with the classification result of the feature extraction.

[0064] Therefore, the data processing method provided in this application improves the recognition rate of actions in long videos by fusing features extracted by two different feature extraction methods, RGB color flow and optical flow, during the training of the action recognition model. This significantly reduces the workload of manual annotation required for machine learning training, achieves automatic action category annotation for long videos, and increases the training samples by automatically annotating multiple short videos divided from a long video, thus improving the training effect. Consequently, when the model is actually applied to actual action recognition, the accuracy of action recognition can be improved based on the above training effect.

[0065] Example 3

[0066] Figure 3 A flowchart of another embodiment of the data processing method provided in this application is shown. The subject executing this method can be various terminal or server devices with image data processing capabilities, or it can be a device or chip integrated on these devices. Figure 3 As shown, the data processing method includes the following steps:

[0067] S301, acquire video data.

[0068] In the embodiments of this application, video data can be obtained from various data sources. For example, in a video capture scenario, real-time captured data can be obtained from a video capture device such as a camera, or unrecognized data or sample data that has been recognized and labeled can be obtained from a data server that stores such video capture data.

[0069] S302 divides the video data into image segments in the RGB color channel and optical flow channel to generate multiple image frames and multiple optical flows respectively.

[0070] In step S302, the video data acquired in step S301 can be segmented into images. Specifically, in this embodiment, the video data acquired in step S301 can be segmented in the color (RGB) channel, i.e., divided into multiple image frames. That is, traditionally, video data is statically divided into frames to obtain a static image (color, RGB) reflecting each unit of time. In addition, in this embodiment, in step S302, optical flow segmentation can also be performed in the optical flow channel. That is, the correspondence between the previous frame and the current frame is determined by using the changes of pixels in the image sequence in the time domain and the correlation between adjacent frames, thereby calculating the motion information of objects between adjacent frames, and thus using optical flow to represent the changes of foreground objects relative to the background in the video.

[0071] S303 performs convolution processing on multiple image frames and optical flow in the RGB channel and optical flow channel respectively to generate a first convolution result and a second convolution result in time and space respectively.

[0072] After generating image frames and optical flow in step S302, convolution processing can be further performed on the respective channels of the image frames and optical flow to generate two-dimensional spatial-temporal convolutions (D×T). For example, convolution processing of the image frame sequence can obtain a first convolution result including a first motion feature extracted from the image frames, and convolution processing of the optical flow can obtain a second convolution result including a second motion feature extracted from the optical flow. In other words, in this embodiment, by dividing the video data into static (RGB image frames) and dynamic (optical flow) segments and performing convolution processing in step S303, it is possible to extract results containing motion features from static images and motion features from dynamic optical flow, reflecting changes in motion, thereby preserving motion information in the video data to the greatest extent.

[0073] S304, RGB filtering is performed on the first convolution result to generate the first attention data and the corresponding first attention weight data of the image frame.

[0074] S305 performs optical flow filtering on the second convolution result to generate the second attention data of the optical flow and the corresponding second attention weight data.

[0075] S306, The first attention data and the second attention data are fused based on the first attention data, the first attention weight data, the second attention data, and the second attention weight data to generate fused attention data.

[0076] In this embodiment, since actions are usually relatively continuous, such as existing in multiple consecutive frames, extracting action features by considering only a single frame or a single optical flow would result in incomplete action feature extraction, or some features being ignored, or some features playing a misleading role, because the relationship between the current frame or the current optical flow and other frames or other optical flows is severed. Therefore, in steps S304 and S305, the convolution results generated based on RGB image frames and optical flows can be further filtered to obtain attention data of the frame or the optical flow relative to other frames or other optical flows. Of course, in this embodiment, weight data of the attention data can be further obtained to reflect the importance of the attention data.

[0077] Next, in step S306, the attention data and corresponding attention weight data of the two channels obtained in this way can be further fused to obtain more comprehensive fused attention data that reflects the importance or status of the current action in all video data.

[0078] S307, generate first suppression data for the first action feature and second suppression data for the second action feature based on the first convolution result and the second convolution result and the fused attention data, respectively.

[0079] S308, Perform RGB classification processing on the first suppressed data to generate the first suppressed classification result.

[0080] S309, Perform optical flow classification processing on the second suppression data to generate the second suppression classification result.

[0081] S310, Generate a fusion suppression classification result based on the first suppression classification result and the second suppression classification result.

[0082] In this embodiment, when identifying the action of a moving object from each frame of image or optical flow, for example in step S303, the action is usually identified based on the difference between the identified object and the background. In particular, it is necessary to extract the image difference between the identified object and the background as action features, and thus identify different actions based on these extracted features. However, since such difference features do not consider the consistency of actions, i.e., the same action features are usually similar, existing action feature extraction schemes that only consider the difference between the object and the background image are prone to errors, such as mistaking two consecutive actions with little change as background actions, resulting in inaccurate confirmation of the action occurrence time.

[0083] Specifically, to enhance the accuracy of the model in recognizing foreground actions, the data processing method of this application introduces background images, i.e., background samples without action. However, since background samples are not necessarily solid colors, they often contain various targets or objects. Therefore, without annotation of these background samples, it is difficult for the model to correctly identify them as background during training. Thus, existing technologies typically use manual annotation of such background samples, such as labeling them as 1 or truth, to enable the data processing method of this application to correctly identify the background class. However, such annotation actually leads the model to consider these background classes as positive samples, and the correctly identified action class is also a positive sample 1. Therefore, this approach easily leads to a high probability that the background is also identified as action.

[0084] Therefore, in this embodiment, in step S307, suppression data for the first action feature extracted from the RGB image frame and suppression data for the second action feature extracted from the optical flow can be generated from the first convolution result and the second convolution result generated based on the image frame and optical flow, as well as the attention fusion data generated in step S306. In steps S308 and S309, these two suppression data are classified to obtain, for example, the probability (score) of belonging to each predetermined category, and the final classification result can be obtained by sorting the scores. Furthermore, in step S310, the RGB classification result and the optical flow classification result obtained in steps S308 and S309 can be fused to obtain the suppression classification result for both channels, thereby obtaining a suppression result that comprehensively reflects the influence of the background class in both channels.

[0085] S311, perform color classification processing and optical flow classification processing on the first action feature and the second action feature respectively to obtain the first action classification result in the image frame and the second action classification result in the optical flow.

[0086] Therefore, after obtaining the first motion feature reflecting the motion characteristics in the static image frame and the motion feature reflecting the motion changes in the optical flow, i.e. the changes of the foreground object or object relative to the background, the obtained first motion feature and second motion feature can be further classified and calculated in step S311.

[0087] Specifically, a classifier can be used to calculate the probability, i.e., the score, of the first action feature and the second action feature belonging to each pre-classified category, and the classification result of the first action feature and the second action feature can be determined based on the score ranking.

[0088] S312, perform fusion processing on the first action feature and the second action feature to generate the first fused action extraction result of the video data.

[0089] In step S312, the first motion feature of the RGB channel and the second motion feature of the optical flow channel are further fused to generate fused motion feature data, which can then be used to identify the motion in the target video.

[0090] Specifically, in existing technologies, motion features acquired from the optical flow and color (RGB) channels are trained separately, and then the two motion features are averaged at the end. However, this simple averaging process can easily lead to motion in some frames being misidentified as background.

[0091] Therefore, compared with the prior art, the embodiments of this application, by setting a fully connected layer after the two feature extraction modules of color RGB flow feature extraction and optical flow feature extraction, calculate the global action characteristics in step S312 by comparing the outputs of the two self-networks. The fusion result after feature fusion is then multiplied by the suppression components extracted from the color RGB flow feature extraction and optical flow features, respectively, and then input into the classification model to obtain the classification result. Finally, the action recognition result is obtained by fusing the classification result of the suppression components with the classification result of the feature extraction.

[0092] S313, Based on the fusion suppression classification results, the background influence suppression is applied to the first fusion action extraction results to generate the second fusion action extraction results.

[0093] As mentioned above, in practical applications, the introduction of background samples can easily lead to an excessive influence of positive samples and overlapping of action recognition localization results. Therefore, in this embodiment, the fusion of the RGB channel and optical flow channel obtained in step S313 can be used to suppress the classification results, as well as the first and second action features extracted directly based on the RGB image frame and optical flow in step S312, to suppress the negative impact of the background class on action samples during training.

[0094] After the above training process, a fully connected layer is set after the two feature extraction modules of color image frame feature extraction and optical flow feature extraction. This layer calculates global action characteristics by comparing the outputs of the two sub-networks. The fused result of the feature extraction is then multiplied by the color RGB flow feature extraction result and the suppression component extracted from the optical flow feature extraction, and then input into the classification model to obtain the classification result. Subsequently, the classification result of the suppression component is fused with the classification result of the feature extraction to obtain the action recognition result. Therefore, automatic training of the learning model can be achieved without manually labeling a large number of samples. Specifically, in this embodiment, long video data with a playback length greater than a predetermined threshold can be used as training samples during training. The fused action extraction result obtained by training with these samples is used to divide the long video into multiple short video data containing similar actions. This allows for matching action labels based on similar actions identified through the above training process, assigning the matched action labels to the short video data containing similar actions as pseudo-labels for that video data.

[0095] Therefore, it is possible to directly use a large amount of long video data that has not been manually labeled to generate multiple short videos with pseudo-labels that identify action categories, and then use these pseudo-labeled short videos as samples to train the model. This greatly reduces the workload of manual labeling for model training, improves model training efficiency, and is especially suitable for training models that identify action categories.

[0096] After obtaining the trained model as described above, in practical use, the video data to be recognized obtained in step S301, such as that from a video acquisition device from a camera, can be input into the model. The training fusion action extraction result obtained in step S312 or S313 based on the training samples can be used as a value mask to perform matrix multiplication with the extraction result obtained in step S312 or S313 based on the video data to be recognized, thereby obtaining the final action recognition result.

[0097] Therefore, the data processing method provided in this application improves the recognition rate of actions in long videos by fusing features extracted by two different feature extraction methods, RGB color flow and optical flow, during the training of the action recognition model. This significantly reduces the workload of manual annotation required for machine learning training, achieves automatic action category annotation for long videos, and increases the training samples by automatically annotating multiple short videos divided from a long video, thus improving the training effect. Consequently, when the model is actually applied to actual action recognition, the accuracy of action recognition can be improved based on the above training effect.

[0098] Example 4

[0099] Figure 4 This is a schematic diagram of the structure of an embodiment of the data processing apparatus provided in this application, which can be used to perform, such as Figure 2 and Figure 3 The method steps are shown. (As shown) Figure 4 As shown, the data processing device may include: an acquisition module 41, a segmentation module 42, a first convolution module 43, a second convolution module 44, a first classifier 45, a second classifier 46, and a first fusion module 47.

[0100] The acquisition module 41 can be used to acquire video data.

[0101] In this embodiment of the application, the acquisition module 41 can acquire video data from various data sources. For example, in a video acquisition scenario, it can acquire real-time data from a video acquisition device such as a camera, or it can acquire unrecognized data or sample data that has been recognized and labeled from a data server that stores such video acquisition data.

[0102] The segmentation module 42 can be used to segment video data into image segments in the color RGB channel and optical flow channel respectively to generate multiple image frames and multiple optical flows respectively.

[0103] Specifically, the segmentation module 42 can perform image segmentation on the video data acquired by the acquisition module 41. In particular, in this embodiment, the segmentation module 42 can segment the video data acquired by the acquisition module 41 in the color (RGB) channel, that is, divide it into multiple image frames. That is, traditionally, video data is statically divided by frame to obtain a static image (color, RGB) reflecting each unit of time. In addition, in this embodiment, the segmentation module 42 can also perform optical flow segmentation in the optical flow channel, that is, use the changes of pixels in the image sequence in the time domain and the correlation between adjacent frames to determine the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames, and thus using optical flow to represent the changes of foreground objects relative to the background in the video.

[0104] The first convolution module 43 can be used to convolve multiple image frames in the RGB channels to generate a first convolution result in time and space.

[0105] The second convolution module 44 can be used to convolve the optical flow in the optical flow channel to generate a second convolution result in time and space.

[0106] The first convolution module 43 and the second convolution module 44 can further perform convolution processing on the image frames and optical flow obtained by the segmentation module 42 in their respective channels to generate a two-dimensional spatial-temporal convolution (D×T). For example, by performing convolution processing on the image frame sequence, a first convolution result including a first motion feature extracted from the image frames can be obtained, and by performing convolution processing on the optical flow, a second convolution result including a second motion feature extracted from the optical flow can be obtained. In other words, in this embodiment, by dividing the video data into static (RGB image frames) and dynamic (optical flow) segments and performing convolution processing, it is possible to extract results containing motion features in static images and results containing motion features in dynamic optical flow, reflecting changes in motion, thereby preserving motion information in the video data to the greatest extent.

[0107] The first classifier 45 can be used to perform color RGB classification processing on the first action features contained in the first convolution result to obtain the first action classification result in the image frame.

[0108] The second classifier 46 can be used to perform optical flow classification on the second action features in the second convolution result to obtain the second action classification result in the optical flow.

[0109] In addition, the data processing apparatus according to the embodiments of this application may further include: a first filter 48, a second filter 49, and a second fusion module 410.

[0110] The first filter 48 can be used to perform RGB filtering on the first convolution result to generate the first attention data and the corresponding first attention weight data of the image frame.

[0111] The second filter 49 can be used to perform optical flow filtering on the second convolution result to generate second attention data of optical flow and corresponding second attention weight data.

[0112] The second fusion module 410 can be used to fuse the first attention data and the second attention data according to the first attention data, the first attention weight data and the second attention data, the second attention weight data to generate fused attention data.

[0113] In this embodiment, since actions are usually relatively continuous, such as existing in multiple consecutive frames, extracting action features by considering only a single frame or a single optical flow would result in incomplete action feature extraction, or some features being ignored, or some features playing a misleading role, because the relationship between the current frame or the current optical flow and other frames or other optical flows is severed. Therefore, the first filter 48 and the second filter 49 can further filter the convolution results generated based on RGB image frames and optical flows, thereby obtaining attention data of the frame or the optical flow relative to other frames or other optical flows. Of course, in this embodiment, weight data of the attention data can be further obtained to reflect the importance of the attention data.

[0114] Next, the second fusion module 410 can further fuse the attention data and corresponding attention weight data of the two channels obtained in this way, so as to obtain more comprehensive fused attention data that reflects the importance or status of the current action in all video data.

[0115] Furthermore, the data processing apparatus according to the embodiments of this application may further include: a first suppression module 411, a second suppression module 412, and a third fusion module 413.

[0116] The first suppression module 411 can be used to generate first suppression data for the first action feature based on the first convolution result and the fused attention data.

[0117] The second suppression module 412 can be used to generate second suppression data for the second action features based on the second convolution result and the fused attention data.

[0118] Therefore, the first classifier 45 can be further used to perform RGB classification processing on the first suppressed data to generate a first suppression classification result, and the second classifier 46 can be further used to perform optical flow classification processing on the second suppressed data to generate a second suppression classification result.

[0119] Therefore, the third fusion module 413 can be used to generate a fusion suppression classification result based on the first suppression classification result and the second suppression classification result.

[0120] In this embodiment, when the first convolution module 43 identifies the action of a moving object from each frame image or the second convolution module 44 identifies the action of the object from the optical flow, it is usually based on the difference between the identified object and the background. In particular, it is necessary to extract the image difference between the identified object and the background as action features, and thus identify different actions based on the extracted features. However, since such difference features do not consider the consistency of actions, i.e., the same action features are usually similar, for existing action feature extraction schemes that only consider the difference between the object and the background image, it is easy to mistakenly regard two consecutive actions with little change as background actions, resulting in inaccurate confirmation of the action occurrence time.

[0121] Specifically, to enhance the accuracy of the model in recognizing foreground actions, the data processing method of this application introduces background images, i.e., background samples without action. However, since background samples are not necessarily solid colors, they often contain various targets or objects. Therefore, without annotation of these background samples, it is difficult for the model to correctly identify them as background during training. Thus, existing technologies typically use manual annotation of such background samples, such as labeling them as 1 or truth, to enable the data processing method of this application to correctly identify the background class. However, such annotation actually leads the model to consider these background classes as positive samples, and the correctly identified action class is also a positive sample 1. Therefore, this approach easily leads to a high probability that the background is also identified as action.

[0122] Therefore, in this embodiment, the first suppression module 411 and the second suppression module 412 can generate suppression data for the first action feature extracted from the RGB image frame and suppression data for the second action feature extracted from the optical flow based on the first convolution result and the second convolution result generated based on the image frame and optical flow, as well as the attention fusion data generated by the second fusion module 410. The first classifier 45 and the second classifier 46 can then classify the two generated suppression data separately to obtain, for example, the probability (score) of belonging to each predetermined category, thereby obtaining the final classification result based on the score ranking. Furthermore, the third fusion module 413 can fuse the RGB classification result and the optical flow classification result obtained by the first classifier 45 and the second classifier 46 to obtain the suppression classification result for both channels, thereby obtaining a suppression result that comprehensively reflects the influence of the background class in both channels.

[0123] The first fusion module 47 can be used to fuse the first action feature and the second action feature to generate the first fused action extraction result of the video data.

[0124] The first fusion module 47 can further fuse the first motion feature of the RGB channel and the second motion feature of the optical flow channel to generate fused motion feature data, thereby enabling the recognition of motion in the target video based on the fused motion feature data.

[0125] Specifically, in existing technologies, motion features acquired from the optical flow and color (RGB) channels are trained separately, and then the two motion features are averaged at the end. However, this simple averaging process can easily lead to motion in some frames being misidentified as background.

[0126] Therefore, compared with the prior art, the embodiments of this application, by setting a fully connected layer after the two feature extraction modules of color RGB flow feature extraction and optical flow feature extraction, calculates global action characteristics by comparing the outputs of the two sub-networks. The fused result of the feature extraction is then multiplied by the suppression components extracted from the color RGB flow feature extraction and optical flow features, respectively, and then input into the classification model to obtain the classification result. Finally, the classification result of the suppression components is fused with the classification result of the feature extraction to obtain the action recognition result.

[0127] Therefore, the first fusion module 47 can be further used to suppress the background influence of the first fusion action extraction result based on the fusion suppression classification result, so as to generate the second fusion action extraction result.

[0128] As mentioned above, in practical applications, the introduction of background samples can easily lead to an excessive influence of positive samples and overlapping of action recognition localization results. Therefore, in this embodiment, the first fusion module 47 can use the fusion of the RGB channel and optical flow channel obtained by the third fusion module 413 to suppress the classification results, and the first convolution module 43 and the second convolution module 44 can process the first action features and the second action features extracted directly based on the RGB image frame and optical flow to suppress the negative impact of the background class on the action samples during training.

[0129] After the above training process, a fully connected layer is set after the two feature extraction modules of color image frame feature extraction and optical flow feature extraction. This layer calculates global action characteristics by comparing the outputs of the two sub-networks. The fused result of the feature extraction is then multiplied by the color RGB flow feature extraction result and the suppression component extracted from the optical flow feature extraction, and then input into the classification model to obtain the classification result. Subsequently, the classification result of the suppression component is fused with the classification result of the feature extraction to obtain the action recognition result. Therefore, automatic training of the learning model can be achieved without manually labeling a large number of samples. Specifically, in this embodiment, long video data with a playback length greater than a predetermined threshold can be used as training samples during training. The fused action extraction result obtained by training with these samples is used to divide the long video into multiple short video data containing similar actions. This allows for matching action labels based on similar actions identified through the above training process, assigning the matched action labels to the short video data containing similar actions as pseudo-labels for that video data.

[0130] Therefore, it is possible to directly use a large amount of long video data that has not been manually labeled to generate multiple short videos with pseudo-labels that identify action categories, and then use these pseudo-labeled short videos as samples to train the model. This greatly reduces the workload of manual labeling for model training, improves model training efficiency, and is especially suitable for training models that identify action categories.

[0131] After obtaining the trained model as described above, in practical use, the video data to be recognized obtained in step S301, such as the video acquisition device from the camera, can be input into the model. The training fusion action extraction result obtained based on the training samples can be used as a value mask to perform matrix multiplication with the extraction result obtained based on the video data to be recognized, thereby obtaining the final action recognition result.

[0132] Therefore, the data processing apparatus provided in this application improves the recognition rate of action recognition in long videos by fusing features extracted by two different feature extraction methods, RGB color flow and optical flow, during the training of the action recognition model. This significantly reduces the workload of manual annotation required for machine learning training, achieves automatic action category annotation for long videos, and increases the training samples by automatically annotating multiple short videos divided from a long video, thus improving the training effect. Consequently, when the model is actually applied to actual action recognition, the accuracy of action recognition can be improved based on the above training effect.

[0133] Example 5

[0134] The above describes the internal functions and structure of a data processing device, which can be implemented as an electronic device. Figure 5 A schematic diagram illustrating the structure of an embodiment of the electronic device provided in this application. (See attached diagram.) Figure 5 As shown, the electronic device includes a memory 51 and a processor 52.

[0135] Memory 51 is used to store programs. In addition to the programs described above, memory 51 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0136] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0137] Processor 52 is not limited to a central processing unit (CPU), but may also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural network processor (NPU), or an artificial intelligence (AI) chip. Processor 52 is coupled to memory 51 and executes the program stored in memory 51. When the program runs, it performs the data processing methods described in embodiments two and three above.

[0138] Furthermore, such as Figure 5 As shown, the electronic device may also include other components such as a communication component 53, a power supply component 54, an audio component 55, and a display 56. Figure 5 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 5 The components shown.

[0139] Communication component 53 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 53 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 53 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0140] Power supply component 54 provides power to various components of the electronic device. Power supply component 54 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0141] Audio component 55 is configured to output and / or input audio signals. For example, audio component 55 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 51 or transmitted via communication component 53. In some embodiments, audio component 55 also includes a speaker for outputting audio signals.

[0142] Display 56 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0143] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, comprising: Acquire video data; The video data is divided into image segments in the RGB color channel and the optical flow channel to generate multiple image frames and multiple optical flows respectively; The plurality of image frames and the optical flow are convolved in the RGB color channel and the optical flow channel respectively to generate a first convolution result and a second convolution result in time and space respectively, wherein the first convolution result includes a first motion feature extracted from the image frame, and the second convolution result includes a second motion feature extracted from the optical flow; The first action feature and the second action feature are subjected to RGB color classification processing and optical flow classification processing, respectively, to obtain the first action classification result and the second action classification result in the optical flow of the image frame; The first action feature and the second action feature are fused together to generate a first fused action extraction result of the video data; The first convolution result is subjected to RGB color filtering to generate the first attention data and the corresponding first attention weight data of the image frame. The first attention data is used to reflect the correlation between the image frame and other image frames. The second convolution result is subjected to temporal filtering to generate the second attention data of the optical flow and the corresponding second attention weight data. The second attention data is used to reflect the correlation between the optical flow and other optical flows. Based on the first attention data, the first attention weight data, the second attention data, and the second attention weight data, the first attention data and the second attention data are fused together to generate fused attention data. Based on the fused attention data, the first convolution result, and the second convolution result, a fusion suppression classification result is generated; Based on the fusion suppression classification results, the background influence suppression is applied to the first fusion action extraction result to generate the second fusion action extraction result; The step of generating fusion suppression classification results based on the fused attention data, the first convolution result, and the second convolution result includes: First suppression data for the first action feature and second suppression data for the second action feature are generated based on the first convolution result and the second convolution result, respectively, and the fused attention data. The first suppressed data is subjected to color RGB classification processing to generate the first suppression classification result; The second suppression data is subjected to optical flow classification processing to generate a second suppression classification result; Based on the first suppression classification result and the second suppression classification result, a fusion suppression classification result is generated.

2. The data processing method according to claim 1, wherein, The data processing method further includes: The training fusion action extraction result is used as a value mask to perform matrix multiplication with the first fusion action extraction result, wherein the training fusion action extraction result is the fusion action extraction result obtained by training with samples.

3. The data processing method according to claim 1, wherein, The video data is first video data whose playback length is greater than a predetermined threshold, and the data processing method further includes: The first video data is divided into multiple second video data using the training fusion action extraction result, wherein the training fusion action extraction result is the fusion action extraction result obtained by training with samples, and wherein each of the second video data contains similar actions. Based on the similar actions, match action tags to assign the matched action tags to the second video data containing the similar actions as pseudo tags for the second video data.

4. A data processing apparatus, comprising: The acquisition module is used to acquire video data; The segmentation module is used to segment the video data in the RGB color channel and the optical flow channel respectively to generate multiple image frames and multiple optical flows respectively; The first convolution module is used to perform convolution processing on the multiple image frames in the RGB color channels to generate a first convolution result in time and space, wherein the first convolution result includes a first action feature extracted from the image frames. The second convolution module is used to perform convolution processing on the optical flow channel to generate a second convolution result in time and space, wherein the second convolution result includes a second motion feature extracted from the optical flow. A first classifier is used to perform RGB color classification processing on the first action feature to obtain the first action classification result in the image frame; A second classifier is used to perform optical flow classification processing on the second action feature to obtain the second action classification result in the optical flow. The first fusion module is used to fuse the first action feature and the second action feature to generate a first fused action extraction result of the video data; A first filter is used to perform RGB color filtering on the first convolution result to generate first attention data and corresponding first attention weight data of the image frame. The first attention data is used to reflect the correlation between the image frame and other image frames. The second filter is used to perform optical flow filtering on the second convolution result to generate the second attention data and the corresponding second attention weight data of the image frame. The second attention data is used to reflect the correlation between the optical flow and other optical flows. The second fusion module is used to fuse the first attention data and the second attention data according to the first attention data, the first attention weight data, and the second attention data and the second attention weight data to generate fused attention data. The third fusion module is used to generate a fusion suppression classification result based on the fused attention data, the first convolution result, and the second convolution result. The fourth fusion module is used to suppress the background influence of the first fusion action extraction result based on the fusion suppression classification result, so as to generate the second fusion action extraction result; The third fusion module is also used for: First suppression data for the first action feature and second suppression data for the second action feature are generated based on the first convolution result and the second convolution result, respectively, and the fused attention data. The first suppressed data is subjected to color RGB classification processing to generate the first suppression classification result; The second suppression data is subjected to optical flow classification processing to generate a second suppression classification result; Based on the first suppression classification result and the second suppression classification result, a fusion suppression classification result is generated.

5. An electronic device, comprising: Memory, used to store programs; A processor for running the program stored in the memory to perform the data processing method as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon that can be executed by a processor, wherein, When the program is executed by the processor, it implements the data processing method as described in any one of claims 1 to 3.

Images