Behavior recognition method and device based on multi-modal data, edge computing equipment and medium

Abstract

The invention discloses a behavior recognition method and device based on multi-modal data, edge computing equipment and a medium, and belongs to the technical field of behavior recognition. The method comprises the following steps: collecting video data containing actions of a target object, and synchronously collecting an electroencephalogram signal of the target object; extracting a plurality of first image frames representing changes of a space scene where the target object is located from the video data, and extracting a plurality of second image frames representing motion changes of the target object from the video data; and recognizing the behavior of the target object according to the plurality of first image frames, the plurality of second image frames and the electroencephalogram signals. According to the embodiment of the invention, behavior recognition can be carried out by fusing the multi-modal data of the video data and the electroencephalogram signals, so that the accuracy of behavior recognition is improved.

Description

Technical Field

[0001] The present application belongs to the field of behavior recognition technology, and in particular, relates to a behavior recognition method, device, edge computing device and medium based on multimodal data. Background Art

[0002] Behavior recognition is a technology that involves detecting and analyzing human behavior. It is widely used in many fields such as human-computer interaction, medical health monitoring, smart home and security monitoring.

[0003] In related technologies, behavior recognition is usually performed by analyzing video images to extract features such as the movement trajectory, posture, and expression of a person or object. However, since the information provided by visual images is relatively single, it is usually difficult to accurately reflect the background and intention of a behavior, resulting in inaccurate recognition. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a behavior recognition method, device, edge computing device and medium based on multimodal data to reduce the mapping error between different modal images and improve the accuracy of eye movement data mapping.

[0005] In a first aspect, the present application provides a behavior recognition method based on multimodal data, comprising:

[0006] Collecting video data containing the target object's movements, and synchronously collecting the target object's EEG signals;

[0007] Extracting a plurality of first image frames representing changes in a spatial scene where the target object is located from the video data, and extracting a plurality of second image frames representing changes in actions of the target object from the video data;

[0008] The behavior of the target object is identified according to the plurality of the first image frames, the plurality of the second image frames and the EEG signal.

[0009] According to the behavior recognition method based on multimodal data of the present application, video data containing the action of the target object is collected, and the EEG signal of the target object is collected synchronously; multiple first image frames representing the change of the spatial scene where the target object is located are extracted from the video data, and multiple second image frames representing the change of the action of the target object are extracted from the video data; the behavior of the target object is identified according to the multiple first image frames, the multiple second image frames and the EEG signal. The embodiment of the present application synchronously collects the action video data and EEG signal of the target object, and extracts key image frames representing the change of the spatial scene and the change of the action from the video data, and combines the EEG signal to deeply understand the neural activity behind the behavior, and performs behavior recognition by fusing the multimodal data of video data and EEG signals, comprehensively utilizing visual behavior and physiological behavior, reducing the problem of insufficient accuracy of single modality data in behavior recognition, and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0010] According to one embodiment of the present application, collecting video data containing the target object's action includes:

[0011] Set the acquisition frame rate of the RGB camera to the preset frame rate;

[0012] The target object is photographed by the RGB camera using the preset frame rate, and the photographed image frames are adjusted to a preset size to obtain video data containing the target object's movements.

[0013] In this embodiment, the acquisition frame rate of the RGB camera is set to a preset frame rate to improve the smoothness and continuity of the video data, and the captured image frame is adjusted to a preset size, which can not only adapt to the input requirements of the subsequent processing system, but also optimize the storage and processing efficiency while maintaining the integrity of the video content.

[0014] In this embodiment, extracting a plurality of first image frames representing changes in the spatial scene where the target object is located from the video data, and extracting a plurality of second image frames representing changes in the action of the target object from the video data, comprises:

[0015] Determining a capture frame rate of the video data;

[0016] Determine the extraction stride according to the acquisition frame rate;

[0017] The image frames in the video data are sampled according to the sampling step to obtain a plurality of first image frames and / or a plurality of second image frames.

[0018] In this embodiment, by determining the extraction step according to the acquisition frame rate, it is possible to analyze the speed of change of the spatial scene where the target object is located and the continuous change of the target object's action in time according to the acquisition frame rate, and then the extraction step can be determined, so that the sampled first image frame and the second image frame can accurately represent the changes in the spatial scene where the target object is located and the continuous change of the target object's action in time, providing accurate spatial information and time information for behavior recognition.

[0019] According to an embodiment of the present application, extracting a plurality of first image frames representing changes in a spatial scene where the target object is located from the video data, and extracting a plurality of second image frames representing changes in an action of the target object from the video data, comprises:

[0020] Determining a capture frame rate of the video data;

[0021] Determining a first decimation step and a second decimation step according to the acquisition frame rate; the second decimation step is smaller than the first decimation step;

[0022] The image frames in the video data are sampled according to the first sampling step to obtain a plurality of first image frames; and the image frames in the video data are sampled according to the second sampling step to obtain a plurality of second image frames.

[0023] In this embodiment, since the continuous change of action in time is faster than the change of spatial scene, the second extraction step is smaller than the first extraction step, and the image frames can be sampled more densely, so that the number of second image frames per unit time is greater than the number of first image frames, and more action details of the target object can be captured, so that the second image frame can accurately represent the action changes of the target object.

[0024] According to an embodiment of the present application, the identifying the behavior of the target object according to the plurality of the first image frames, the plurality of the second image frames and the EEG signal includes:

[0025] Performing frequency domain analysis on the EEG signal, and obtaining period information of the EEG signal based on the frequency of the EEG signal in the frequency domain;

[0026] Converting the EEG signal into a two-dimensional tensor based on multiple cycles according to the cycle information;

[0027] Performing feature extraction on the two-dimensional tensor to obtain an EEG feature map;

[0028] The behavior of the target object is identified according to the plurality of the first image frames, the plurality of the second image frames and the EEG characteristic map.

[0029] In this embodiment, by performing frequency domain analysis on the EEG signal, the amplitude distribution of the EEG signal at different frequencies can be captured, so that the periodic information of the EEG signal can be determined, and the EEG signal is converted into a two-dimensional tensor using the periodic information. Feature extraction is performed on the two-dimensional tensor to capture the changes of the EEG signal at different times to obtain an EEG feature map. The first image frame and the second image frame are then combined to perform behavior recognition using multimodal data that combines video data and EEG signals. Visual behavior and physiological behavior are comprehensively utilized, which reduces the problem of insufficient accuracy of single modality data in behavior recognition and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0030] According to an embodiment of the present application, the identifying the behavior of the target object according to the plurality of the first image frames, the plurality of the second image frames and the EEG characteristic graph includes:

[0031] Acquire a plurality of the first image frames and a plurality of the second image frames extracted from image frames in the same time period in the video data, and acquire the electroencephalogram corresponding to the time of the plurality of the first image frames and the plurality of the second image frames;

[0032] Extracting features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map respectively, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map;

[0033] The behavior of the target object is identified based on the fused features.

[0034] In this embodiment, by extracting the first image frame and the second image frame from image frames within the same time period and obtaining the EEG feature map corresponding to the image frame time, the synchronization and correlation of the input data are improved, and features are extracted from the image frames and the EEG signals respectively, and these features are fused. This feature fusion not only includes the spatial scene and motion change information at the visual level, but also includes the EEG activity information at the neural level, thereby providing a comprehensive perspective to understand the behavior of the target object, so that the behavior of the target object can be identified more accurately and can adapt to complex and changing environments.

[0035] According to an embodiment of the present application, respectively extracting features of the plurality of first image frames, the plurality of second image frames, and the EEG feature map, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the EEG feature map, includes:

[0036] Extracting spatial features representing changes in the spatial scene where the target object is located from the plurality of first image frames;

[0037] Extracting a temporal feature representing a change in the motion of the target object from a plurality of the second image frames;

[0038] Extracting the EEG features of the target object from the EEG feature map;

[0039] The spatial features, temporal features and EEG features are fused.

[0040] In this embodiment, the first image frame, the second image frame and the EEG feature map correspond to the spatial scene changes, motion changes and EEG signals in the video data, respectively. Based on these data, the spatial features representing the spatial scene changes of the target object, the temporal features representing the motion changes of the target object and the EEG features of the target object can be extracted respectively. These features are fused, and multimodal data is comprehensively utilized. Features of different dimensions are fused to identify the behavior of the target object, which can better cope with complex environments and changeable behaviors, and further improve the accuracy of behavior recognition.

[0041] In a second aspect, the present application provides a behavior recognition device based on multimodal data, comprising:

[0042] An acquisition module, used to acquire video data containing the target object's movements, and synchronously acquire the target object's EEG signals;

[0043] An extraction module, used to extract a plurality of first image frames representing changes in the spatial scene where the target object is located from the video data, and to extract a plurality of second image frames representing changes in the action of the target object from the video data;

[0044] A recognition module is used to recognize the behavior of the target object according to the plurality of the first image frames, the plurality of the second image frames and the EEG signal.

[0045] According to the behavior recognition device based on multimodal data of the present application, by collecting video data containing the action of the target object, and synchronously collecting the EEG signal of the target object; extracting multiple first image frames representing the change of the spatial scene where the target object is located from the video data, and extracting multiple second image frames representing the change of the action of the target object from the video data; identifying the behavior of the target object according to the multiple first image frames, multiple second image frames and the EEG signal. The embodiment of the present application synchronously collects the action video data and EEG signal of the target object, and extracts key image frames representing the change of the spatial scene and the change of the action from the video data, and combines the EEG signal to deeply understand the neural activity behind the behavior, and performs behavior recognition by fusing the multimodal data of video data and EEG signal, comprehensively utilizing visual behavior and physiological behavior, reducing the problem of insufficient accuracy of single modality data in behavior recognition, and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0046] In a third aspect, the present application provides an edge computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the behavior recognition method based on multimodal data as described in the first aspect above is implemented.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the behavior recognition method based on multimodal data as described in the first aspect above is implemented.

[0048] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the behavior recognition method based on multimodal data as described in the first aspect above.

[0049] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the behavior recognition method based on multimodal data as described in the first aspect above.

[0050] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0051] According to the behavior recognition method based on multimodal data of the present application, video data containing the action of the target object is collected, and the EEG signal of the target object is collected synchronously; multiple first image frames representing the change of the spatial scene where the target object is located are extracted from the video data, and multiple second image frames representing the change of the action of the target object are extracted from the video data; the behavior of the target object is identified according to the multiple first image frames, the multiple second image frames and the EEG signal. The embodiment of the present application synchronously collects the action video data and EEG signal of the target object, and extracts key image frames representing the change of the spatial scene and the change of the action from the video data, and combines the EEG signal to deeply understand the neural activity behind the behavior, and performs behavior recognition by fusing the multimodal data of video data and EEG signals, comprehensively utilizing visual behavior and physiological behavior, reducing the problem of insufficient accuracy of single modality data in behavior recognition, and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0052] Furthermore, in some embodiments, by setting the acquisition frame rate of the RGB camera to a preset frame rate, the smoothness and continuity of the video data are improved, and the captured image frames are adjusted to a preset size, which can not only adapt to the input requirements of the subsequent processing system, but also optimize the storage and processing efficiency while maintaining the integrity of the video content.

[0053] Furthermore, in some embodiments, by determining the sampling step according to the acquisition frame rate, it is possible to analyze the speed of change of the spatial scene where the target object is located and the continuous change of the target object's action in time according to the acquisition frame rate, and then the sampling step can be determined, so that the sampled first image frame and the second image frame can accurately represent the changes in the spatial scene where the target object is located and the continuous change of the target object's action in time, providing accurate spatial information and temporal information for behavior recognition.

[0054] Furthermore, in some embodiments, since the continuous change of action in time is faster than the change of spatial scene, the second extraction step is smaller than the first extraction step, and the image frames can be sampled more densely, so that the number of second image frames per unit time exceeds the number of first image frames, and more action details of the target object can be captured, so that the second image frame can accurately represent the action changes of the target object.

[0055] Furthermore, in some embodiments, by performing frequency domain analysis on the EEG signal, the amplitude distribution of the EEG signal at different frequencies can be captured, so that the periodic information of the EEG signal can be determined, and the EEG signal is converted into a two-dimensional tensor using the periodic information. Feature extraction is performed on the two-dimensional tensor to capture the changes of the EEG signal at different times to obtain an EEG feature map. The first image frame and the second image frame are then combined to perform behavior recognition using multimodal data that combines video data and EEG signals. This comprehensively utilizes visual behavior and physiological behavior, reduces the problem of insufficient accuracy of single modality data in behavior recognition, and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0056] Furthermore, in some embodiments, by extracting the first image frame and the second image frame from image frames within the same time period and obtaining the EEG feature map corresponding to the image frame time, the synchronization and correlation of the input data are improved, and features are extracted from the image frames and the EEG signals respectively, and these features are fused. This feature fusion not only includes the spatial scene and motion change information at the visual level, but also includes the EEG activity information at the neural level, thereby providing a comprehensive perspective to understand the behavior of the target object, so that the behavior of the target object can be identified more accurately and can adapt to complex and changing environments.

[0057] Furthermore, in some embodiments, the first image frame, the second image frame and the EEG feature map correspond to the spatial scene changes, motion changes and EEG signals in the video data, respectively, and the spatial features representing the spatial scene changes of the target object, the temporal features representing the motion changes of the target object and the EEG features of the target object can be extracted based on these data, and these features are fused, making comprehensive use of multimodal data, and the features of different dimensions are fused to identify the behavior of the target object, which can better cope with complex environments and changeable behaviors, and further improve the accuracy of behavior recognition.

[0058] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1It is a flowchart of a behavior recognition method based on multimodal data provided in an embodiment of the present application;

[0061] Figure 2 It is a structural diagram of a multimodal behavior recognition model provided in an embodiment of the present application;

[0062] Figure 3 is a schematic diagram of the structure of a behavior recognition device based on multimodal data provided in an embodiment of the present application;

[0063] Figure 4 It is a structural diagram of the edge computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0065] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0066] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0067] In the following, in combination with the accompanying drawings, the behavior recognition method, device, edge computing device and medium based on multimodal data provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0068] Among them, the behavior recognition method based on multimodal data can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0069] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0070] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0071] The behavior recognition method based on multimodal data provided in the embodiment of the present application may be executed by an edge computing device or a functional module or functional entity in the edge computing device that can implement the behavior recognition method based on multimodal data. The edge computing device mentioned in the embodiment of the present application may be a portable device (such as a mobile phone, a tablet computer), a wearable device (such as a smart watch, a smart bracelet), an eye tracker, an AR (Augmented Reality) / VR (Virtual Reality) / XR (Extended Reality) / MR (Mixed Reality) device, an edge server, an edge gateway, a vehicle-mounted device, a cloud computing device, etc. The behavior recognition method based on multimodal data provided in the embodiment of the present application is described below using the edge computing device as an example of the execution subject.

[0072] like Figure 1 As shown, the behavior recognition method based on multimodal data includes: step 110, step 120 and step 130.

[0073] Step 110: Collect video data containing the target object's movements, and synchronously collect the target object's EEG signals.

[0074] In the embodiments of the present application, the target object may be different objects according to different application scenarios. For example, in a simulated driving scenario, the target object may be a driver, in a medical and health scenario, the target object may be a patient, and in a test scenario, the target object may be a person being tested.

[0075] In some embodiments, an image acquisition device, such as a high-definition camera, a mobile device, etc., may be used to photograph the target object, thereby obtaining video data containing the target object's movements.

[0076] Electroencephalography (EEG) is an electrical signal generated by the activity of brain neurons. It reflects the superposition of postsynaptic potentials generated by a large number of neurons and is a map of brain activity. Neurons in the brain are connected to each other through synapses to form a complex neural network. When neurons are activated, bioelectric phenomena are generated. These electrical signals can be captured by placing electrodes on the scalp or implanted directly into the brain.

[0077] In some embodiments, an EEG acquisition device may be used to acquire EEG signals of the target object. Specifically, electrodes may be placed on the scalp of the target object to acquire EEG signals of the target object. The EEG acquisition device may be a head-mounted device, and after the target object correctly wears the head-mounted device, the EEG signals of the target object are acquired through the electrodes in the head-mounted device.

[0078] In the embodiment of the present application, in order to improve the consistency between data of different modalities, the video data containing the target object's actions and the target object's EEG signals can be synchronously collected to facilitate the subsequent processing of data of different modalities. For example, in one scene, the target object is in a room, and the EEG signals of the target object are collected through a head-mounted device worn by the target object. A high-definition camera can also be fixed in the room, and the high-definition optical camera can synchronously shoot the target object.

[0079] Step 120: extract a plurality of first image frames representing changes in the spatial scene where the target object is located from the video data, and extract a plurality of second image frames representing changes in the action of the target object from the video data.

[0080] In an embodiment of the present application, in order to accurately identify the behavior of the target object, when processing the video data, analysis can be performed from the dimension of changes in the spatial scene where the target object is located, and from the dimension of continuous changes in time of the target object.

[0081] Specifically, since the change of the spatial scene is relatively slow compared to the change of the target object's action, a certain number of image frames can be extracted from the video data, and the characteristics of the spatial scene change can be better captured according to the change of the content in the extracted image frames. For example, if the frame rate of the video data is 32 frames, 2 frames can be selected as the first image frame in a unit time.

[0082] Since the movement of the target object changes faster than the change of the spatial scene, more image frames are needed to capture the subtle changes of the movement. More image frames can be extracted from the video data, and the changes of the target object in time can be better captured according to the changes in the content of the extracted image frames. For example, if the frame rate of the video data is 32 frames, 4 frames can be selected as the first image frame in a unit time.

[0083] It should be noted that the number of first image frames and second image frames selected within a unit time may also be other values, which is not limited in the embodiments of the present application.

[0084] Step 130: Identify the behavior of the target object according to the multiple first image frames, the multiple second image frames, and the EEG signal.

[0085] In the embodiment of the present application, feature extraction can be performed on a plurality of first image frames, a plurality of second image frames and an EEG signal, and then the extracted features are fused, and the behavior of the target object is identified based on the fused features.

[0086] Specifically, in some embodiments, convolutional neural networks (CNN) can be used to extract spatial features representing changes in the spatial scene where the target object is located from multiple first image frames, and to extract temporal features representing changes in the target object's action from multiple second image frames.

[0087] For EEG signals, they can be filtered to remove high-frequency noise and other false signals to obtain clearer data. Since different actions or emotional states may be associated with specific EEG frequency band activities, the spectrum characteristics of EEG signals can be extracted through Fourier transform or wavelet transform, and the amplitude values ​​of specific frequency bands, such as α waves, β waves, etc., can be analyzed to obtain EEG features that represent the target object's actions or emotional changes.

[0088] In some embodiments, the extracted spatial features, temporal features, and EEG features can be connected in the same space. For example, features of different modalities can be fused together using vector connections to obtain a larger feature array. Specifically, a multimodal deep learning framework, such as a fusion of a convolutional neural network and a recurrent neural network, can be used to extract features of multiple first image frames, multiple second image frames, and EEG signals, and the extracted spatial features, temporal features, and EEG features can be fused, and the behavior of the target object can be identified based on the fused features. Furthermore, the model can also combine the attention mechanism to dynamically weight spatial features, temporal features, and EEG features, thereby further improving the accuracy of target object behavior recognition.

[0089] According to the behavior recognition method based on multimodal data of the present application, by collecting video data containing the action of the target object, and synchronously collecting the EEG signal of the target object; extracting multiple first image frames representing the change of the spatial scene where the target object is located from the video data, and extracting multiple second image frames representing the change of the action of the target object from the video data; identifying the behavior of the target object according to the multiple first image frames, multiple second image frames and EEG signals. The embodiment of the present application synchronously collects the action video data and EEG signals of the target object, and extracts key image frames representing the change of the spatial scene and the change of the action from the video data, and combines the EEG signals to deeply understand the neural activity behind the behavior, and performs behavior recognition by fusing the multimodal data of video data and EEG signals, comprehensively utilizing visual behavior and physiological behavior, reducing the problem of insufficient accuracy of single modality data in behavior recognition, and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0090] In some embodiments, collecting video data containing target object motion includes:

[0091] Set the acquisition frame rate of the RGB camera to the preset frame rate;

[0092] The target object is photographed by an RGB camera using a preset frame rate, and the photographed image frames are adjusted to a preset size to obtain video data containing the target object's movements.

[0093] In this embodiment, the frame rate refers to the number of image frames that the camera can capture per second, usually expressed as FPS (Frames Per Second). Before shooting the target object, a suitable frame rate can be determined, and the acquisition frame rate can be determined according to the speed of the target object, the complexity of the movement, the required video quality, etc. For example, for fast-moving objects, a higher acquisition frame rate can be set, such as setting the preset frame rate to 60FPS or higher, to obtain clear motion details; and for slower movements, a lower acquisition frame rate can be set, such as setting the preset frame rate to 30FPS.

[0094] In this embodiment, the acquisition frame rate can be set to a preset frame rate using a setting menu of the RGB camera or by configuring the camera through a software interface. The RGB camera can be placed in a suitable position so that the target object is within the field of view of the RGB camera.

[0095] In one example, the video data collection and processing process is as follows:

[0096] Video acquisition: Use a high-definition RGB camera to obtain the action video of the target object. In order to make the resolution of each frame of the video data high enough to clearly capture the action details, the acquisition frame rate can be set to 32FPS;

[0097] Image brightness and contrast adjustment: adjust the brightness and contrast of each frame in the video data to within the threshold range to reduce the impact of lighting changes on subsequent behavior recognition;

[0098] Image normalization: Normalize the size of each frame of the video data to a fixed size of N×N pixels for easy subsequent processing, where N is a preset value, such as 224, 256 or other values.

[0099] Denoising and enhancement: Gaussian filtering technology can be used to remove noise in each frame of the image. Since each image frame in the video data collected by the RGB camera contains information of three channels: red (R), green (G), and blue (B), the dimension of each frame of the image is N×N×3.

[0100] In this embodiment, the acquisition frame rate of the RGB camera is set to a preset frame rate to improve the smoothness and continuity of the video data, and the captured image frame is adjusted to a preset size, which can not only adapt to the input requirements of the subsequent processing system, but also optimize the storage and processing efficiency while maintaining the integrity of the video content.

[0101] In some embodiments, extracting a plurality of first image frames representing changes in a spatial scene where a target object is located from video data, and extracting a plurality of second image frames representing changes in an action of the target object from the video data, comprises:

[0102] Determine the acquisition frame rate of the video data;

[0103] Determine the extraction step according to the acquisition frame rate;

[0104] The image frames in the video data are sampled according to the sampling step to obtain a plurality of first image frames and / or a plurality of second image frames.

[0105] In this embodiment, the extraction step refers to the number of image frames in the video data at which an image frame is extracted as the first image frame. The size of the extraction step directly affects the number of extracted image frames. The larger the extraction step, the fewer the number of image frames extracted per unit time.

[0106] Different extraction steps can be set according to different acquisition frame rates. For example, if the acquisition frame rate is 32FPS, the extraction step can be set to 16, that is, one image frame is extracted every 16 frames as the first image frame or the second image frame, and two first image frames or second image frames are obtained; if the acquisition frame rate is 64FPS, the extraction step can be set to 32, that is, one image frame is extracted every 32 frames as the first image frame or the second image frame, and two first image frames or second image frames are obtained.

[0107] In this embodiment, by determining the extraction step according to the acquisition frame rate, it is possible to analyze the speed of change of the spatial scene where the target object is located and the continuous change of the target object's action in time according to the acquisition frame rate, and then the extraction step can be determined, so that the sampled first image frame and the second image frame can accurately represent the changes in the spatial scene where the target object is located and the continuous change of the target object's action in time, providing accurate spatial information and time information for behavior recognition.

[0108] In some embodiments, extracting a plurality of first image frames representing changes in a spatial scene where a target object is located from video data, and extracting a plurality of second image frames representing changes in an action of the target object from the video data, comprises:

[0109] Determine the acquisition frame rate of the video data;

[0110] Determining a first decimation step and a second decimation step according to the acquisition frame rate; the second decimation step is smaller than the first decimation step;

[0111] The image frames in the video data are sampled according to the first sampling step to obtain a plurality of first image frames; and the image frames in the video data are sampled according to the second sampling step to obtain a plurality of second image frames.

[0112] In this embodiment, since the change of the spatial scene is relatively slow compared to the change of the movement of the target object, a smaller number of first image frames can represent the change of the spatial scene where the target object is located, and a larger first extraction step can be set.

[0113] Specifically, a higher acquisition frame rate means that the video data contains more image frames per second, so the extraction step of the first image frame can be determined according to the value of the acquisition frame rate, thereby controlling the number of first image frames. For example, if the acquisition frame rate is 32FPS, the first extraction step can be set to 16, that is, one image frame is extracted as the first image frame every 16 frames, and 2 first image frames are obtained; if the acquisition frame rate is 64FPS, the first extraction step can be set to 32, that is, one image frame is extracted as the first image frame every 32 frames, and 2 first image frames are obtained.

[0114] In this embodiment, since the target object's motion changes faster than the spatial scene, more image frames are needed to capture the subtle changes in the motion. Compared with the first extraction step, the second extraction step is larger, so that the number of the second image frames extracted per unit time content can be greater than the number of the first image frames.

[0115] For example, if the acquisition frame rate is 32FPS, the first extraction step can be set to 8, that is, one image frame is extracted as the second image frame every 8 frames, and 4 first image frames are obtained; if the acquisition frame rate is 64FPS, the first extraction step can be set to 16, that is, one image frame is extracted as the first image frame every 16 frames, and 4 second image frames are obtained.

[0116] In this embodiment, since the continuous change of action in time is faster than the change of spatial scene, the second extraction step is smaller than the first extraction step, and the image frames can be sampled more densely, so that the number of second image frames per unit time exceeds the number of first image frames, and more action details of the target object can be captured, so that the second image frame can accurately represent the action changes of the target object.

[0117] In some embodiments, identifying the behavior of the target object according to the plurality of first image frames, the plurality of second image frames, and the EEG signal includes:

[0118] Perform frequency domain analysis on the EEG signal, and obtain the period information of the EEG signal based on the frequency of the EEG signal in the frequency domain;

[0119] The EEG signal is converted into a two-dimensional tensor based on multiple cycles according to the cycle information;

[0120] Perform feature extraction on the two-dimensional tensor to obtain an EEG feature map;

[0121] The behavior of the target object is identified according to the multiple first image frames, the multiple second image frames and the electroencephalogram feature map.

[0122] In this embodiment, different frequency bands of EEG signals, such as α wave, β wave, θ wave and other frequency bands, are closely related to human physiological and psychological activities. Through frequency domain analysis, EEG signals can be converted from time domain to frequency domain, so as to obtain the amplitude values ​​of each frequency band of EEG signals. For example, the EEG signals can be converted from time domain to frequency domain by Fourier transform to obtain the amplitude values ​​of EEG signals in each frequency band, and the frequencies with larger amplitude values ​​near α wave and β wave are determined. These frequencies correspond to the main cycles of brain activity, and the period information of EEG signals is determined according to the frequencies.

[0123] In this embodiment, after determining the period information, the one-dimensional EEG signal can be segmented according to the period information, the EEG signal can be divided into multiple subsequences with a length equal to the period length, and the subsequences can be arranged in a preset arrangement to form multiple two-dimensional tensors. The columns of each two-dimensional tensor correspond to different moments in the period, and the rows correspond to different periods.

[0124] In this embodiment, after obtaining the two-dimensional tensor, a deep learning model, such as CNN, can be used to perform a convolution operation on the two-dimensional tensor to extract features from the two-dimensional tensor and obtain an EEG feature map.

[0125] Furthermore, feature extraction and feature fusion may be performed on the plurality of first image frames, the plurality of second image frames and the EEG feature map to identify the behavior of the target object.

[0126] In an example, the process of EEG signal acquisition and processing is as follows:

[0127] EEG signal acquisition: Check whether the target object is wearing the EEG acquisition device correctly. If the target object is wearing the EEG acquisition device correctly, synchronously acquire the EEG signal of the target object.

[0128] Signal filtering: The collected EEG signals are processed by Butterworth filtering to remove low-frequency drift and high-frequency noise. A bandpass filter is used to remove low-frequency drift and high-frequency noise and retain the signal within a specific frequency range, which contains important information related to brain activity.

[0129] Artifact removal: The independent component analysis method is used to remove artifacts caused by blinking, eye movement, head movement, etc. These artifacts will interfere with the correct analysis of EEG signals;

[0130] Period extraction: For a one-dimensional EEG signal with a time series length of T, use Fast Fourier Transform (FFT) to convert the one-dimensional EEG signal from the time domain to the frequency domain to obtain the amplitude of the one-dimensional EEG signal in different frequency domains; considering the sparsity of the frequency domain and avoiding the noise caused by meaningless high frequencies, the first three frequencies with larger amplitude values ​​near the β wave and α wave can be selected; these three frequencies correspond to the three most significant period lengths, thus obtaining the main period information of the data;

[0131] Sequence folding: Based on the three cycles selected above, perform folding operations on the one-dimensional EEG signal in the time domain. If the time series length of the one-dimensional EEG signal cannot be divided by the cycle length, 0 can be added to the end of the one-dimensional EEG signal so that the time series length can be divided by the cycle length; divide the one-dimensional EEG signal with a time series length of T into multiple subsequences with a length of the cycle length, and then arrange these subsequences into two-dimensional tensors in a certain way. Among them, the columns of each two-dimensional tensor correspond to different moments in the cycle, and the rows correspond to different cycles. In this way, the one-dimensional time series is converted into a two-dimensional tensor based on multiple cycles;

[0132] Normalization: Normalize the two-dimensional tensor so that the signals collected from different channels and different time periods are at the same level, which is convenient for subsequent feature extraction and analysis;

[0133] The fixed parameter encoder is used to extract features from the two-dimensional tensor to obtain an EEG feature map with a dimension of N×N×3. The fixed parameter encoder is an encoder that uses a predetermined parameter configuration to extract features from input data.

[0134] In this embodiment, by performing frequency domain analysis on the EEG signal, the amplitude distribution of the EEG signal at different frequencies can be captured, so that the periodic information of the EEG signal can be determined, and the EEG signal is converted into a two-dimensional tensor using the periodic information. Feature extraction is performed on the two-dimensional tensor to capture the changes of the EEG signal at different times to obtain an EEG feature map. The first image frame and the second image frame are then combined to perform behavior recognition using multimodal data that combines video data and EEG signals. Visual behavior and physiological behavior are comprehensively utilized, which reduces the problem of insufficient accuracy of single modality data in behavior recognition and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0135] In some embodiments, identifying the behavior of the target object according to the plurality of first image frames, the plurality of second image frames, and the EEG feature map includes:

[0136] Acquire a plurality of first image frames and a plurality of second image frames extracted from image frames in the same time period in the video data, and acquire an electroencephalogram characteristic map corresponding to the time of the plurality of first image frames and the plurality of second image frames;

[0137] Extracting features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map respectively, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map;

[0138] Identify the behavior of the target object based on the fused features.

[0139] In this embodiment, in order to make the data of different modalities consistent in the time dimension so that the data of different modalities can be effectively fused in the subsequent processing process, multiple first image frames and multiple second image frames extracted from the image frames in the same time period of the video data can be obtained, and the EEG feature map corresponding to the time of the multiple first image frames and multiple second image frames can be obtained. For example, the first image frame obtained is an image frame extracted from the video data in the time period of 2s-3s, and the second image frame obtained is also an image frame extracted from the video data in the time period of 2s-3s, and the EEG feature map is obtained based on the EEG signal in the time period of 2s-3s.

[0140] In some embodiments, a preset multimodal behavior recognition model can be used to respectively extract features of multiple first image frames, multiple second image frames, and EEG feature maps, and fuse the features of multiple first image frames, multiple second image frames, and EEG feature maps; and identify the behavior of the target object based on the fused features.

[0141] In this embodiment, the multimodal behavior recognition model can use a multimodal deep learning framework, such as a fusion of convolutional neural networks and recurrent neural networks, etc., by pre-training the constructed multimodal behavior recognition model so that the model can learn to extract effective features from image data and EEG feature maps, and perform feature fusion to identify the behavior of the target object.

[0142] In this embodiment, the multimodal behavior recognition model can extract spatial features representing changes in the spatial scene where the target object is located from multiple first image frames, extract temporal features representing changes in the target object's movements from multiple second image frames, extract EEG features representing changes in the target object's movements or emotions from the EEG feature map, and fuse the extracted spatial features, temporal features, and EEG features, and recognize the behavior of the target object based on the fused features. Furthermore, the model can also combine the attention mechanism to dynamically weight spatial features, temporal features, and EEG features, thereby further improving the accuracy of target object behavior recognition.

[0143] In this embodiment, by extracting the first image frame and the second image frame from image frames within the same time period and obtaining the EEG feature map corresponding to the image frame time, the synchronization and correlation of the input data are improved, and features are extracted from the image frames and the EEG signals respectively, and these features are fused. This feature fusion not only includes the spatial scene and motion change information at the visual level, but also includes the EEG activity information at the neural level, thereby providing a comprehensive perspective to understand the behavior of the target object, so that the behavior of the target object can be identified more accurately and can adapt to complex and changing environments.

[0144] In some embodiments, extracting features of the plurality of first image frames, the plurality of second image frames, and the EEG feature map respectively, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the EEG feature map comprises:

[0145] Extracting spatial features representing changes in the spatial scene where the target object is located from the plurality of first image frames;

[0146] extracting temporal features representing changes in the motion of the target object from the plurality of second image frames;

[0147] Extracting EEG features of the target object from the EEG feature map;

[0148] The spatial features, temporal features and EEG features are fused.

[0149] In some embodiments, the multimodal behavior recognition model includes an input layer and a feature processing layer; the input layer includes spatial input, temporal input and EEG input; the feature processing layer includes a spatial feature extraction flow, a temporal feature extraction flow and an EEG feature extraction flow;

[0150] A spatial input, used to transmit the plurality of first image frames to the spatial feature extraction stream, so that the spatial feature extraction stream extracts spatial features representing changes in the spatial scene where the target object is located from the plurality of first image frames;

[0151] A time input, used to transmit the plurality of second image frames to the time feature extraction stream, so that the time feature extraction stream extracts the time features representing the change of the target object's motion from the plurality of second image frames;

[0152] The EEG input is used to transmit the EEG feature map to the EEG feature extraction stream, so that the EEG feature extraction stream can extract the EEG features of the target object from the EEG feature map.

[0153] In this embodiment, if Figure 2 As shown, the multimodal behavior recognition model may include an input layer, a feature processing layer, a feature fusion layer, and an output prediction layer.

[0154] The input layer may include spatial input, temporal input and EEG input, which are responsible for transmitting different types of data to corresponding feature extraction streams. Specifically, the spatial input is used to transmit multiple first image frames representing the spatial scene changes where the target object is located to the spatial feature extraction stream. The temporal input is used to transmit multiple second image frames representing the target object's action changes to the temporal feature extraction stream. The EEG input is used to transmit the EEG feature map to the EEG feature extraction stream.

[0155] In this embodiment, the processing flow of the feature processing layer may include a spatial feature extraction flow, a temporal feature extraction flow, and an EEG feature extraction flow. The spatial feature extraction flow may use a convolutional neural network to extract spatial features representing changes in the spatial scene where the target object is located from multiple first image frames. The extracted spatial features may reflect information such as the shape, position, and color of the target object, which is helpful for understanding the overall changes in the spatial scene.

[0156] The temporal feature extraction flow can use a recurrent neural network (RNN) or a long short-term memory network (LSTM) to capture the dynamic changes of the target object in the temporal dimension, thereby extracting temporal features representing the changes in the motion of the target object from multiple second image frames. The temporal features reflect the correlation between the motion changes of the object and time, such as speed, acceleration, etc.

[0157] The EEG feature extraction flow can use a recurrent neural network or a convolutional neural network to extract the EEG features of the target object from the EEG feature map. EEG features can reflect the emotional state, attention level, etc. of the target object, and can provide background information of the psychological state for behavior recognition.

[0158] In this embodiment, the feature fusion layer can splice feature vectors from different streams together to form a comprehensive feature vector. Specifically, weights can be assigned according to the importance of different features to perform weighted averaging, or the attention mechanism can be used to dynamically adjust the contribution of different features to enhance the model's attention to important features. Finally, the fused features are input to the output prediction layer, which can include a fully connected layer, which processes the fused features through the fully connected layer and activates them using an activation function to identify the behavior of the target object.

[0159] Among them, the fully connected layer is responsible for linear processing of the fused features, and the activation function is set after the fully connected layer, which is responsible for introducing nonlinearity so that the network in the model can flexibly identify the behavior of the target object. Specifically, the fully connected layer can map the fused features to a new feature space, and the dimension of this new feature space is pre-set. The fully connected layer achieves this mapping by multiplying each element of the fused features by a weight, then summing all weighted inputs, and adding a bias term, that is, achieving a linear transformation of the fused features, so that the network in the model can capture more complex feature relationships. In order to enable the network to learn and simulate nonlinear relationships, an activation function is set after the fully connected layer. The activation function can be ReLU (Rectified Linear Unit), Sigmoid, etc. The activation function can perform nonlinear transformation on the output of the fully connected layer, so that the network can not only identify simple linear relationships, but also more complex nonlinear behaviors. The fused features are processed and analyzed by the fully connected layer and the activation function, and finally the behavior of the target object is identified.

[0160] In one example, taking the acquisition frame rate of video data as 32FPS as an example, the first extraction stride is set to 16, and 2 frames of first image frames are obtained, and the dimension of the spatial input is 2×224×224×3; where 2 represents 2 frames of first image frames, 224×224 represents the size of the image frame, and 3 represents the number of channels. The second extraction stride is set to 8, and 4 frames of second image frames are obtained. The dimension of the temporal input is 4×224×224×3; where 4 represents 4 frames of second image frames. The EEG input is an EEG feature map, and the input dimension is 1×224×224×3; where 1 represents 1 frame of EEG feature map.

[0161] The extraction process of the spatial feature extraction flow is shown in Table 1.

[0162] Table 1

[0163]

[0164] in:

[0165] B1. Use a 3D convolution kernel of size 1×7×7×3, 16 3D convolution kernels, and a step size parameter of 1×2×2 to perform a 3D convolution operation on the spatial input to obtain a feature map P1 of dimension 2×112×112×16, i.e., the feature map P1 includes 2 frames of images, with a size of 112×112 and 16 channels; the step size parameter of 1×2×2 means that the step size in the time dimension is 1 and the step size in the spatial dimension is 2×2;

[0166] B2. Use a 3D convolution kernel of size 1×3×3×3, 16 convolution kernels, and a step size of 1×2×2 to perform depthwise separable convolution on the feature map P1 to obtain a feature map P2 of size 2×56×56×16.

[0167] B3. Use three 3D convolution kernels on the feature map P2 in sequence, with parameters of 1×1×1, 16, 1×3×3, 16, 1×1×1, 32, and step size parameters of 1×1×1. Repeat the above 3D convolution operation M times to obtain a feature map P3 with a size of 2×56×56×32.

[0168] B4. Apply three 3D convolution kernels to the feature map P3 in sequence, with parameters of 1×1×1, 32, 1×3×3, 32, and 1×1×1, 64. Repeat the above 3D convolution operation N times. The step size of the last 3D convolution is 1×2×2, and the feature map P4 with a size of 2×28×28×64 is obtained.

[0169] B5. Use three 3D convolution kernels on the feature map P4 in turn, with parameters of 1×1×1, 64, 1×3×3, 64, 1×1×1, 128, and a step size of 1×2×2. Repeat the above 3D convolution operation K times. The step size of the last 3D convolution is 1×2×2, and the feature map P5 with a size of 2×14×14×128 is obtained.

[0170] B6. Use three 3D convolution kernels on the feature map P5 in turn, with parameters of 1×1×1, 128, 1×3×3, 128, 1×1×1, 256, and a step size of 1×2×2. Finally, repeat the above 3D convolution operation L times; the step size parameter of the last 3D convolution is 1×2×2, and a feature map P6 with a size of 2×7×7×256 is obtained.

[0171] The extraction process of the temporal feature extraction flow is shown in Table 2.

[0172] Table 2

[0173]

[0174] in:

[0175] C1, use a 3D convolution kernel of size 1×7×7×3, 4 3D convolution kernels, and a step size parameter of 1×2×2 to perform a 3D convolution operation on the temporal input to obtain a 2×112×112×4 feature map Q1;

[0176] C2, perform depth-wise separable convolution on feature map Q1 to obtain feature map Q2 of 4×56×56×4;

[0177] C3. Use three 3D convolution kernels on the feature map Q2 in sequence, with parameters of 1×1×1, 4, 1×3×3, 4, 1×1×1, 32, and the step size parameter is 1×1×1. Repeat the above 3D convolution operation M times to obtain the feature map Q3 with a size of 4×56×56×16;

[0178] C4. Use three 3D convolution kernels on the feature map Q3 in sequence, with parameters of 1×1×1, 8, 1×3×3, 8, and 1×1×1, 32 respectively. Repeat the above 3D convolution operation N times. The step size parameter of the last 3D convolution is 1×2×2, and the feature map Q4 with a size of 4×28×28×32 is obtained.

[0179] C5. Use three 3D convolution kernels on the feature map Q4 in turn, with parameters of 1×1×1, 16, 1×3×3, 16, 1×1×1, 64, and a step size of 1×2×2. Repeat the above 3D convolution operation K times. The step size of the last 3D convolution is 1×2×2, and the feature map Q5 with a size of 4×14×14×64 is obtained.

[0180] C6. Use three 3D convolution kernels on the feature map Q5 in turn. The parameters are 1×1×1, 32, 1×3×3, 32, 1×1×1, 128, and the step size is 1×2×2. Repeat the above 3D convolution operation L times. The step size parameter of the last 3D convolution is 1×2×2, and the feature map Q6 with a size of 4×7×7×128 is obtained.

[0181] The extraction process of the EEG feature extraction flow is shown in Table 3.

[0182] Table 3

[0183]

[0184] in:

[0185] D0, stack the EEG feature map input with a dimension of 1×224×224×3 to obtain a feature map R0 of 2×224×224×3;

[0186] D1, use a 3D convolution kernel of size 1×7×7×3, 4 3D convolution kernels, and a step size parameter of 1×2×2 to perform a 3D convolution operation on the feature map R0 to obtain a feature map R1 of 2×112×112×4;

[0187] D2, perform depth-wise separable convolution on the feature map R1 to obtain a 2×56×56×4 feature map R2;

[0188] D3. Use three 3D convolution kernels on the feature map R2 in sequence, with parameters of 1×1×1, 8, 1×3×3, 8, 1×1×1, 16, and step size parameters of 1×1×1. Finally, repeat the above 3D convolution operation M times to obtain the feature map R3 with a size of 2×56×56×16.

[0189] D4. Apply three 3D convolution kernels to the feature map R3 in sequence, with parameters of 1×1×1, 16, 1×3×3, 16, and 1×1×1, 32. Repeat the above 3D convolution operation N times. The step size parameter of the last 3D convolution is 1×2×2, and the feature map R4 with a size of 2×28×28×32 is obtained.

[0190] D5. Apply three 3D convolution kernels to the feature map R4 in sequence, with parameters of 1×1×1, 32, 1×3×3, 32, 1×1×1, 64, and a step size of 1×2×2. Repeat the above 3D convolution operation K times. The step size of the last 3D convolution is 1×2×2, and a feature map R5 with a size of 2×14×14×64 is obtained.

[0191] D6. Use three 3D convolution kernels on the feature map R5 in turn. The parameters are 1×1×1, 64, 1×3×3, 64, 1×1×1, 128, and the step size is 1×2×2. Repeat the above 3D convolution operation L times. The step size of the last 3D convolution is 1×2×2, and the feature map R6 with a size of 2×7×7×128 is obtained.

[0192] The calculation steps of the feature fusion layer are as follows:

[0193] E1, merge the channels of feature map P6, feature map Q6 and feature map R6 to form a new feature map with a dimension of 8×7×7×512;

[0194] E2. Perform depth-wise separable convolution on the merged feature map, and output a feature map with a dimension of 8×7×7×512;

[0195] E3. Use 3D convolution with a convolution kernel size of 1×7×7×1 to perform dimension adjustment operation, and output a feature map with a dimension of 8×1×1×512;

[0196] E4. Use a convolution kernel of 8×1×1×512 to perform convolution, and further perform convolution operation on the above result to output a feature map with a dimension of 1×1×1×256.

[0197] The calculation steps of the output prediction layer are as follows:

[0198] Fully connected layer operation: The feature map output in E4 corresponds to 256 neurons, and the fully connected layer is used to map the 256-dimensional features to the same dimension as the number of predicted categories. Assuming the number of predicted categories is C, the final number of output neurons is C, and the dimension of the weight matrix is ​​256×C;

[0199] Function activation operation: Use the softmax function to activate the C neurons mentioned above to obtain the final prediction output.

[0200] In this embodiment, by subdividing the model input layer into spatial input, temporal input and EEG input, which correspond to spatial scene changes, motion changes and EEG signals in the video data respectively, the model can capture behavioral features from different dimensions. The feature processing layer further distributes these inputs to corresponding spatial feature extraction streams, temporal feature extraction streams and EEG feature extraction streams. Each extraction stream is specifically responsible for extracting features from the corresponding input. Such division of labor and cooperation improves the pertinence and efficiency of feature extraction. The model fuses these features and performs behavior recognition based on the fused features. This method not only improves the accuracy of behavior recognition, but also enhances the robustness of the model, enabling the model to better adapt to different behaviors and environmental conditions.

[0201] The behavior recognition method based on multimodal data provided in the embodiment of the present application can be executed by a behavior recognition device based on multimodal data. In the embodiment of the present application, the behavior recognition method based on multimodal data is executed by a behavior recognition device based on multimodal data as an example to illustrate the behavior recognition device based on multimodal data provided in the embodiment of the present application.

[0202] The embodiment of the present application also provides a behavior recognition device based on multimodal data.

[0203] like Figure 3 As shown, the behavior recognition device based on multimodal data includes:

[0204] The acquisition module 310 is used to acquire video data including the target object's actions and synchronously acquire the target object's EEG signals;

[0205] An extraction module 320, configured to extract from the video data a plurality of first image frames representing a change in a spatial scene where a target object is located, and to extract from the video data a plurality of second image frames representing a change in an action of the target object;

[0206] The recognition module 330 is used to recognize the behavior of the target object according to the multiple first image frames, the multiple second image frames and the EEG signal.

[0207] According to the behavior recognition device based on multimodal data of the present application, by collecting video data containing the action of the target object, and synchronously collecting the EEG signal of the target object; extracting multiple first image frames representing the change of the spatial scene where the target object is located from the video data, and extracting multiple second image frames representing the change of the action of the target object from the video data; identifying the behavior of the target object according to the multiple first image frames, multiple second image frames and EEG signals. The embodiment of the present application synchronously collects the action video data and EEG signals of the target object, and extracts key image frames representing the change of the spatial scene and the change of the action from the video data, and combines the EEG signals to deeply understand the neural activity behind the behavior, and performs behavior recognition by fusing the multimodal data of video data and EEG signals, comprehensively utilizing visual behavior and physiological behavior, reducing the problem of insufficient accuracy of single modality data in behavior recognition, and can better cope with complex environments and changeable behaviors, thereby improving the accuracy of behavior recognition.

[0208] In some embodiments, the acquisition module 310 is further configured to:

[0209] Set the acquisition frame rate of the RGB camera to the preset frame rate;

[0210] The target object is photographed by an RGB camera using a preset frame rate, and the photographed image frames are adjusted to a preset size to obtain video data containing the target object's movements.

[0211] In some embodiments, the extraction module 320 is further configured to:

[0212] Determine the acquisition frame rate of the video data;

[0213] Determine the extraction step according to the acquisition frame rate;

[0214] The image frames in the video data are sampled according to the sampling step to obtain a plurality of first image frames and / or a plurality of second image frames.

[0215] In some embodiments, the extraction module 320 is further configured to:

[0216] Determine the acquisition frame rate of the video data;

[0217] Determining a first decimation step and a second decimation step according to the acquisition frame rate; the second decimation step is smaller than the first decimation step;

[0218] The image frames in the video data are sampled according to the first sampling step to obtain a plurality of first image frames; and the image frames in the video data are sampled according to the second sampling step to obtain a plurality of second image frames.

[0219] In some embodiments, the identification module 330 is further configured to:

[0220] Perform frequency domain analysis on the EEG signal, and obtain the period information of the EEG signal based on the frequency of the EEG signal in the frequency domain;

[0221] The EEG signal is converted into a two-dimensional tensor based on multiple cycles according to the cycle information;

[0222] Perform feature extraction on the two-dimensional tensor to obtain an EEG feature map;

[0223] The behavior of the target object is identified according to the multiple first image frames, the multiple second image frames and the electroencephalogram feature map.

[0224] In some embodiments, the identification module 330 is further configured to:

[0225] Acquire a plurality of first image frames and a plurality of second image frames extracted from image frames in the same time period in the video data, and acquire an electroencephalogram characteristic map corresponding to the time of the plurality of first image frames and the plurality of second image frames;

[0226] Extracting features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map respectively, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map;

[0227] Identify the behavior of the target object based on the fused features.

[0228] In some embodiments, the identification module 330 is further configured to:

[0229] Extracting spatial features representing changes in the spatial scene where the target object is located from the plurality of first image frames;

[0230] extracting temporal features representing changes in the motion of the target object from the plurality of second image frames;

[0231] Extracting EEG features of the target object from the EEG feature map;

[0232] The spatial features, temporal features and EEG features are integrated.

[0233] The behavior recognition device based on multimodal data in the embodiment of the present application can be an edge computing device, or a component in the edge computing device, such as an integrated circuit or a chip. The edge computing device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the edge computing device can be a mobile phone, a tablet computer, a laptop computer, an eye tracker, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited. In an optional example, an edge deployment solution in which a behavior recognition device based on multimodal data is deployed as an edge computing device or component on the end side can migrate at least part of the computing tasks to the end-side device, leveraging the advantages of edge computing to improve data processing efficiency, reduce latency and network resource occupancy and dependence, and also facilitate data security and privacy protection.

[0234] The behavior recognition device based on multimodal data in the embodiment of the present application can be a device with an operating system. The operating system can be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0235] In some embodiments, Figure 4As shown, an embodiment of the present application also provides an edge computing device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned behavior recognition method based on multimodal data is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0236] It should be noted that the edge computing devices in the embodiments of the present application include the above-mentioned mobile edge computing devices and non-mobile edge computing devices.

[0237] An embodiment of the present application also provides a computer-readable storage medium, on which programs or instructions are stored. When the programs or instructions are executed by a processor, the various processes of the above-mentioned behavior recognition method embodiment based on multimodal data are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.

[0238] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0239] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned behavior recognition method based on multimodal data when executed by a processor.

[0240] The processor is the processor in the edge computing device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0241] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned behavior recognition method embodiment based on multimodal data, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0242] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0243] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0244] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0245] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0246] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0247] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A behavior recognition method based on multimodal data, characterized in that: include: Collecting video data containing the target object's movements, and synchronously collecting the target object's EEG signals; Extracting a plurality of first image frames representing changes in a spatial scene where the target object is located from the video data, and extracting a plurality of second image frames representing changes in actions of the target object from the video data; The behavior of the target object is identified according to the plurality of the first image frames, the plurality of the second image frames, and the EEG signal.

2. The method according to claim 1, characterized in that The collecting of video data containing the target object's action includes: Set the acquisition frame rate of the RGB camera to the preset frame rate; The target object is photographed by the RGB camera using the preset frame rate, and the photographed image frames are adjusted to a preset size to obtain video data containing the target object's movements.

3. The method according to claim 1, characterized in that The extracting of a plurality of first image frames representing changes in a spatial scene where the target object is located from the video data, and extracting a plurality of second image frames representing changes in an action of the target object from the video data, comprises: Determining a capture frame rate of the video data; Determine the extraction stride according to the acquisition frame rate; The image frames in the video data are sampled according to the sampling step to obtain a plurality of first image frames and / or a plurality of second image frames.

4. The method according to claim 1, characterized in that: The extracting of a plurality of first image frames representing changes in a spatial scene where the target object is located from the video data, and extracting a plurality of second image frames representing changes in an action of the target object from the video data, comprises: Determining a capture frame rate of the video data; Determining a first decimation step and a second decimation step according to the acquisition frame rate; the second decimation step is smaller than the first decimation step; The image frames in the video data are sampled according to the first sampling step to obtain a plurality of first image frames; and the image frames in the video data are sampled according to the second sampling step to obtain a plurality of second image frames.

5. The method according to claim 1, characterized in that The identifying the behavior of the target object according to the plurality of the first image frames, the plurality of the second image frames and the EEG signal comprises: Performing frequency domain analysis on the EEG signal, and obtaining period information of the EEG signal based on the frequency of the EEG signal in the frequency domain; Converting the EEG signal into a two-dimensional tensor based on multiple cycles according to the cycle information; Performing feature extraction on the two-dimensional tensor to obtain an EEG feature map; The behavior of the target object is identified according to the plurality of the first image frames, the plurality of the second image frames and the EEG characteristic map.

6. The method according to claim 5, characterized in that The identifying the behavior of the target object according to the plurality of the first image frames, the plurality of the second image frames and the EEG characteristic graph comprises: Acquire a plurality of the first image frames and a plurality of the second image frames extracted from image frames in the same time period in the video data, and acquire the electroencephalogram corresponding to the time of the plurality of the first image frames and the plurality of the second image frames; Extracting features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map respectively, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the electroencephalogram feature map; The behavior of the target object is identified based on the fused features.

7. The method according to claim 6, characterized in that The extracting the features of the plurality of first image frames, the plurality of second image frames, and the EEG feature map respectively, and fusing the features of the plurality of first image frames, the plurality of second image frames, and the EEG feature map comprises: Extracting spatial features representing changes in the spatial scene where the target object is located from the plurality of first image frames; extracting a temporal feature representing a change in the motion of the target object from a plurality of the second image frames; Extracting the EEG features of the target object from the EEG feature map; The spatial features, temporal features and EEG features are fused.

8. A behavior recognition device based on multimodal data, characterized in that: include: An acquisition module, used to acquire video data containing the target object's movements, and synchronously acquire the target object's EEG signals; An extraction module, used to extract a plurality of first image frames representing changes in the spatial scene where the target object is located from the video data, and to extract a plurality of second image frames representing changes in the action of the target object from the video data; A recognition module is used to recognize the behavior of the target object according to the plurality of the first image frames, the plurality of the second image frames and the EEG signal.

9. An edge computing device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the behavior recognition method based on multimodal data as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the behavior recognition method based on multimodal data as described in any one of claims 1 to 7 is implemented.

Images