Behavior recognition method and system

Abstract

The invention provides a behavior recognition method and system, the method is applied to the technical field of computers, and the method comprises the steps: obtaining CSI data and video data of a target area; the video data comprises multiple frames of images; based on convolutional networks of different dimensions of a target recognition model, performing feature extraction of different dimensions on the CSI data and the video data of the target area to obtain feature information of different dimensions; based on a multi-modal data fusion network of the target identification model, fusing the feature information of different dimensions to obtain fused features; and determining a behavior recognition result corresponding to the target area based on the fusion feature. Therefore, the accuracy of behavior recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a behavior recognition method and system. Background Art

[0002] At present, human actions can be identified through various devices or data signals, which can be divided into two categories: contact identification and contactless identification. Among them, the contact human behavior identification method mainly installs wearable devices directly on the human body or relies on devices such as smart phones held by people. Based on the accelerometer, gyroscope and other devices carried in the device, the peak and trough positions of the footsteps are determined according to the range of change of human movement frequency and linear acceleration vector, and then the current behavior of the device carrier is identified. However, this solution has the following disadvantages: 1) It requires independent equipment and is worn on a specific part of the body, which increases the cost and brings inconvenience to the user's actions; 2) It is necessary to analyze and integrate multiple sensor data, and the calculation cost is large; 3) The existence of the device will cause attention intrusion to the user and affect normal life. In view of the above-mentioned shortcomings of the contact human behavior identification method, more and more contactless human behavior identification methods have been proposed. This method mainly relies on a single non-contact device such as a camera to collect data, and uses a single neural network to realize the behavior identification function of a single person or multiple people in a small range. However, this method has the problem of insufficient feature extraction, and the single neural network used is difficult to achieve accurate data processing, which will reduce the behavior recognition effect. Summary of the invention

[0003] The present application provides a behavior recognition method and system, which can improve the accuracy of behavior recognition.

[0004] The technical solution of this application is implemented as follows:

[0005] The present application provides a behavior recognition method, which is applied to a cloud server of a behavior recognition system, and the method includes:

[0006] Acquire channel state information (CSI) data and video data of a target area; the video data includes multiple frames of images;

[0007] Based on convolutional networks of different dimensions of the target recognition model, feature extraction of different dimensions is performed on the CSI data and video data of the target area to obtain feature information of different dimensions;

[0008] Based on the multimodal data fusion network of the target recognition model, the feature information of different dimensions is fused to obtain fused features;

[0009] Based on the fusion features, a behavior recognition result corresponding to the target area is determined.

[0010] The present application provides a behavior recognition system, which includes a cloud server, a WiFi signal receiving module, a base station tower, and a WiFi signal generating module and a camera module deployed on the base station tower, wherein:

[0011] The WiFi signal generating module is used to transmit WiFi signals to the target area;

[0012] The WiFi signal receiving module is used to receive the WiFi signal sent by the WiFi signal generating module to the target area, determine the CSI data of the target area from the received WiFi signal, and upload the CSI data to the cloud server;

[0013] The camera module is used to obtain video data of the target area and upload the video data to the cloud server; the video data includes multiple frames of images;

[0014] The cloud server is used to obtain CSI data and video data of the target area; based on the convolutional networks of different dimensions of the target recognition model, the features of the CSI data and video data of the target area are extracted in different dimensions to obtain feature information of different dimensions; based on the multimodal data fusion network of the target recognition model, the feature information of different dimensions is fused to obtain fusion features; based on the fusion features, the behavior recognition result corresponding to the target area is determined.

[0015] The present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the behavior recognition method provided by one or more of the aforementioned technical solutions when executing the program.

[0016] The present application provides a computer storage medium, which stores a computer program; after the computer program is executed, it can implement the behavior recognition method provided by one or more of the above-mentioned technical solutions.

[0017] An embodiment of the present application provides a behavior recognition method and system, which is applied to a cloud server of a behavior recognition system, and the method includes: acquiring CSI data and video data of a target area; the video data includes multiple frames of images; based on convolutional networks of different dimensions of a target recognition model, performing feature extraction of different dimensions on the CSI data and video data of the target area to obtain feature information of different dimensions; based on a multimodal data fusion network of the target recognition model, fusing the feature information of different dimensions to obtain fused features; and determining a behavior recognition result corresponding to the target area based on the fused features.

[0018] It can be seen that in the embodiment of the present application, the target recognition model can perform behavior recognition based on the CSI data and video data of the target area, that is, data of two different modalities. Compared with the related art of using only a single modality data obtained by a single device for behavior recognition, it can not only improve the accuracy of behavior recognition, but also reduce the model's dependence on a single data source; in addition, using convolutional networks of different dimensions in the model to extract features of different dimensions from the acquired CSI data and video data can ensure the adequacy of feature extraction. If behavior recognition is performed based on the extracted features of different dimensions, the accuracy of behavior recognition can be further improved to ensure the behavior recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the structure of a behavior recognition system provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a scenario of a behavior recognition system provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of another scenario of a behavior recognition system provided in an embodiment of the present application;

[0022] Figure 4 A flowchart of a behavior recognition method provided in an embodiment of the present application;

[0023] Figure 5A A schematic diagram of a result of preprocessing CSI data provided in an embodiment of the present application;

[0024] Figure 5B Another schematic diagram of the result of preprocessing CSI data provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a multi-dimensional convolutional network provided in an embodiment of the present application;

[0026] Figure 7A schematic diagram of the structure of a multimodal data fusion network provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of a multimodal discrimination network provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a CSI denoising network provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of an initial recognition model provided in an embodiment of the present application;

[0030] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solution in this application will be clearly and completely described below in conjunction with the drawings in this application.

[0032] The present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the embodiments provided herein are only used to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the present application can be implemented in any combination.

[0033] It should be noted that, in the present application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or system including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or system. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a system, for example, a unit may be a part of a processor, a part of a program or software, etc.) in the method or system including the element.

[0034] In the related art, non-contact human behavior recognition methods usually use WIFI devices or cameras to realize the behavior recognition function of a single person or multiple people in a small range. However, the behavior recognition method using WIFI devices has the following disadvantages: 1) Severe noise interference. The CSI data obtained by the WIFI device can reflect the changes in the human body's motion state from a macro level, but the signal granularity provided by the existing WiFi devices is limited, which makes the signal amplitude generated by the perception of the human body susceptible to noise interference; 2) It is difficult to identify complex motion problems, and feature extraction is insufficient. When users perform compound actions, there is usually a contextual logical relationship. For example, to complete a squat action, it is necessary to perform standing and waiting, squatting quickly, standing up quickly, and standing and waiting in sequence. Therefore, there will be a waveform change that is first stable, then rapidly oscillating, and finally stable. The single neural network currently used is difficult to achieve accurate processing of such data; 3) The use space is limited, and the recognition accuracy is greatly dependent on the accuracy of the CSI signal. Therefore, in order to minimize the interference of the CSI signal, it is currently applied to small-scale indoor scenes, and multiple transmitters and receivers are required to be evenly arranged. In addition, the behavior recognition method using camera equipment has the following disadvantages: 1) The recognition robustness is not strong and is limited by the video quality. Such solutions are completely dependent on the quality of the video image. Users who occupy a small or blurred portion of the video image are usually difficult to identify; 2) The problem of difficulty in extracting features from time series data.

[0035] In view of the above problems, the following embodiments are proposed.

[0036] In an embodiment of the present application, the behavior recognition method can be applied to a cloud server of a behavior recognition system. The behavior recognition system is suitable for large-scale outdoor scenes and can perform human behavior recognition on one or more users in the large-scale outdoor scenes.

[0037] Figure 1 A schematic diagram of a behavior recognition system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the behavior recognition system includes a cloud server 10, a WiFi signal receiving module 11, a base station tower 12, and a WiFi signal generating module 121 and a camera module 122 deployed on the base station tower 12.

[0038] Among them, the WiFi signal generating module 121 is used to transmit WiFi signals to the target area; the WiFi signal receiving module 11 is used to receive the WiFi signal sent by the WiFi signal generating module to the target area, and determine the CSI data of the target area from the received WiFi signal, and upload the CSI data to the cloud server 10; the camera module 122 is used to obtain video data of the target area, and upload the video data to the cloud server 10; wherein the video data includes multiple frames of images; it should be noted that each frame of the multiple frames can be one of a manned image and an unmanned image.

[0039] Exemplarily, the target area is also called the scene area to be detected, which represents the large-scale outdoor scene area that can be covered by the WiFi signal generating module 121 and the camera module 122; this area is usually located between the base station tower 12 and the WiFi signal receiving module 11 so as to reflect the impact of behavioral changes of users in the area on the WiFi signal.

[0040] In the embodiment of the present application, the size of the target area is not specifically limited, and can usually be set accordingly according to the degree of building obstruction. For example, the range between 1km and 4km centered on the base station tower can be determined as the target area.

[0041] Here, the base station tower 12 is also called a communication base station tower; illustratively, the existing communication base station tower of the communication operator can be directly used as the base station tower, that is, the WiFi signal generating module 121 and the camera module 122 are directly deployed on the existing communication base station tower. It can be seen that the behavior recognition system can rely on the existing communication base station tower and does not need to build an additional base station tower; that is, except for the need to deploy the WiFi signal generating module 121 and the camera module 122 on the base station tower 12, there is no additional construction cost. In this way, it can be ensured that the behavior recognition system has broad application prospects and promotion value.

[0042] Exemplarily, a WiFi signal generating module 121 and a camera module 122 may be deployed on top of the base station tower 12. This deployment ensures that while the WiFi signal generating module 121 on the base station tower 12 transmits WiFi signals to the target area, the camera module 122 acquires video data in the target area, thereby providing the possibility of acquiring two different modalities of data, namely, CSI data and video data corresponding to the WiFi signal; here, different modalities may represent different sources.

[0043] It can be understood that by deploying the WiFi signal generating module 121 and the camera module 122 on the top of the base station tower 12, the height advantage of the base station tower 12 can be fully utilized, so that the WiFi signal generating module 121 can expand the transmission range as much as possible when transmitting the WiFi signal, and the camera module 122 can also expand the shooting range as much as possible. In this way, it can well meet the use requirements of large-scale outdoor scene areas and solve the problem of limited use space when using WIFI devices for behavior recognition in related technologies.

[0044] In some embodiments, the WiFi signal generating module 121 includes at least four groups of WiFi signal generators, which are respectively deployed in various directions of the base station tower 12. The WiFi signal generating module 121 is also used to transmit WiFi signals in various directions of the target area.

[0045] Exemplarily, when the WiFi signal generating module 121 includes four groups of WiFi signal generators, the four groups of WiFi signal generators may be respectively deployed in the southeast, northwest, and northeast directions of the base station tower 12 to ensure that WiFi signals are transmitted in all directions of the target area.

[0046] In some embodiments, the WiFi signal receiving module 11 includes at least four groups of WiFi signal receivers. The WiFi signal receiving module 11 is also used to receive WiFi signals sent by the WiFi signal generating module 121 to all directions of the target area, and determine the CSI data of the target area from the received WiFi signals, and upload the CSI data to the cloud server 10.

[0047] Exemplarily, the number of WiFi signal receivers is the same as that of WiFi signal generators; it should be noted that each WiFi signal receiver can be deployed in various areas outside the target area, including but not limited to areas such as the top of a building; the WiFi signal receiver is mainly used to receive WiFi signals, and determine the CSI data of the target area through the received WiFi signals, so as to upload the CSI data to the cloud server 10.

[0048] Here, there is no specific limitation on the implementation method of determining CSI data. For example, it can be implemented through a CSI tool or through other methods.

[0049] In some embodiments, the camera module 122 includes at least four groups of cameras, which are respectively deployed in various directions of the base station tower 12. The camera module 122 is also used to obtain video data in various directions of the target area and upload the video data to the cloud server 10. Assuming that a user is located in the target area, the video data in various directions of the target area will include the video data of the user.

[0050] Exemplarily, when the camera module 122 includes four groups of cameras, the four groups of cameras can also be deployed in the southeast, northwest, northeast and southwest directions of the base station tower 12 respectively to ensure shooting in all directions of the target area; at this time, the target area is photographed by the cameras deployed in all directions of the base station tower 12. In this way, when there is a user in the target area, video data when the user's behavior changes can be easily obtained.

[0051] Furthermore, when each group of cameras obtains video data of the target area, the video data will be uploaded to the cloud server 10.

[0052] For example, in combination Figure 2 and Figure 3 It can be seen that the layout of the behavior recognition system from the inside to the outside is respectively the WiFi signal generating module 121, the camera module 122, the target area and the WiFi signal receiving module 11. Among them, the four WiFi signal generators included in the WiFi signal generating module 121 can send WiFi signals in all directions. The WiFi signals pass through the target areas in different directions (for example, the north side), and the WiFi signal receivers in the direction (north side) receive the WiFi signals and analyze the CSI data corresponding to the target area. At the same time, the camera module deployed on the base station tower (including four cameras, with the base station tower as the center, covering 360 degrees on the horizontal plane, and the camera on the north side is used for the target area located on the north side) synchronously shoots the video data of the target area, and then uploads the CSI data and video data of the target area to the cloud server. At this time, the cloud server can receive the CSI data and video data of the target area.

[0053] Exemplarily, the cloud server 10 represents a server with a graphics processing unit (GPU). After the cloud server 10 obtains the CSI data and video data of the target area, the specific behavior posture corresponding to the target area, that is, the behavior recognition result, can be obtained by performing a series of data processing on these two parts of data.

[0054] exist Figures 1 to 3 Based on the following Figure 4 The data processing process of the cloud server is illustrated by way of example.

[0055] Figure 4 A flow chart of a behavior recognition method provided in an embodiment of the present application is as follows: Figure 4 As shown, the process may include:

[0056] Step 100: Acquire CSI data and video data of the target area.

[0057] It should be noted that the sources of the CSI data and video data of the target area have been explained in the above embodiments and will not be repeated here. In the embodiments of the present application, after the cloud server obtains the CSI data and video data of the target area, it is necessary to preprocess these two parts of data separately, and then input the preprocessed data into the target recognition model for subsequent behavior prediction.

[0058] Exemplarily, the target recognition model is deployed on a cloud server. The input data of the model includes preprocessed CSI data and video data, and the output data includes behavior prediction results. Based on the behavior prediction results, the behavior recognition results corresponding to the target area can be determined. The model is based on deep learning technology, takes multimodal data as input, is suitable for large-scale outdoor scenes, and has the characteristics of strong feature extraction ability and good CSI signal denoising ability.

[0059] In some embodiments, preprocessing the CSI data of the target area may include: extracting corresponding subcarriers and data packets from the CSI data of the target area; and converting the CSI data of the target area based on the extracted subcarriers and data packets to obtain an RGB image of a fixed size.

[0060] For example, CSI data is a time series data that characterizes WiFi signal transmission, which has multiple dimensions. Here, two dimensions of CSI information can be selected to characterize the impact of user behavior on CSI signals, namely subcarriers and data packets, where data packets are also called time steps. For each data packet (or time step) t, there is the following formula:

[0061] X t =N T XN R N sub (1)

[0062] Among them, N T 、N R and N sub represent the number of transmitter antennas, receiver antennas and subcarriers per antenna respectively. t It can be regarded as a "CSI image" at time point t. If the data packet (time step) dimension is further considered, a "CSI video" that can characterize human behavior and posture changes can be obtained. A "CSI video" is intercepted according to a fixed data packet (time step) and all the information in it is uniformly represented by the 0-255 value range of the RGB three-channel image, and an RGB image with fixed length, width and number of channels can be obtained; Figure 5A and Figure 5B As shown in Figure 1, by intercepting according to a fixed data packet (500ms), an RGB image with a fixed length, width and number of channels can be obtained, where Figure 5A RGB images representing walking behavior, Figure 5B An RGB image representing the falling behavior is obtained; then the RGB image size is set to a fixed size using image interpolation, for example, it can be set to 512×512, thereby completing the conversion of the CSI data from the communication signal to the RGB image. At this point, all CSI data preprocessing work can be completed.

[0063] In some embodiments, preprocessing the video data of the target area may include: performing a frame extraction operation on the video data of the target area according to a set frame extraction frequency to obtain a preset number of image frames.

[0064] Here, the value of the frame extraction frequency can be set according to actual conditions, and the embodiment of the present application does not limit this. For example, it can be set to 60ms or 100ms, etc.

[0065] Exemplarily, by performing a frame extraction operation on the video data at a set frame extraction frequency, a plurality of time-sequentially associated image frames, also called an image sequence, can be obtained; it should be noted that the size of each image frame is the same as the size of the RGB image obtained by converting the above CSI data. If the size of the RGB image obtained according to the CSI data is 512×512, the size of each image frame obtained by frame extraction is fixed to 512×512; the following description takes the size of 512×512 as an example.

[0066] Here, the implementation method of the frame extraction operation can be determined according to the actual situation, and the embodiment of the present application does not make any specific limitation on this. For example, it can be implemented using the algorithm provided by OpenCV, or it can be implemented using other methods.

[0067] Step 101: Based on the convolutional networks of different dimensions of the target recognition model, feature extraction of different dimensions is performed on the CSI data and video data of the target area to obtain feature information of different dimensions.

[0068] In an embodiment of the present application, the target recognition model may include three convolutional networks of different dimensions, namely, a CSI dimension convolutional network, a spatial dimension convolutional network, and a time dimension convolutional network; it should be noted that the network structures of the CSI dimension convolutional network, the spatial dimension convolutional network, and the time dimension convolutional network are the same, but they do not share parameters with each other.

[0069] In some embodiments, convolutional networks of different dimensions based on a target recognition model perform feature extraction of different dimensions on the CSI data and video data of the target area to obtain feature information of different dimensions, which may include: performing feature extraction on the CSI data of the target area based on a CSI dimension convolutional network of the target recognition model to obtain CSI dimension features; performing feature extraction on the video data of the target area based on a time dimension convolutional network of the target recognition model to obtain time dimension features; acquiring key frame images from the video data of the target area, and performing feature extraction on the key frame images based on a space dimension convolutional network of the target recognition model to obtain space dimension features.

[0070] For example, after obtaining the preprocessed CSI data, i.e., a single RGB image, according to the above steps, the single RGB image can be input into the CSI dimensional convolutional network, and the information in the RGB image is parsed by the CSI dimensional convolutional network to output the CSI dimensional feature f representing the image information. csi Specifically, see Figure 6 , it can be seen that the CSI dimension convolutional network includes four different convolutional layers (corresponding to Figure 6 Convolution 1 to Convolution 4 in ) and adaptive pooling layer (corresponding to Figure 6 Adaptive pooling in); first, a single RGB image (corresponding to 1×512×512×3 in the figure, where 1 represents the number of images, 512×512 represents the image size, and 3 represents the number of color channels) is input into convolution 1. After convolution 1 performs convolution processing on the single RGB image, it performs batch normalization processing and maximum pooling processing in sequence to obtain output data. After that, the output data is input into convolution 2 as input data; the data processing methods of convolution 2, convolution 3 and convolution 4 are similar to those of convolution 1, which will not be repeated here; after obtaining the output data of the last convolution layer (convolution 4), the output data is used as the input of the adaptive pooling layer. The output of the adaptive pooling layer is the final output of the CSI dimension convolution network, that is, the CSI dimension feature f csi , where f csi The dimension is 512×4×4, where 512 represents the number of feature channels and 4×4 represents the feature size.

[0071] For example, after the preprocessed video data is obtained according to the above steps, that is, a plurality of RGB images (corresponding to Figure 6 After the image sequence N×512×512×3 is obtained, where N represents the number of images and N is an integer greater than 1), a single image with the median value can be extracted from multiple RGB images as a key frame image, and then the key frame image is input into the spatial dimension convolutional network, and the spatial information in the RGB image is parsed by the spatial dimension convolutional network, and the spatial dimension feature f representing the spatial information of the image is output. s; Among them, f s The dimension is 512 × 4 × 4. It can be understood that the purpose of extracting spatial dimension features is to obtain relevant information such as the scene depicted by the video data. This feature is mainly used for image classification and can be achieved using a single frame image.

[0072] Further, see Figure 6 , it can be seen that the spatial dimension convolutional network also includes four different convolutional layers and adaptive pooling layers; it should be noted that for extracting spatial dimension features f s The method can refer to the above CSI dimension convolutional network to extract CSI dimension features f csi The method will not be described here.

[0073] For example, after obtaining multiple RGB images with time-series association according to the above steps, that is, an image sequence N×512×512×3, grayscale processing and channel dimension splicing can be performed first, and then input into the time dimension convolutional network for processing to output the time dimension feature f t ; Among them, f t The dimension is 512 × 4 × 4. It can be understood that the purpose of extracting time dimension features is to obtain the relative motion state of the camera module and the user in the area, and it is not necessary to pay special attention to the specific category information of the user in the area. In order to save computing resources, multiple grayscale images RGB can be used for feature extraction.

[0074] Further, see Figure 6 , it can be seen that the time dimension convolutional network also includes four different convolutional layers and adaptive pooling layers; it should be noted that for extracting the time dimension feature f t The method can refer to the above CSI dimension convolutional network to extract CSI dimension features f csi The method will not be described here.

[0075] It should be noted that the process of feature extraction through the above three convolutional networks of different dimensions can be parallel. Figure 6 The convolution kernel size, number of channels and stride of each convolution layer included in the convolutional networks of various dimensions shown, as well as the convolution kernel size of the adaptive pooling layer are only examples, and the embodiments of the present application do not make specific limitations on this.

[0076] It can be seen that in the embodiment of the present application, three convolutional networks of different dimensions are used to perform feature extraction on the acquired CSI data and video data. That is, data from different sources can be split into different types of dimensions, so that each convolutional network can focus more on extracting its own dimensional information during feature extraction, which not only reduces the difficulty of feature extraction, but also ensures the adequacy of feature extraction, thereby improving the behavior recognition accuracy of the target recognition model.

[0077] Step 102: Based on the multimodal data fusion network of the target recognition model, feature information of different dimensions is fused to obtain fused features.

[0078] Exemplarily, the target recognition model may further include a multimodal data fusion network; according to the above steps, it can be seen that the feature information of different dimensions may include CSI dimension features f csi , spatial dimension feature f s and the time dimension feature f t That is, the embodiment of the present application can be based on a multimodal data fusion network to perform CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f t The fusion is performed to obtain the fusion features.

[0079] In some embodiments, the multimodal data fusion network includes a pooling layer and a fully connected layer. The multimodal data fusion network based on the target recognition model fuses feature information of different dimensions to obtain fusion features, including: csi , spatial dimension feature f s and the time dimension feature f t splicing to obtain a first splicing feature; using a pooling layer to process the first splicing feature to obtain a pooling feature; using a fully connected layer to process the pooling feature to obtain a CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f t Respective weights; based on CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f t The feature information of different dimensions is fused according to their respective weights to obtain fused features.

[0080] Exemplarily, the pooling layer includes a maximum pooling layer and an average pooling layer, and the pooling feature includes a maximum pooling feature and an average pooling feature; using the pooling layer to process the first splicing feature to obtain the pooling feature may include: using the maximum pooling layer to process the first splicing feature to obtain the maximum pooling feature; using the average pooling layer to process the first splicing feature to obtain the average pooling feature.

[0081] Accordingly, the fully connected layer is used to process the pooled features to obtain the CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f tThe respective weights may include: concatenating the maximum pooling feature and the average pooling feature to obtain a second concatenated feature; processing the second concatenated feature using a fully connected layer to obtain a CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f t Their respective weights; then, according to the CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f t The respective weights are used for the CSI dimension feature f csi , spatial dimension feature f s and the time dimension feature f t Fusion is performed to obtain fusion features; the following is combined Figure 7 The feature fusion process is illustrated by way of example.

[0082] For example, refer to Figure 7 , after obtaining three features of different dimensions (f csi , f s , f t ), and then splicing in the spatial dimension, we can get the first splicing feature f with a dimension of 1536x4x4 concate ; Then, the maximum pooling layer and the average pooling layer are used to respectively concate After processing and average pooling, we get the maximum pooling feature f with a dimension of 1x1536 pool_max And the average pooling feature f of the same dimension pool_avg ; Afterwards, the maximum pooling feature f pool_max And the average pooling feature f pool_avg The concatenation is performed to obtain a concatenated feature with a dimension of 1x3072, and the concatenated feature is input into the fully connected layer.

[0083] Exemplarily, the multimodal data fusion network includes two fully connected layers (corresponding to Figure 7 After obtaining the feature with dimension 1x3072, the concatenated feature is input into the fully connected 1 and fully connected 2 in turn for dimensionality reduction, and the feature with dimension 3 is output after the softmax activation function; wherein, these three dimensions can be represented by λ 1 , 2 and λ 3 Indicates that the CSI dimension feature f is given when multimodal feature fusion is represented. csi , spatial dimension feature f s And the time dimension feature f tIt can be understood that by determining the weights, the model can determine which dimension of information is more reliable during the prediction process, thereby improving the overall robustness and effect of the model.

[0084] Furthermore, after obtaining the CSI dimension feature f csi , time dimension feature f t and spatial dimension feature f s The respective weights λ 1 , 2 and λ 3 After that, it can be fused by formula (2) to obtain the fusion feature f mul :

[0085] f mul =concate(λ 1 *f csi ,λ 2 *f s ,λ 3 *f t ) (2)

[0086] Among them, the fusion feature f mul The dimension is 1536x4x4, and concate is a concatenation function, which means that each variable is concatenated along the channel dimension.

[0087] It should be noted that the weight λ 1 , 2 and λ 3 It is not a constant value, but changes with the input data. Because the softmax activation function is used to determine the weights, the sum of the three weights is always 1.

[0088] For example, for a specific input at a certain moment, if the model believes that the information of the CSI dimension is the most reliable, the time dimension information is second, and the space dimension information is the least reliable, then the weight relationship of the features of each dimension is λ 1 >λ 3 >λ 2 That is, the model can transform the reliability of the corresponding dimensional information into numerically computable variables through the size of the three weights, so as to reflect the reliability of the information on the prediction results and ensure the prediction effect of the model.

[0089] It can be seen that in the embodiment of the present application, the multimodal data fusion network can automatically learn and assign weights to feature data of different dimensions, thereby enabling the model to take advantage of the strengths and weaknesses of the two modal data according to different situations, thereby further improving the robustness and accuracy of the model.

[0090] Step 103: Based on the fused features, determine the behavior recognition result corresponding to the target area.

[0091] In the embodiment of the present application, the target recognition model may also include a multimodal discriminant network. After obtaining the fusion feature f in the above step, mul Afterwards, the fused features can be further processed based on the multimodal discriminant network to obtain the behavior recognition results corresponding to the target area.

[0092] In some embodiments, determining the behavior recognition result corresponding to the target area based on the fused features may include: processing the fused features based on a multimodal discriminant network of the target recognition model to obtain a first prediction result; and determining the behavior recognition result corresponding to the target area based on the first prediction result.

[0093] Exemplarily, the first prediction result may include predicted probability values ​​of multiple preset behavior categories; wherein the behavior recognition result corresponding to the target area may be one of the multiple preset behavior categories, or may not be a preset behavior category; here, the multiple preset behavior categories are various behavior categories that can be recognized by a preset model; the preset behavior categories may include but are not limited to walking behavior, falling behavior, fighting behavior and smoking behavior.

[0094] It should be noted that if there is no user in the target area, the behavior recognition result corresponding to the target area is the recognition result without the user; if there is a single user in the target area, the behavior recognition result corresponding to the target area is the behavior recognition result of the user; if there are multiple users in the target area, the behavior recognition result corresponding to the target area includes the behavior recognition results of the multiple users.

[0095] For example, see Figure 8 , the multimodal discriminant network includes two fully connected layers (corresponding to Figure 8 In the full connection 1 and full connection 2), the fusion feature f with a dimension of 1536x4x4 is obtained. mul After that, it is processed by full connection 1 and full connection 2 for dimensionality reduction, and the first prediction result pred is output after the softmax activation function. mul ; It should be noted that the first prediction result pred mul The dimension is the same as the number of preset behavior categories. When the multiple preset behavior categories include walking behavior, falling behavior, fighting behavior, and smoking behavior, that is, the number of preset behavior categories that the model can identify is 4, the first prediction result pred mul The dimension is 1x4, and the sum of the vectors in each dimension is 1. The vectors in each dimension correspond to the predicted probability values ​​of various preset behavior categories in turn.

[0096] Further, after obtaining the first prediction result pred mulAfter that, the maximum value among the multiple predicted probability values ​​is determined, and the preset behavior category corresponding to the maximum value is determined as the behavior recognition result corresponding to the target area. For example, assuming that the predicted probability values ​​of walking behavior, falling behavior, fighting behavior, and smoking behavior in the first prediction result are 0.4, 0.1, 0.2, and 0.3 respectively, since the predicted probability value of walking behavior 0.4 is the maximum value among the four predicted probability values, walking behavior can be determined as the behavior recognition result corresponding to the target area.

[0097] In some embodiments, the above method also includes: processing the CSI dimensional features based on the CSI denoising network of the target recognition model to obtain a second prediction result; determining the behavior recognition result corresponding to the target area based on the first prediction result, which may include: determining the behavior recognition result corresponding to the target area based on the first prediction result and the second prediction result.

[0098] For example, see Fig. 9 The target recognition model also includes a CSI denoising network, which includes three fully connected layers (corresponding to Fig. 9 In the figure, the first two fully connected layers are followed by the ReLu activation function, and the third fully connected layer is followed by the softmax activation function. csi After that, the CSI dimension feature f csi The dimension changes from 512x4x4 to 1x8192, and then passes through three fully connected layers and softmax activation function to output the second prediction result pred csi ; It should be noted that the second prediction result pred csi The dimension of the first prediction result pred mul The same; that is, in the first prediction result pred mul When the dimension is 1x4 and the sum of the vectors in each dimension is 1, the second prediction result pred csi The dimension of is also 1x4, and the sum of the vectors of each dimension is 1. The vectors of each dimension correspond to the predicted probability values ​​of various preset behavior categories in turn.

[0099] Exemplarily, according to the above content, it can be seen that the first prediction result and the second prediction result both include prediction probability values ​​of various preset behavior categories; after obtaining the first prediction result and the second prediction result, the weight coefficients of the two prediction results can be determined, and the prediction probability values ​​of the same preset behavior categories are multiplied by the corresponding weight coefficients and then added together to obtain the accumulated value of each preset behavior category, and the maximum value among the multiple accumulated values ​​is determined, and the preset behavior category corresponding to the maximum value is determined as the behavior recognition result corresponding to the target area.

[0100] For example, assuming that the predicted probability values ​​of walking behavior, falling behavior, fighting behavior and smoking behavior in the first prediction result are 0.4, 0.1, 0.2, and 0.3 respectively, assuming that the predicted probability values ​​of walking behavior, falling behavior, fighting behavior and smoking behavior in the second prediction result are 0.3, 0.4, 0.2, and 0.1 respectively, and the weight coefficients corresponding to the first prediction result and the second prediction result are both 0.5, then the predicted probability value of each preset behavior category in the first prediction result is multiplied by the weight coefficient 0.5, and 0.2, 0.05, 0. 1, 0.15; multiply the predicted probability value of each preset behavior category in the second prediction result by the weight coefficient 0.5, and obtain 0.15, 0.2, 0.1, and 0.05 respectively; add up the two products corresponding to the same preset behavior categories, and obtain the cumulative values ​​of walking behavior, falling behavior, fighting behavior, and smoking behavior, which are 0.35, 0.25, 0.2, and 0.2 respectively. Since the cumulative value 0.35 corresponding to the walking behavior is the maximum value among the four cumulative values, the walking behavior can be determined as the behavior recognition result corresponding to the target area.

[0101] The embodiment of the present application provides a behavior recognition method and system, the method comprising: obtaining CSI data and video data of a target area; the video data comprises multiple frames of images; based on the convolutional networks of different dimensions of the target recognition model, extracting features of different dimensions from the CSI data and video data of the target area to obtain feature information of different dimensions; based on the multimodal data fusion network of the target recognition model, fusing feature information of different dimensions to obtain fusion features; based on the fusion features, determining the behavior recognition result corresponding to the target area. It can be seen that in the embodiment of the present application, the target recognition model can perform behavior recognition based on the CSI data and video data of the target area, that is, data of two different modalities. Compared with the behavior recognition performed by only using a single modal data obtained by a single device in the related art, it can not only improve the accuracy of behavior recognition, but also reduce the dependence of the model on a single data source; in addition, using the convolutional networks of different dimensions in the model to extract features of different dimensions from the acquired CSI data and video data can ensure the adequacy of feature extraction. If behavior recognition is performed based on the extracted features of different dimensions, the accuracy of behavior recognition can be further improved to ensure the effect of behavior recognition.

[0102] It can be understood that before using the target recognition model to perform behavior recognition on the target area, it is necessary to first construct an initial recognition model and train the initial recognition model to obtain a trained target recognition model; the training process of the initial recognition model is exemplified below.

[0103] In some embodiments, the above method may also include: acquiring historical CSI data and label data of the target area; performing feature extraction on the historical CSI data based on the CSI dimension convolutional network of the initial recognition model to obtain CSI dimension sample features; inputting the CSI dimension sample features into the CSI denoising network of the initial recognition model for processing to obtain a third prediction result; constructing a first loss function based on the third prediction result and the label data; adjusting the parameters of the initial recognition model based on the first loss function to obtain a target recognition model.

[0104] Understandably, in related technologies, CSI data is often used for positioning people and identifying behavior postures in small-scale indoor scenes, and the distance between the data acquisition device and the user cannot be too far. This is because in open environments and long distances, CSI data is easily disturbed by the surrounding environment, resulting in a large amount of noise in the CSI data, which affects the accuracy of user behavior recognition. In order to solve this problem, a CSI denoising network is constructed separately in the initial recognition model to clean the CSI data to adapt to large-scale outdoor scenes and improve recognition performance.

[0105] In the embodiment of the present application, the historical CSI data and label data of the target area are first obtained; wherein the label data is used to indicate the actual behavior results corresponding to the target area in the historical time period; the model structure of the initial recognition model can refer to Fig.10 If the historical CSI data is input into the CSI dimension convolution network of the initial recognition model for feature extraction, the CSI dimension sample features can be obtained; then, the CSI dimension sample features are input into the CSI denoising network of the initial recognition model for processing, and a third prediction result can be obtained; wherein, the third prediction result includes prediction probability values ​​of multiple preset behavior categories; the specific implementation process is similar to the above-mentioned process of determining the second prediction result, and will not be repeated here.

[0106] In order to achieve the denoising effect of the model, a first loss function can be constructed based on the third prediction result of the CSI denoising network and the label data; then, the parameters of the initial recognition model are adjusted based on the first loss function to obtain the target recognition model. Here, the type of the first loss function is not specifically limited, for example, it can be a cross entropy loss function, or other types of loss functions can be used.

[0107] For example, when the first loss function is a cross entropy loss function, if the third prediction result is represented as pred' csi , the label data is represented by y gt , then based on the third prediction result pred' csi and label data y gt The first loss function constructed is loss csi_noise As shown in formula (3):

[0108]

[0109] Here, n represents the number of samples in the historical time period.

[0110] Exemplarily, after obtaining the first loss function loss csi_noise After that, the corresponding loss value can be determined based on the loss function, and then the parameters of the initial recognition model can be adjusted based on the loss value until convergence, that is, the preset training stop condition is reached. Among them, the parameters of the initial recognition model can be parameters related to the performance of the initial recognition model to be trained. Adjusting the parameters of the initial recognition model to be trained according to the loss value can make the prediction performance of the initial recognition model better and better, and the predicted behavior category results are closer and closer to the real behavior category.

[0111] Here, the training stop condition can be the first loss function loss csi_noise The loss value reaches the preset minimum value, or the model performance of the intermediate model has not been significantly improved after a preset number of consecutive iterations. At this point, the trained target recognition model can be obtained.

[0112] In some embodiments, the above method may also include: obtaining historical video data of the target area, extracting features from the historical video data based on the time dimension convolution network of the initial recognition model, and obtaining time dimension sample features; the historical video data includes multiple frames of sample images; obtaining key frame sample images from the historical video data, extracting features from the key frame sample images based on the space dimension convolution network of the initial recognition model, and obtaining space dimension sample features; fusing CSI dimension sample features, time dimension sample features and space dimension sample features based on the multimodal data fusion network of the initial recognition model to obtain sample fusion features; processing the sample fusion features based on the multimodal discriminant network of the initial recognition model to obtain a fourth prediction result; constructing a second loss function based on the fourth prediction result and label data.

[0113] It should be noted that the process of determining the sample fusion feature is similar to the process of determining the fusion feature described above, and will not be repeated here. Exemplarily, after obtaining the sample fusion feature, the sample fusion feature is input into the multimodal discriminant network of the initial recognition model to obtain a fourth prediction result; wherein the fourth prediction result also includes the prediction probability values ​​of multiple preset behavior categories; the specific implementation process is similar to the process of determining the first prediction result described above, and will not be repeated here.

[0114] In an embodiment of the present application, a second loss function can be constructed based on the fourth prediction result and label data of the multimodal discriminant network; wherein the type of the second loss function is the same as the first loss function, that is, it can be a cross entropy loss function, or other types of loss functions can be used.

[0115] In some embodiments, adjusting the parameters of the initial recognition model based on the first loss function to obtain the target recognition model may include: constructing a target loss function based on the first loss function and the second loss function; and adjusting the parameters of the initial recognition model based on the target loss function to obtain the target recognition model.

[0116] Exemplarily, the target loss function can be a comprehensive loss function formed by combining the first loss function and the second loss function; here, there is no specific limitation on the combination of the first loss function and the second loss function, for example, it can be a weighted summation combination or other types of combination.

[0117] Exemplarily, when the combination method is weighted summation, a weight coefficient may be introduced. The weight coefficient may be a value set based on experience or experiments, which will not be described in detail here.

[0118] Exemplarily, when the second loss function is a cross entropy loss function, if the fourth prediction result is represented as pred' mul , then based on the fourth prediction result pred' mul and label data y gt Constructed second loss function loss pose As shown in formula (4):

[0119]

[0120] Furthermore, when the combination method is weighted summation, the target loss function loss total As shown in formula (5):

[0121] loss total =loss csi_noise +loss pose (5)

[0122] Exemplarily, after obtaining the target loss function, the corresponding target loss value can be determined based on the target loss function, and then the parameters of the initial recognition model can be adjusted based on the target loss value until convergence, that is, the preset training stop condition is reached. Adjusting the parameters of the initial recognition model to be trained according to the target loss value can make the prediction performance of the initial recognition model better and better, and the predicted behavior category results are closer and closer to the real behavior category.

[0123] Here, the training stop condition can be the loss of the target loss function total The target loss value reaches the preset minimum value, or the model performance of the intermediate model has not been significantly improved after a preset number of consecutive iterations. At this point, the trained target recognition model can be obtained.

[0124] It can be seen that the embodiment of the present application trains the initial recognition model through the above steps, and after obtaining the trained target recognition model, the behavior category results of users in the target area can be accurately identified when multimodal data (video data and CSI data) are obtained; in addition, since the target recognition model includes a CSI denoising network, through the above training, the noise caused by factors such as human movements, wall isolation, and plant interference in the original CSI data can be filtered out, ensuring the accuracy of the CSI features used by the model, and effectively solving the problem of CSI data containing more noise caused by large-scale outdoor scene transmission, so that the model can be well applied to large-scale outdoor scenes.

[0125] An embodiment of the present application provides a computer storage medium, which stores a computer program; after the computer program is executed, any behavior recognition method of the aforementioned embodiments can be implemented.

[0126] Based on the same technical concept as the above embodiments, see Fig.11 , which shows an electronic device 400 provided in an embodiment of the present application, which may include: a memory 401 and a processor 402; wherein,

[0127] The memory 401 is used to store computer programs and data; the processor 402 is used to execute the computer program stored in the memory to implement any one of the behavior recognition methods in the aforementioned embodiments.

[0128] In practical applications, the memory 401 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 402.

[0129] The processor 402 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that for different behavior recognition systems, the electronic device used to implement the processor function may also be other, which is not specifically limited in the embodiments of the present application.

[0130] In some embodiments, the functions or modules included in the system provided in the embodiments of the present application can be used to execute the method described in the method embodiment above, and its specific implementation can refer to the description of the method embodiment above, and for the sake of brevity, it will not be repeated here. The above description of each embodiment tends to emphasize the differences between the embodiments, and the same or similar aspects can be referenced to each other, and for the sake of brevity, it will not be repeated here.

[0131] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments. The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] The above are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application.

Claims

1. A behavior recognition method, characterized in that: Applied to a cloud server of a behavior recognition system, the method comprises: Acquire CSI data and video data of a target area; the video data includes multiple frames of images; Based on convolutional networks of different dimensions of the target recognition model, feature extraction of different dimensions is performed on the CSI data and video data of the target area to obtain feature information of different dimensions; Based on the multimodal data fusion network of the target recognition model, the feature information of different dimensions is fused to obtain fused features; Based on the fusion features, a behavior recognition result corresponding to the target area is determined.

2. The method according to claim 1, characterized in that The determining, based on the fusion feature, a behavior recognition result corresponding to the target area includes: Based on the multimodal discriminant network of the target recognition model, the fusion feature is processed to obtain a first prediction result; the first prediction result includes prediction probability values ​​of multiple preset behavior categories; Based on the first prediction result, a behavior recognition result corresponding to the target area is determined.

3. The method according to claim 1 or 2, characterized in that: The convolutional networks of different dimensions based on the target recognition model perform feature extraction of different dimensions on the CSI data and video data of the target area to obtain feature information of different dimensions, including: Based on the CSI dimension convolution network of the target recognition model, feature extraction is performed on the CSI data of the target area to obtain CSI dimension features; Based on the time dimension convolution network of the target recognition model, feature extraction is performed on the video data of the target area to obtain time dimension features; A key frame image is obtained from the video data of the target area, and features are extracted from the key frame image based on the spatial dimension convolutional network of the target recognition model to obtain spatial dimension features; the network structures of the CSI dimension convolutional network, the time dimension convolutional network and the spatial dimension convolutional network are the same, and no parameters are shared between them.

4. The method according to claim 3, characterized in that The multimodal data fusion network includes a pooling layer and a fully connected layer. The multimodal data fusion network based on the target recognition model fuses the feature information of different dimensions to obtain fusion features, including: Splicing the CSI dimension feature, the time dimension feature, and the space dimension feature to obtain a first splicing feature; Using the pooling layer to process the first splicing feature to obtain a pooling feature; The pooling feature is processed by the fully connected layer to obtain the weights of the CSI dimension feature, the time dimension feature and the space dimension feature; Based on the respective weights of the CSI dimension feature, the time dimension feature and the space dimension feature, the feature information of the different dimensions is fused to obtain the fused feature.

5. The method according to claim 4, characterized in that The pooling layer includes a maximum pooling layer and an average pooling layer, the pooling feature includes a maximum pooling feature and an average pooling feature, and the first splicing feature is processed by the pooling layer to obtain the pooling feature, including: The maximum pooling layer is used to process the first splicing feature to obtain the maximum pooling feature; the average pooling layer is used to process the first splicing feature to obtain the average pooling feature; Accordingly, the fully connected layer is used to process the pooling feature to obtain the weights of the CSI dimension feature, the time dimension feature, and the space dimension feature, respectively, including: Concatenating the maximum pooling feature and the average pooling feature to obtain a second concatenated feature; The second concatenated feature is processed by the fully connected layer to obtain respective weights of the CSI dimension feature, the time dimension feature, and the space dimension feature.

6. The method according to claim 3, characterized in that The method further comprises: Based on the CSI denoising network of the target recognition model, the CSI dimension feature is processed to obtain a second prediction result; The determining, based on the first prediction result, a behavior recognition result corresponding to the target area includes: Based on the first prediction result and the second prediction result, a behavior recognition result corresponding to the target area is determined.

7. The method according to claim 1, characterized in that The method further comprises: Acquire historical CSI data and label data of the target area; the label data is used to indicate the actual behavior results corresponding to the target area in the historical time period; Based on the CSI dimension convolution network of the initial recognition model, feature extraction is performed on the historical CSI data to obtain CSI dimension sample features; Inputting the CSI dimension sample feature into the CSI denoising network of the initial recognition model for processing to obtain a third prediction result; Based on the third prediction result and the label data, construct a first loss function; The parameters of the initial recognition model are adjusted based on the first loss function to obtain the target recognition model.

8. The method according to claim 7, characterized in that The method further comprises: Acquire historical video data of the target area, and extract features from the historical video data based on the time dimension convolutional network of the initial recognition model to obtain time dimension sample features; the historical video data includes multiple frames of sample images; Acquire key frame sample images from the historical video data, and perform feature extraction on the key frame sample images based on the spatial dimension convolutional network of the initial recognition model to obtain spatial dimension sample features; Based on the multimodal data fusion network of the initial recognition model, the CSI dimension sample features, the time dimension sample features and the space dimension sample features are fused to obtain sample fusion features; Based on the multimodal discriminant network of the initial recognition model, the sample fusion feature is processed to obtain a fourth prediction result; Constructing a second loss function based on the fourth prediction result and the label data; Accordingly, adjusting the parameters of the initial recognition model based on the first loss function to obtain the target recognition model includes: Constructing a target loss function based on the first loss function and the second loss function; The parameters of the initial recognition model are adjusted based on the target loss function to obtain the target recognition model.

9. A behavior recognition system, characterized in that: The system includes a cloud server, a WiFi signal receiving module, a base station tower, and a WiFi signal generating module and a camera module deployed on the base station tower, wherein: The WiFi signal generating module is used to transmit WiFi signals to the target area; The WiFi signal receiving module is used to receive the WiFi signal sent by the WiFi signal generating module to the target area, determine the CSI data of the target area from the received WiFi signal, and upload the CSI data to the cloud server; The camera module is used to obtain video data of the target area and upload the video data to the cloud server; the video data includes multiple frames of images; The cloud server is used to obtain CSI data and video data of the target area; based on the convolutional networks of different dimensions of the target recognition model, the features of the CSI data and video data of the target area are extracted in different dimensions to obtain feature information of different dimensions; based on the multimodal data fusion network of the target recognition model, the feature information of different dimensions is fused to obtain fusion features; based on the fusion features, the behavior recognition result corresponding to the target area is determined.

10. The system according to claim 9, characterized in that The WiFi signal generating module includes at least four groups of WiFi signal generators, which are respectively deployed in various directions of the base station tower. The WiFi signal generating module is also used to transmit WiFi signals in various directions of the target area.

11. The system according to claim 10, characterized in that The WiFi signal receiving module includes at least four groups of WiFi signal receivers. The target area is located between the base station tower and the at least four groups of WiFi signal receivers. The WiFi signal receiving module is also used to receive the WiFi signals sent by the WiFi signal generating module to all directions of the target area, and determine the CSI data of the target area from the received WiFi signals, and upload the CSI data to the cloud server.

Images