Behavior recognition method, system and equipment based on multi-convolutional network, and medium

Abstract

The invention discloses a behavior recognition method, system and device based on a multi-convolution network and a medium, and the method comprises the following steps: processing an input image, replacing an input RGB image with skeleton point collection, and obtaining skeleton point input data; based on an improved traditional time domain convolution module and a combined network of a graph convolution network and a diffusion convolution network, recognition processing of the skeleton point input data is carried out; according to the method, the skeleton points can be used as input data, the parameter quantity and the calculation quantity are greatly reduced, and the calculation efficiency is improved; aiming at the problem that traditional time domain convolution is single in feature extraction, the problems that traditional time domain convolution and graph convolution are inflexible and insufficient in local area range are improved, a fusion model and a thermal accounting method are adopted, and the problem of solving a discrete topological graph neighborhood radius is converted into the problem of solving continuous diffusion time. The optimal neighborhood size and the special propagation range are adaptively learned, and the local modeling capability is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of target recognition technology, and in particular to a behavior recognition method, system, device and medium based on a multi-convolutional network. Background Art

[0002] With the advancement of deep learning technology, behavior recognition technology has developed unprecedentedly and has been applied to many fields such as public safety, traffic safety, and medical health. For example, in the field of public safety, behavior recognition technology can monitor suspicious and dangerous behaviors; in the field of traffic safety, behavior recognition technology can identify traffic violations; in the field of medical health, this technology can monitor the activities of patients to prevent falls.

[0003] The emergence of Convolutional Neural Networks (CNN) has broken the shackles of traditional manual feature extraction and promoted research in multiple fields such as image classification, target detection, and speech recognition. It has been widely used because of its advantages such as large data processing volume, local perception, and resource sharing. CNN technology mostly processes RGB data. In order to enhance the correlation of spatial structure and reduce redundant information, skeleton point data is applied to multiple fields such as behavior recognition and gesture recognition. However, since the topological graph formed by skeleton points does not meet the translation invariance, CNN is no longer applicable. Graph Convolution Network (GCN) can effectively process non-Euclidean data such as topological graphs and capture spatiotemporal information in the spatial and temporal domains. In order to capture the relationship between spatial graph structures, Yan et al. applied graph convolution networks to the field of behavior recognition, adopted a three-partition strategy, and proposed spatiotemporal graph convolution for information fusion in the spatial and temporal domains. On this basis, the subsequent expert team improved the spatial convolution module, using adaptability graph convolution to improve the problem of inflexible topological graph structure; using decoupled graph convolution to increase the expressiveness of spatial fusion without increasing time and memory costs; using shifted graph convolution to achieve feature exchange between channels through shifting and point-by-point convolution, and improving the lightweight degree; improving the neighborhood matrix in spatial graph convolution to solve the problem of biased weights; introducing high-level semantic features to establish spatial and temporal connections; by capturing the relationship between nodes in the image, GCN can effectively extract key information about the action, which is extremely important for behavior recognition. The application of GCN in the field of behavior recognition has proved its significant advantages in processing multi-joint movements and capturing action details, thereby improving the accuracy and robustness of the behavior recognition system. With the continuous evolution of technology, behavior recognition technology is moving towards a more intelligent, real-time and precise direction, providing strong support for the realization of a safer and more convenient social environment.

[0004] However, after analyzing the existing solutions, it can be found that they have the following defects:

[0005] (1) Traditional time-domain convolution has a single scale, and the convolution operation can only capture information within a time segment of a specific length.

[0006] (2) The single convolution scale leads to a fixed receptive field and a single feature extraction. In order to increase the size of the receptive field to capture information at a longer distance, it is usually necessary to stack multiple convolutional layers. This not only increases the computational cost, but also may lead to problems such as gradient vanishing or gradient exploding, affecting the training stability of the model.

[0007] (3) Graph convolutional networks aim to aggregate the neighborhood information and self-information of nodes to obtain feature representations, so capturing local information is key. However, the node associations extracted by spatial graph convolution are usually insufficient.

[0008] (4) Insufficient and inflexible local scope limits the representation level of graph convolution. How to effectively improve the local scope and obtain the optimal neighborhood size is a key issue. Summary of the invention

[0009] The object of the present invention is to provide a behavior recognition method, system, device and medium based on a multi-convolutional network, thereby solving all or one of the above-mentioned problems existing in the prior art.

[0010] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0011] In one aspect, the present invention provides a behavior recognition method based on a multi-convolutional network, comprising the following steps:

[0012] The input image is processed, and the input RGB image is replaced by the skeleton point acquisition to obtain the skeleton point input data;

[0013] Based on the improved traditional time-domain convolution module and the combined network of the graph convolution network and the diffusion convolution network, the recognition processing of the skeleton point input data is performed.

[0014] In one implementation, the step of replacing the input RGB image with the skeleton point acquisition includes:

[0015] The RGB video data is preliminarily processed, or a multimodal sensor is used to obtain skeleton data; the skeleton data only contains information about skeleton points; wherein, when the RGB video data is preliminarily processed, the skeleton point data is obtained based on the posture estimation method to complete the conversion of the RGB video data into skeleton data.

[0016] In one implementation, the graph convolutional network is represented as:

[0017]

[0018] Where I is the identity matrix, W l is a trainable weight matrix and σ(·) represents the activation function.

[0019] In one implementation, the improved traditional time-domain convolution module includes six branches, three of which use dilated convolutions with dilation rates of 1, 2, and 3.

[0020] In one of the implementation methods, in the improved traditional time domain convolution module, horizontal convolution and vertical convolution are used successively to extract features; the other two branches use maximum pooling and average pooling to extract edge information and global information respectively; another branch is 7×1 convolution; finally, channel splicing and fusion of multi-branch features are used to realize the extraction of multi-scale time domain features.

[0021] In one implementation manner, the combination mode of the combination network is: step-by-step fusion in parallel form.

[0022] In one implementation, the parallel step-wise fusion includes:

[0023] Information is exchanged between the ADC module and the GCN unit to ensure that the shape of the ADC output feature map is consistent with the output of the AM-GCN unit.

[0024] On the other hand, the present invention also provides a behavior recognition system based on a multi-convolutional network, comprising:

[0025] The data skeletonization module is used to: process the input image, replace the input RGB image with the skeleton point acquisition, and obtain the skeleton point input data;

[0026] The improved network module is used to: perform recognition processing of the skeleton point input data based on an improved traditional time domain convolution module and a combined network of a graph convolution network and a diffusion convolution network.

[0027] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the behavior recognition method based on a multi-convolutional network are implemented.

[0028] On the other hand, the present invention further provides a computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein:

[0029] The memory is used to store computer programs;

[0030] The processor is used to execute the steps of the behavior recognition method based on multiple convolutional networks by running the program stored in the memory.

[0031] The beneficial effects of the technical solution of the present invention are:

[0032] 1. The behavior recognition method based on multi-convolutional network described in the present invention can achieve the problem of large amount of RGB video data and large amount of parameters, obtain skeleton point data through posture estimation, and obtain a higher level of feature representation; using skeleton points as input data greatly reduces the amount of parameters and calculation, improves the calculation efficiency, and is robust to changes in viewpoint, clothing, background, etc.; in view of the problem that traditional time domain convolution extracts single features, the traditional time domain convolution is improved, and an improved time domain convolution module is proposed, and dilated convolution is used to expand the receptive field while keeping the parameter amount unchanged, and multi-branch, multi-scale asymmetric convolution is used to enrich the time domain information; in view of the problem that the local range of graph convolution is inflexible and insufficient, the present invention combines ADC with graph convolution network to obtain a fusion model, and adopts the hot kernel algorithm to transform the problem of obtaining the neighborhood radius of discrete topological graph into the problem of obtaining continuous diffusion time, adaptively learn the optimal neighborhood size and dedicated propagation range, and enhance local modeling capabilities.

[0033] 2. The behavior recognition system based on multi-convolutional networks described in the present invention can realize the behavior recognition method based on multi-convolutional networks described in the present invention through the mutual cooperation of system modules.

[0034] 3. The computer-readable storage medium described in the present invention can guide the system modules to cooperate, thereby realizing the behavior recognition method based on multi-convolutional networks described in the present invention, and the computer-readable storage medium described in the present invention also effectively improves the operability of the behavior recognition method based on multi-convolutional networks.

[0035] 4. The computer device described in the present invention can store and execute the computer-readable storage medium, thereby realizing the behavior recognition method based on multi-convolutional networks described in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flowchart of the behavior recognition method based on multi-convolutional networks described in Example 1 of the present invention;

[0038] Figure 2 It is a schematic diagram of key points in the behavior recognition method based on multi-convolutional networks described in Example 1 of the present invention;

[0039] Figure 3 It is a schematic diagram of multi-layer graph convolution in the behavior recognition method based on multi-convolutional network described in Example 1 of the present invention;

[0040] Figure 4 Schematic diagram of the ITC module in the behavior recognition method based on multi-convolutional network described in Example 1 of the present invention;

[0041] Figure 5 It is a schematic diagram of diffusion convolution in the behavior recognition method based on multi-convolutional network described in Example 1 of the present invention;

[0042] Figure 6 Schematic diagram of the architecture of the behavior recognition system based on multi-convolutional network according to Embodiment 2 of the present invention;

[0043] Figure 7 is a schematic diagram of the structure of the computer device described in Example 4 of the present invention;

[0044] The symbols in the accompanying drawings are explained as follows:

[0045] 1501, processor; 1502, communication interface; 1503, memory; 1504, communication bus. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0047] In the description of the present invention, it should be noted that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0048] The terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0049] In the description of the present invention, it should be noted that GCN (Graph Convolutional Network) is a graph convolutional network; CNN (Convolution Neural Network) is a convolutional neural network; TC (Temporal Convolution) is a time domain convolution; ITC (Improved Temporal Convolution) is an improved time domain convolution; GDC (Graph Diffusion Convolution) is a graph diffusion convolution; ADC (Adaptive Diffusion Convolution) is an adaptive diffusion convolution.

[0050] Example 1

[0051] This embodiment provides a behavior recognition method based on a multi-convolutional network. Figure 1 to Figure 5 As shown, the following steps are included:

[0052] S100, processing the input image, replacing the input RGB image with skeleton point acquisition, and obtaining skeleton point input data, as follows:

[0053] S101. Replace RGB images with skeleton point collection: Since human behaviors are mostly stored and processed in the form of videos, RGB video data leads to a sharp increase in the number of parameters. Limited by hardware conditions such as GPU and memory, the model training time is long and the calculation cost is high. Therefore, it is necessary to use lighter skeleton data. Skeleton data can be obtained by preliminarily processing the RGB video data or using a multimodal sensor (such as Kinect).

[0054] Among them, skeleton data only contains information about bone points, and the number of parameters is greatly compressed. Moreover, bone point data is less affected by factors such as light intensity, shooting angle, background, and signer's clothing. The robustness of bone point data to the environment provides ideas for sign language recognition.

[0055] Among them, the present invention obtains skeleton point data based on the posture estimation method and completes the conversion of RGB video data into skeleton data. Human posture estimation takes the joint points of human skeleton as the research object. By detecting the position information corresponding to the joint points, estimating the connection between the joint points and reconstructing the human limbs, it is the basis for completing advanced tasks such as behavior recognition. Therefore, the whole body key point annotation method provided by MMPose is used, and HRNet is used as the backbone of the whole body posture estimator. It aims to detect the positions of k key points from an image of size W×H×3. The problem is converted into k heat maps of estimated size W′×H′, each heat map represents the position confidence of the kth key point, and an accurate posture estimation map is obtained after fusing multi-resolution representation.

[0056] Among them, the large number of nodes and edges in the whole body skeleton graph caused the model to introduce a lot of noise, so the graph simplification operation was used, and finally a total of 25 key points of the whole body were obtained, such as Figure 2 shown.

[0057] S200, based on the improved traditional time domain convolution module and the combined network of the graph convolution network and the diffusion convolution network, the recognition processing of the skeleton point input data is performed, and the specifics of each network are as follows:

[0058] S201. Relevant representation of graph convolutional network: In graph convolutional network, for a graph G = (V, E), where V represents the nodes of the graph and E represents the edges of the graph. The input feature graph is represented as X, the feature matrix is ​​represented as Z, and the adjacency matrix is ​​A. The l-layer neural network can be represented as: H l+1 =f(H l ,A) (1);H 0 =X,H l =Z (2); Traditional graph convolution can be viewed as adding neighborhood information to each node, expressed as f(H l ,A)=σ(AH l W l ) (3); On the one hand, the feature matrix is ​​multiplied by the adjacency matrix A, which means that the node has the adjacent node information added, but does not contain its own information, so a self-loop is added to each node. On the other hand, the adjacency matrix A needs to be regularized so that the sum of each row is 1. The present invention adopts Regularization method, where D is the degree matrix, which is also a diagonal matrix, and the elements on the diagonal are the degrees of each vertex; therefore, graph convolution can be expressed as:

[0059]

[0060] Where I is the identity matrix, W l is a trainable weight matrix and σ(·) represents the activation function.

[0061] Among them, for the semi-supervised classification problem, the expected cross entropy of the labeled nodes is used as the loss function:

[0062]

[0063] Among them, y L Represents a set of labeled nodes, trained using gradient descent.

[0064] in, Figure 3 This is a schematic diagram of a multi-layer graph convolutional network for semi-supervised learning, with C as the input channel and X as the input data. i , the output is Z i , the output feature map is F, and the label is Y i express.

[0065] S202, improving the traditional time domain convolution: the present invention improves the time domain convolution module (ITC), such as Figure 4 As shown; ITC contains a total of 6 branches, three of which use dilated convolutions with dilation rates of 1, 2, and 3, which expand the receptive field to varying degrees and solve the problem of increased parameters due to increased receptive field in ordinary convolution. Horizontal convolution and vertical convolution are used to extract features. Compared with square convolution, it not only has advantages in terms of parameter quantity and computational complexity, but also enhances the skeleton part of the convolution kernel and has robustness to rotational deformation. The other two branches use maximum pooling and average pooling to extract edge information and global information respectively, reduce the estimated mean offset and estimated value variance, and improve feature extraction capabilities. The other branch is a 7×1 convolution, which increases the diversity of convolution scales. Finally, channel splicing and fusion of multi-branch features realize the extraction of multi-scale time domain features. In the figure, BN represents the normalization operation, and act represents the activation function (ReLU function is selected);

[0066] Among them, the multi-branch process of the ITC module is expressed as:

[0067]

[0068] x out =x branch1 ||x branch2 ||x branch3 ||x branch4 ||x branch5 ||x branch6 (14);

[0069] in, and All are ordinary convolutions, and the convolution kernels are 1×5, 1×1, and 7×1 respectively. and They represent dilated convolutions with kernel size of 5×1 and dilation rates of 1, 2, and 3, respectively. Average pooling and maximum pooling are denoted by Avgpool(·) and Maxpool(·). The normalization operation is denoted by bn(·), and the activation function is denoted by act(·); the output x of this module is out The six branches are spliced ​​by channels, enriching the multi-scale feature fusion.

[0070] S203, Diffusion Convolution Module: The graph diffusion convolution module GDC is a spatially localized graph convolution that breaks the limitation of using only direct neighboring nodes in traditional graph models; GDC extends the discrete information transfer process in GCN to a diffusion process, enabling it to aggregate information from a larger neighborhood. By using generalized graph diffusion, the problems of noise and arbitrarily defined edges in real graphs are alleviated. Graph diffusion convolution uses generalized graph diffusion to expand and propagate neighborhood information. Generalized graph diffusion is expressed as:

[0071]

[0072] Among them, θ k is the weight coefficient, and T is the transfer matrix. The selection of the two should ensure the convergence of the above formula. The transfer matrix in an undirected graph includes a random walk matrix T with random columns. rw and the symmetric transfer matrix T sym , expressed as:

[0073] T rw =AD -1 (16);

[0074]

[0075] Among them, self-loops are added to the original adjacency matrix to adjust the random walk:

[0076]

[0077] One algorithm used for graph diffusion is the Heat Kernel. The key lies in the implicit mapping from the data space to the usually higher-dimensional feature space to better capture the inherent structure in the data. Simplified, the neighborhood radius of the graph is equal to the diffusion time, and the discrete multi-hop model problem is transformed into a continuous diffusion time problem:

[0078] T=T rw (20);

[0079]

[0080] Among them, Figure 5As shown in the figure, after graph diffusion, a dense matrix is ​​usually obtained, and a sparse matrix is ​​obtained by sparseening the dense matrix. There are two methods for matrix sparseness: the top-k method, which uses the k items with the largest weights; the threshold method, which sets a threshold and sets the items below the threshold to 0. Finally, the sparse matrices are combined to form a new graph, and the given model is run based on the graph;

[0081] In order to adaptively learn the neighborhood radius, that is, adaptively learn the diffusion time t, Adaptive Diffusion Convolution (ADC) is used. ADC can automatically learn the optimal neighborhood size from the data, and learn a dedicated propagation neighborhood for each layer and each feature channel, so that the backbone system is fully coupled with the graph structure.

[0082] S204, graph convolution network combined with diffusion convolution: The edge_index variable of the adaptive diffusion convolution module represents the physical connection relationship of the nodes, and the elements are the node numbers; the corresponding variable in the graph convolution is the neighborhood matrix A, which consists of 0 and 1 elements; when the node v i With v j When there is no connection, the element is 0; when the node v i With v j When there is a connection, the element is 1;

[0083]

[0084] Among them, although the two have similar meanings, the element values ​​and matrix shapes are very different. Therefore, the main body of the framework combines the diffusion convolution with the graph convolution network in a parallel form.

[0085] Among them, the heat kernel algorithm satisfies the neighborhood radius and diffusion time to be equal, and transforms the discrete multi-hop problem into the problem of the continuous optimal value of diffusion time. The present invention adopts the heat kernel algorithm to implement, sets the initial value of diffusion time to 5, and continuously learns during the training process, learns a dedicated propagation neighborhood for each layer and each feature channel, and enhances the flexibility of the model.

[0086] There are two main ways to integrate the ADC module with GCN:

[0087] 1. Direct parallel connection. There is no information exchange between the ADC module and the GCN unit. The output of the stacked ADC modules is then integrated with the backbone framework. The difference lies in the different embedded positions of the modules.

[0088] 2. Step-by-step fusion: There is information exchange between the ADC module and the GCN unit. The ADC is more closely integrated with the backbone network and has a higher degree of coupling with the main framework. It is only necessary to ensure that the shape of the ADC output feature map is consistent with the output of the AM-GCN unit. The present invention adopts a step-by-step fusion method.

[0089] It should be noted that the above examples are only for explaining the present invention and cannot limit the protection scope of the present invention.

[0090] Example 2

[0091] This embodiment is based on the same inventive concept as the behavior recognition method based on a multi-convolutional network described in Example 1, and provides a behavior recognition system based on a multi-convolutional network, such as Figure 6 As shown, including:

[0092] The data skeletonization module is used to: process the input image, replace the input RGB image with the skeleton point acquisition, and obtain the skeleton point input data;

[0093] The improved network module is used to: perform recognition processing of the skeleton point input data based on an improved traditional time domain convolution module and a combined network of a graph convolution network and a diffusion convolution network.

[0094] Example 3

[0095] This embodiment provides a computer-readable storage medium, including:

[0096] The storage medium is used to store computer software instructions used to implement the behavior recognition method based on multi-convolutional networks described in the above-mentioned Example 1, which includes a program for executing the above-mentioned program set for the behavior recognition method based on multi-convolutional networks; specifically, the executable program can be built into the behavior recognition system based on multi-convolutional networks described in Example 2, so that the behavior recognition system based on multi-convolutional networks can implement the behavior recognition method based on multi-convolutional networks described in Example 1 by executing the built-in executable program.

[0097] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof.

[0098] Example 4

[0099] This embodiment provides an electronic device, such as Figure 7 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0100] Memory 1503, used for storing computer programs;

[0101] The processor 1501 is used to implement the steps of the behavior recognition method based on multi-convolutional network described in the above embodiment 1 when executing the computer program stored in the memory 1503.

[0102] As an embodiment of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] As an implementation mode of the present invention, the communication interface is used for communication between the above-mentioned terminal and other devices.

[0104] As an embodiment of the present invention, the memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0105] As an embodiment of the present invention, the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0106] Different from the prior art, the present invention adopts a behavior recognition method, system, device and medium based on a multi-convolutional network, which can obtain skeleton point data through posture estimation to obtain a higher level of feature representation in response to the problem of large RGB video data volume and large parameter volume; using skeleton points as input data greatly reduces the parameter volume and calculation volume, improves the calculation efficiency, and is robust to changes in viewpoint, clothing, background, etc.; in response to the problem of single feature extraction of traditional time domain convolution, the traditional time domain convolution is improved, and an improved time domain convolution module is proposed, and dilated convolution is used to expand the receptive field while keeping the parameter volume unchanged, and multi-branch, multi-scale asymmetric convolution is used to enrich the time domain information; in response to the problem that the local range of graph convolution is inflexible and insufficient, the present invention combines ADC with graph convolution network to obtain a fusion model, and adopts a hot kernel algorithm to transform the problem of obtaining the discrete topological graph neighborhood radius into the problem of obtaining the continuous diffusion time, adaptively learn the optimal neighborhood size and dedicated propagation range, and enhance local modeling capabilities.

[0107] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0108] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0109] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0111] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0113] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0115] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A behavior recognition method based on multi-convolutional networks, characterized in that: The following steps are involved: The input image is processed, and the input RGB image is replaced by the skeleton point acquisition to obtain the skeleton point input data; Based on the improved traditional time-domain convolution module and the combined network of the graph convolution network and the diffusion convolution network, the recognition processing of the skeleton point input data is performed.

2. The behavior recognition method based on multi-convolutional networks according to claim 1, characterized in that: The method of replacing the input RGB image with the skeleton point acquisition includes: The RGB video data is preliminarily processed, or a multimodal sensor is used to obtain skeleton data; the skeleton data only contains information about skeleton points; wherein, when the RGB video data is preliminarily processed, the skeleton point data is obtained based on the posture estimation method to complete the conversion of the RGB video data into skeleton data.

3. The behavior recognition method based on multi-convolutional networks according to claim 1, characterized in that: The graph convolutional network is represented as: Where I is the identity matrix, W l is a trainable weight matrix and σ(·) represents the activation function.

4. The behavior recognition method based on multi-convolutional networks according to claim 1, characterized in that: The improved traditional time-domain convolution module includes six branches, three of which use dilated convolutions with dilation rates of 1, 2, and 3.

5. The behavior recognition method based on multi-convolutional networks according to claim 1, characterized in that: In the improved traditional time domain convolution module, horizontal convolution and vertical convolution are used successively to extract features; the other two branches use maximum pooling and average pooling to extract edge information and global information respectively; the other branch is 7×1 convolution; finally, channel splicing and fusion of multi-branch features are used to realize the extraction of multi-scale time domain features.

6. The behavior recognition method based on multi-convolutional networks according to claim 1, characterized in that: The combination mode of the combination network is: step-by-step fusion in parallel form.

7. The behavior recognition method based on multi-convolutional networks according to claim 6, characterized in that: The step-by-step fusion in parallel form includes: Information is exchanged between the ADC module and the GCN unit to ensure that the shape of the ADC output feature map is consistent with the output of the AM-GCN unit.

8. A behavior recognition system based on multi-convolutional networks, characterized in that: include: The data skeletonization module is used to: process the input image, replace the input RGB image with the skeleton point acquisition, and obtain the skeleton point input data; The improved network module is used to: perform recognition processing of the skeleton point input data based on an improved traditional time domain convolution module and a combined network of a graph convolution network and a diffusion convolution network.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the behavior recognition method based on a multi-convolutional network in any one of claims 1 to 7 are implemented.

10. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; wherein: The memory is used to store computer programs; The processor is used to execute the steps of the behavior recognition method based on multi-convolutional networks in any one of claims 1 to 7 by running the program stored in the memory.

Images