Object classification method, device, electronic device and storage medium
By constructing peer relationship matrix and spatiotemporal data tensor, temporal and spatial features are extracted, and personnel classification is combined with graph convolutional neural network, the problem of low classification accuracy in the existing technology is solved, the classification accuracy is improved and over-context is avoided combine.
Patent Information
- Application Number
- CN202010816697.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-14
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-08-14
AI Technical Summary
The existing personnel classification method based on space-time trajectory is due to the large amount of data and complex information, which leads to low classification accuracy and overfitting.
By acquiring image data, constructing peer relationship matrix and spatiotemporal data tensors, using convolution kernels to extract temporal and spatial features, and combining with graph convolutional neural networks for classification prediction, reducing data dimensions and avoiding overfitting.
The accuracy of personnel classification is improved, and by combining peer relationships and spatiotemporal data, the object attribute categories are more accurately predicted, reducing the risk of overfitting in the classification process.
Smart Images

Figure CN112132175B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to an object classification method, device, electronic device and storage medium. Background Art
[0002] With the development of artificial intelligence technology, the concepts and projects of smart communities and smart cities are increasingly mentioned and implemented. In the projects of smart communities and smart cities, information management and information analysis of people in communities and cities are one of the most important tasks. In projects such as smart communities, a large number of image acquisition cameras are often deployed. Using these image acquisition cameras, we can obtain a large amount of personnel image data. These personnel image data represent the trajectory information of when and where a person appeared. Based on these trajectory information, data analysis of personnel behavior or classification of personnel can be performed. When classifying personnel, the attributes of personnel are usually judged by trajectory information, and then attribute classification is performed. However, since the trajectory information contains a large amount of time and space information, for example, 100,000 people are sampled by 1,000 image acquisition cameras. If the time granularity is 1 hour, the data volume is 100,000*1000*24=2.4*10 9 The data combines time and space information, resulting in a high degree of information complexity, which makes the classification results overfit and reduces the accuracy of personnel classification. Therefore, the existing methods of personnel classification based on spatiotemporal trajectories have the problem of low classification accuracy. Summary of the invention
[0003] The embodiment of the present invention provides an object classification method, which can improve the accuracy of personnel classification.
[0004] In a first aspect, an embodiment of the present invention provides an object classification method, comprising:
[0005] Acquire image data of N objects, and extract peer relationships of the N objects according to the image data to construct an adjacency matrix;
[0006] Extracting original spatiotemporal data tensors of the N objects according to the image data, wherein the original spatiotemporal data tensors include data of time dimension and space dimension;
[0007] Performing a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension;
[0008] Performing a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time feature of the original spatiotemporal data tensor in the spatial dimension;
[0009] Determine the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features;
[0010] The adjacency matrix and the spatiotemporal data tensor are input into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects.
[0011] In a second aspect, an embodiment of the present invention provides an object classification device, including:
[0012] A first extraction module is used to obtain image data of N objects, and extract the peer relationship of the N objects according to the image data to construct an adjacency matrix;
[0013] A second extraction module, configured to extract original spatiotemporal data tensors of the N objects according to the image data, wherein the original spatiotemporal data tensors include data of a time dimension and a space dimension;
[0014] A third extraction module, configured to perform a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract spatial features of the original spatiotemporal data tensor in a time dimension;
[0015] a fourth extraction module, configured to perform a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract a time feature of the original spatiotemporal data tensor in a spatial dimension;
[0016] A first processing module, used for determining the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features;
[0017] The first classification module inputs the adjacency matrix and the spatiotemporal data tensor into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects.
[0018] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the object classification method provided in the embodiment of the present invention when executing the computer program.
[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the object classification method provided in the embodiment of the invention are implemented.
[0020] In an embodiment of the present invention, image data of N objects are obtained, and the peer relationship of the N objects is extracted according to the image data to construct an adjacency matrix; the original spatiotemporal data tensor of the N objects is extracted according to the image data, and the original spatiotemporal data tensor includes data of the time dimension and the space dimension; a convolution operation is performed on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension; a convolution operation is performed on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensor in the space dimension; the spatiotemporal data tensor of the N objects is determined according to the spatial features and the time features; the adjacency matrix and the spatiotemporal data tensor are input into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects. By combining peer relationships with spatiotemporal data to classify object attributes, the object attribute categories can be predicted more accurately, improving the classification accuracy. Moreover, by extracting time features and spatial features separately, the dimension of the original spatiotemporal data tensor is reduced, thereby avoiding overfitting in the classification process and further improving the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0022] Figure 1 is a framework diagram of an object classification system provided by an embodiment of the present invention;
[0023] Figure 2 is a flow chart of an object classification method provided by an embodiment of the present invention;
[0024] Figure 3 is a flow chart of a method for calculating orientation information provided by an embodiment of the present invention;
[0025] Figure 4 is a flow chart of a method for determining whether a user is facing toward a first device provided by an embodiment of the present invention;
[0026] Figure 5 is a flow chart of a method for obtaining sample orientation information provided by an embodiment of the present invention;
[0027] Figure 6 is a schematic diagram of the structure of an object classification device provided by an embodiment of the present invention;
[0028] Figure 7 is a structural schematic diagram of a control module provided by an embodiment of the present invention;
[0029] Figure 8 is a structural schematic diagram of an extraction module provided by an embodiment of the present invention;
[0030] Fig. 9 is a structural diagram of a computing submodule provided by an embodiment of the present invention;
[0031] Fig.10 is a schematic diagram of the structure of another object classification device provided by an embodiment of the present invention;
[0032] Fig.11 is a structural schematic diagram of a second acquisition module provided by an embodiment of the present invention;
[0033] Fig.12 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] See also Figure 1 , Figure 1 is a flow chart of an object classification method provided by an embodiment of the present invention. Figure 1 As shown, the object classification method includes:
[0036] 101. Obtain image data of N objects, and extract peer relationships of the N objects according to the image data to construct an adjacency matrix.
[0037] In this step, the above-mentioned object may be a person object, and the above-mentioned image data may be an image of a person object captured by an image acquisition device at a certain time in a certain area, and may be a face image or a full-body image of a person object. In the embodiment of the present invention, the above-mentioned image data is preferably a face image. The above-mentioned face image may be a face image extracted after face detection is performed based on a full-body image of a person. For example, if the image acquisition device captures a large image, and the large image includes H person objects, the faces of the H person objects can be detected by a face detection engine, and the face images of the H person objects are extracted respectively. The face image can be understood as a small image extracted from the large image.
[0038] After acquiring the facial image, the facial image and the corresponding person are archived, and an image archive is established for all facial images of a person object, and the image archives of each person object are obtained and stored in the archive library, which includes the image data of the above-mentioned N objects. Further, the above-mentioned image data can be a structured or semi-structured facial image, and the above-mentioned structured or semi-structured facial image at least includes the facial image information, the facial image time information, and the facial image location information. Among them, the above-mentioned time information can be determined by the time when the image acquisition device acquires the facial image. The above-mentioned location information can also be called spatial information, which can be determined by the location where the image acquisition device is set, and further, it can also be determined by the number of the image acquisition device, wherein the number of the image acquisition device is not repeated, and then the area is set by binding the number, so that the spatial information can be determined according to the number of the image acquisition device.
[0039] The above-mentioned peer relationship refers to two or more person objects appearing together in a certain place. In the image data, the peer person objects can be analyzed by time information and space information. For example, in two image files, under the same image acquisition device, the acquisition time point difference of two face images is less than the preset time, such as 2 seconds or 3 seconds, which is regarded as one peer relationship. When the number of peer encounters in two image files is greater than or equal to the preset number, it can be determined that the person objects corresponding to the two image files have a peer relationship.
[0040] The above adjacency matrix refers to the adjacency relationship between each two personnel objects in N objects. If there is a peer relationship between two personnel objects, the adjacency relationship between the two personnel objects is a peer relationship. If there is no peer relationship between the two personnel objects, it can be represented by a preset value, such as 1. If there is no peer relationship between the two personnel objects, it can be represented by another preset value, such as 0. For N objects, an N*N adjacency matrix A can be constructed. The adjacency matrix A is a second-order tensor. In the adjacency matrix, A i,j represents the adjacency relationship between person object i and person object j, which can also be understood as the adjacency relationship between image file i and image file j. For example, if A i,j =1, it can be said that the adjacency relationship between personnel object i and personnel object j is a peer relationship. i,j =0, it can be indicated that the adjacency relationship between personnel object i and personnel object j is not a peer relationship. Specifically, the above adjacency matrix can be shown in Table 1:
[0041] 1 … i … j … N 1 <![CDATA[A 1,1 ]]> … <![CDATA[A 1,i ]]> … <![CDATA[A 1,j ]]> … <![CDATA[A 1,N ]]> … … … … … … … … i <![CDATA[A i,1 ]]> … <![CDATA[A i,i ]]> … <![CDATA[A i,j ]]> … <![CDATA[A i,N ]]> … … … … … … … … j <![CDATA[A j,1 ]]> … <![CDATA[A j,i ]]> … <![CDATA[A j,j ]]> … <![CDATA[A j,N ]]> … … … … … … … … N <![CDATA[A N,1 ]]> … <![CDATA[A N,i ]]> … <![CDATA[A N,j ]]> … <![CDATA[A N,N ]]>
[0042] Table 1
[0043] In Table 1, A1,1 , A i,i , A j,j , A N,N The matrix elements located on the diagonal line represent the peer relationship between the person and himself, which can be represented by no peer relationship. If 1 represents peer relationship and 0 represents no peer relationship, then 0 can be used to represent the above A 1,1 , A i,i , A j,j , A N,N Matrix elements that are equally located on the diagonal.
[0044] In the embodiment of the present invention, the adjacency matrix can play a priori role and provide prior guidance for classification through peer relationships, thereby improving the accuracy of classification.
[0045] 102. Extract original spatiotemporal data tensors of N objects according to the image data.
[0046] The original spatiotemporal data tensor includes data of time dimension and space dimension. The above image data may be a structured or semi-structured face image, and the structured or semi-structured face image at least includes the face image information, the face image time information, and the face image location information. The data of the above time dimension may be determined according to the face image time information, and the data of the above space dimension may be determined according to the face image location information. The above time dimension is expressed by a preset time granularity, for example, the time granularity is 1 hour, which means 24 hours of collection statistics, and the original spatiotemporal data tensor includes 24-dimensional time information. The above space dimension is determined by the number of image acquisition devices, for example, if 1000 image acquisition devices are used for image acquisition, the original spatiotemporal data tensor includes 1000-dimensional time information. The original spatiotemporal data tensor of the above N personnel objects can be expressed as a third-order tensor. Assuming that the time dimension is T and the space dimension is L, the original spatiotemporal data tensor of the N personnel objects is expressed as a third-order tensor of N*T*L. The third-order tensor expresses the image acquisition data of N personnel objects in T time periods under L image acquisition devices. However, since the third-order tensor has many 0 values (not every personnel object will be captured by all image acquisition devices), the third-order tensor is a highly sparse data tensor, containing more redundant information, and due to the high data complexity (data complexity is N*T*L), if the third-order tensor is directly used for classification, it is easy to cause overfitting during classification.
[0047] 103. Perform a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension.
[0048] In this step, the number of the first convolution kernels may be a preset number, and each first convolution kernel slides along the time dimension of the original spatiotemporal data tensor to extract spatial features in different time periods.
[0049] For details, please refer to Figure 2 , Figure 2 is a flow chart of a spatial feature extraction method provided by an embodiment of the present invention. Figure 2 As shown, including:
[0050] 201. Obtain the pre-trained K1 first convolution kernels.
[0051] Among them, the above-mentioned first convolution kernel includes a first time dimension and a first space dimension, the above-mentioned first time dimension is in the same direction as the time dimension of the original spatiotemporal data tensor, the above-mentioned first space dimension is in the same direction as the space dimension of the original spatiotemporal data tensor, and the above-mentioned first time dimension is smaller than the time dimension of the above-mentioned original spatiotemporal data tensor, and the above-mentioned first space dimension is equal to the space dimension of the above-mentioned original spatiotemporal data tensor.
[0052] Specifically, the size of the first convolution kernel can be s t *L, indicating that each first convolution kernel is an s t *L is a second-order tensor. Among them, the above s t is the first time dimension of the first convolution kernel, and L is the first space dimension of the first convolution kernel. The first space dimension L is equal to the space dimension L in the original spatiotemporal data tensor. The first time dimension s t is smaller than the time dimension T in the original spatiotemporal data tensor, i.e., s t <T, so that sliding space can be left in the time dimension of the original spatiotemporal data tensor, so that the first convolution kernel can slide on the time dimension of the original spatiotemporal data tensor. For example, when the original spatiotemporal data tensor is N*T*L=100000*24*1000, the first convolution kernel is s t *1000,s t <24.
[0053] In this step, the training of the first convolution kernel can be performed through training data, which includes label values. During the training process, the result obtained by the first convolution kernel is calculated with the label value to obtain a loss error. The error is propagated back through back propagation, and the parameters of the first convolution kernel are adjusted. The above process is iterated to a preset number of times (for example, 200 times) or the loss error converges, that is, when the error change is less than the preset value, the final trained first convolution kernel is obtained.
[0054] 202. Perform a convolution operation on the time dimension of the original spatiotemporal data tensor with a preset convolution step size through the K1 first convolution kernels to obtain K1 spatial features.
[0055] In this step, each first convolution kernel can obtain a spatial feature after performing a convolution operation on the time dimension of the above-mentioned original spatiotemporal data tensor with a preset convolution step size, and K1 first convolution kernels can obtain K1 spatial features. The dimension of the spatial feature is related to the number of sliding times of the first convolution kernel, which can be specifically the number of sliding times + 1.
[0056] In a possible embodiment, the original spatiotemporal data tensor is N*T*L, and the first convolution kernel is s t *L, when the convolution step size is 1, the number of slides is T-s t -1 times, the dimension of the spatial feature obtained after convolution is T-s t .
[0057] 203. Based on the K1 spatial features, the spatial features of the original spatiotemporal data tensor in the time dimension are calculated.
[0058] In step 202, K1 spatial features are calculated, and the dimension of the spatial features is T-s t , which represents each spatial feature is a first-order tensor, which can also be called a spatial feature vector.
[0059] Among the K1 first convolution kernels, the first time dimension s of each first convolution kernel t can be the same or different, but the first spatial dimension L of each first convolution kernel is the same, that is, since there is a first time dimension s in the K1 first convolution kernels t Different situations make the dimensions of the K1 spatial features obtained after convolution different. At this time, the K1 spatial features can be downsampled or linearly transformed to obtain spatial features of preset dimensions, which can be the dimensions T-s of different spatial features. t In addition, the preset dimension in the embodiment of the present invention can also be 1, that is, the dimension T-s of each different spatial feature is t Downsampling or linear transformation into a 1-dimensional feature value, then K1 spatial features will be obtained, and the K1-dimensional spatial feature is a first-order tensor used to represent the spatial features of the corresponding person object in each time period.
[0060] 104. Perform a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensor in the spatial dimension.
[0061] In this step, the number of the second convolution kernels may be a preset number, and each second convolution kernel slides along the spatial dimension direction of the original spatiotemporal data tensor to extract temporal features in different spaces.
[0062] For details, please refer to Figure 3 , Figure 3 is a flow chart of a time feature extraction method provided by an embodiment of the present invention. Figure 3 As shown, including:
[0063] 301. Obtain K2 pre-trained second convolution kernels.
[0064] Among them, the above-mentioned second convolution kernel includes a second time dimension and a second space dimension, the above-mentioned second time dimension is in the same direction as the time dimension of the original space-time data tensor, the above-mentioned second space dimension is in the same direction as the space dimension of the original space-time data tensor, and the above-mentioned second time dimension is equal to the time dimension of the above-mentioned original space-time data tensor, and the above-mentioned second space dimension is smaller than the space dimension of the above-mentioned original space-time data tensor.
[0065] Specifically, the size of the second convolution kernel can be T*s l , indicating that each second convolution kernel is a T*s l The second-order tensor of . Among them, the above s l is the second spatial dimension of the second convolution kernel, and the above T is the second time dimension of the second convolution kernel. The above second time dimension T is equal to the time dimension T in the original spatiotemporal data tensor. The above second spatial dimension s l is smaller than the spatial dimension L of the original spatiotemporal data tensor, i.e., s l <L, so that sliding space can be left in the spatial dimension of the original spatiotemporal data tensor, so that the second convolution kernel can slide on the spatial dimension of the original spatiotemporal data tensor. For example, when the original spatiotemporal data tensor is N*T*L=100000*24*1000, the first convolution kernel is 24*s l ,s l <1000.
[0066] In this step, the training of the second convolution kernel can be performed through training data, which includes label values. During the training process, the result obtained by the first convolution kernel is calculated with the label value to obtain the loss error. The error is propagated back through back propagation, and the parameters of the second convolution kernel are adjusted. The above process is iterated to a preset number of times (for example, 200 times) or the loss error converges, that is, when the error change is less than the preset value, the final trained second convolution kernel is obtained.
[0067] It should be noted that the above-mentioned K1 and the above-mentioned K2 can be the same or different, that is, the number of the first convolution kernels and the number of the second convolution kernels can be the same or different, and there is no necessary relationship between the number of the first convolution kernels and the number of the second convolution kernels.
[0068] 302. Perform a convolution operation on the spatial dimension of the original spatiotemporal data tensor with a preset convolution step size through the K2 second convolution kernels to obtain K2 time features.
[0069] In this step, each second convolution kernel can obtain a time feature after performing a convolution operation on the spatial dimension of the above-mentioned original spatiotemporal data tensor with a preset convolution step size, and K2 second convolution kernels can obtain K2 time features. The dimension of the time feature is related to the sliding number of the second convolution kernel, which can be specifically the sliding number + 1.
[0070] In a possible embodiment, the original spatiotemporal data tensor is N*T*L, and the second convolution kernel is T*s l , when the convolution step size is 1, the number of slides is L-s l -1 times, the dimension of the spatial feature obtained after convolution is L-s l .
[0071] 303. Based on the K2 spatial features, the spatial features of the original spatiotemporal data tensor in the time dimension are calculated.
[0072] In step 302, K2 spatial features are calculated, and the dimension of the spatial features is L-s l , which represents each time feature, is a first-order tensor, which can also be called a time feature vector.
[0073] Among the K2 second convolution kernels, the second spatial dimension of each second convolution kernel is s l can be the same or different, but the second time dimension T of each second convolution kernel is the same, that is, since there is a second spatial dimension s in the K2 second convolution kernels l In different situations, the dimensions of the K2 time features obtained after convolution are also different. At this time, the K2 time features can be downsampled or linearly transformed to obtain time features of preset dimensions, which can be the dimensions L-s of different time features. l In addition, the preset dimension in the embodiment of the present invention can also be 1, that is, the dimension L-s of each different time feature is lDownsampling or linear transformation into a 1-dimensional feature value, then K2 time features will be obtained, and the K2-dimensional time feature will be a first-order tensor used to represent the time features of the corresponding person object in each space.
[0074] The above-mentioned downsampling can be a pooling operation, which can be a maximum pooling or an average pooling. For different time feature dimensions L-s l , different pooling kernels can be used for sampling, such as L-s l =3, then a pooling kernel with a length of 3 is used for sampling. Suppose the time feature is (1, 3, 5), and it is downsampled to a 1-dimensional eigenvalue. After average sampling through the pooling kernel, the average value of the eigenvalue (1, 3, 5) is 3. After maximum sampling through the pooling kernel, the maximum value of the eigenvalue (1, 3, 5) is 5.
[0075] 105. Determine the spatiotemporal data tensor of N objects according to the spatial features and the temporal features.
[0076] In this step, the above-mentioned spatial features and temporal features may be concatenated to obtain the spatiotemporal data tensor of each object, and further obtain the spatiotemporal data tensors of N objects.
[0077] In a possible embodiment, the spatial features and the temporal features may be aligned in dimension to obtain aligned spatial features and temporal features; the aligned spatial features and temporal features are then spliced to obtain the spatiotemporal data tensor of the N objects. Specifically, for example, the spatial features and the temporal features are aligned to K-dimensional features, that is, K-dimensional temporal features and K-dimensional spatial features, and then a 2K-dimensional spatiotemporal data tensor is obtained after splicing, and the dimension of the spatiotemporal data tensor of the N objects is N*2K.
[0078] In another possible embodiment, by combining step 203 and step 303, K1-dimensional spatial features and K2-dimensional temporal features are obtained. The K1-dimensional spatial features and K2-dimensional temporal features can be spliced to obtain a K1+K2-dimensional spatiotemporal data tensor. Further, the dimension of the spatiotemporal data tensor of N objects is N*(K1+K2). Of course, when K1=K2=K, the dimension of the spatiotemporal data tensor of N objects is N*2K.
[0079] Through step 105, the time feature and the space feature are fused to obtain a spatiotemporal data tensor. Compared with the original spatiotemporal data tensor, the second-order spatiotemporal data tensor is significantly smaller than the original spatiotemporal data tensor in data complexity. The spatiotemporal data tensor is equivalent to the data after dimensionality reduction of the original spatiotemporal data tensor.
[0080] 106. Input the adjacency matrix and the spatiotemporal data tensor into the preset graph convolutional neural network for classification prediction to obtain the classification results of N objects.
[0081] In this step, the above-mentioned graph convolutional neural network includes a preset weight matrix, which includes category distribution information. The above-mentioned category distribution information is used to predict the category of each object person among N object persons, thereby completing the classification prediction of the N object persons.
[0082] Specifically, the calculation can be performed through the adjacency matrix, the spatiotemporal data tensor, and the above-mentioned weight matrix. For example, assuming that the tensor dimension of the adjacency matrix is N*N, the dimension of the spatiotemporal data tensor is N*2K, and the tensor dimension of the weight matrix is 2K*F, where F is the number of categories, indicating that there are F categories. In the tensor operation, N*N·N*2K=N*2K, N*2K·2K*F=N*F, and finally a N*F category distribution matrix is obtained. For a certain person object among the N person objects, the category with the largest corresponding F value is the category predicted for the certain person object.
[0083] Optional, see Figure 4 , Figure 4 is a flow chart of a classification prediction method provided by an embodiment of the present invention, wherein in the embodiment of the present invention, the graph convolutional neural network includes at least a preset first weight matrix and a second weight matrix, and the first weight matrix and the second weight matrix include category distribution information. Figure 4 As shown, including:
[0084] 401. Calculate a first result matrix according to the adjacency matrix, the spatiotemporal data tensor, and the first weight matrix.
[0085] 402. Calculate a category distribution matrix according to the adjacency matrix, the first result matrix, and the second weight matrix.
[0086] 403. Determine classification results of the N objects according to the category distribution matrix.
[0087] The first weight matrix in the above step 401 includes first category distribution information, which can be understood as an intermediate implicit category. Similarly, the above first result matrix can also be understood as an intermediate implicit result. The second weight matrix in step 402 includes second category distribution information. If there are only two layers in the graph convolutional neural network, corresponding to the first weight matrix and the second weight matrix respectively, the above second category distribution information can be understood as the final category distribution information. For example, assuming that the first weight matrix is W0, the tensor dimension of W0 is 2K*C, the second weight matrix is W1, and the tensor dimension of W1 is C*F, then at this time, the first weight matrix W0 is the weight matrix from the input layer to the hidden layer, and the first weight matrix W0 has C categories of feature mapping (indicating that there are C intermediate implicit categories), and the second weight matrix W1 is the weight matrix from the hidden layer to the output layer, and the second weight matrix W1 has F categories of feature mapping (indicating that there are F categories). At the same time, assuming that the tensor dimension of the adjacency matrix is N*N, and the dimension of the spatiotemporal data tensor is N*2K, then in the first layer of tensor operations of the graph convolutional neural network, N*N·N*2K=N*2K, N*2K·2K*C=N*C, where N*C is the first result matrix. In the second layer of tensor operations of the graph convolutional neural network, N*N·N*C=N*C, N*C·C*F=N*F, and finally a N*F category distribution matrix is obtained. For a certain person object among the N person objects, the category with the largest corresponding F value is the category predicted for that person object.
[0088] Furthermore, in step 401 and step 402, the adjacency matrix may be preprocessed first, and the preprocessing may be to calculate the normalized Laplacian matrix of the adjacency matrix, and then perform subsequent tensor operations. In this way, the speed of data fitting can be improved.
[0089] Specifically, the category distribution matrix can be calculated by the following objective function:
[0090]
[0091] Among them, the above Z is the category distribution matrix, the above X is the spatiotemporal data tensor, and the above A is the adjacency matrix. In the first layer of graph convolutional network and the second layer of graph convolutional network, It is the tensor operation of the first layer of graph convolutional network, and its result is used as the first result matrix and also as the input of the second layer of graph convolutional network. is the tensor operation of the second layer graph convolution network. It can be the adjacency matrix after the adjacency matrix A is preprocessed, or it can be the Laplace matrix after the adjacency matrix is normalized. Specifically, it can be:
[0092]
[0093] The above D is a diagonal matrix. Each element on the diagonal line of the above diagonal matrix D is the sum of the row or column of the adjacency matrix A corresponding to the element. For example, assuming that the adjacency matrix A is:
[0094] Assuming the sum of the elements in each row of the adjacency matrix A, the diagonal matrix D is:
[0095] For example, the tensor dimension of the adjacency matrix A is N*N, and the tensor dimension of the diagonal matrix is also N*N. Therefore, The tensor dimension is also N*N. In the first layer of the graph convolutional network above, The tensor dimension of is also N*N, the tensor dimension of X is N*2K, and the tensor dimension of W0 is 2K*C, so the tensor operation In the above example, N*N·N*2K=N*2K, N*2K·2K*C=N*C, where N*C is the first result matrix. The tensor dimension of W1 is C*F, and the tensor operation In the equation, N*N·N*C=N*C, N*C·C*F=N*F, and finally a category distribution matrix of N*F is obtained. For a certain person object among the N person objects, the category with the largest corresponding F value is the category predicted for the certain person object.
[0096] The adjacency matrix is preprocessed by calculating the -1 / 2 exponential power of each element in the diagonal matrix D to obtain the normalized Laplace matrix of the adjacency matrix, which can make the value stable and avoid gradient explosion or gradient disappearance.
[0097] In an embodiment of the present invention, image data of N objects are obtained, and the peer relationship of the N objects is extracted according to the image data to construct an adjacency matrix; the original spatiotemporal data tensor of the N objects is extracted according to the image data, and the original spatiotemporal data tensor includes data of the time dimension and the space dimension; a convolution operation is performed on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension; a convolution operation is performed on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensor in the space dimension; the spatiotemporal data tensor of the N objects is determined according to the spatial features and the time features; the adjacency matrix and the spatiotemporal data tensor are input into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects. By combining peer relationships with spatiotemporal data to classify object attributes, the object attribute categories can be predicted more accurately, improving the classification accuracy. Moreover, by extracting time features and spatial features separately, the dimension of the original spatiotemporal data tensor is reduced, thereby avoiding overfitting in the classification process and further improving the classification accuracy.
[0098] It should be noted that the object classification method provided in the embodiment of the present invention can be applied to devices such as mobile phones, monitors, computers, servers, and computer peripherals that can perform object classification.
[0099] See also Figure 5 , Figure 5 is a flowchart of a graph convolutional network training method provided by an embodiment of the present invention, such as Figure 5 As shown, the method includes:
[0100] 501. Obtain sample image data of M objects, and extract peer relationships of the M objects according to the image data to construct an adjacency matrix.
[0101] The M objects include Y classified objects, the M is greater than Y, and the Y classified objects include corresponding labels.
[0102] In this step, the object may be a person object, and the image data may be an image of a person object captured by an image acquisition device at a certain time in a certain area, and may be a face image or a full-body image of a person object. In the embodiment of the present invention, the image data is preferably a face image. After acquiring the face image, the face image and the corresponding person are archived, and an image archive is established for all face images of a person object, and image archives of each person object are obtained and stored in an archive library, which includes the image data of the M objects.
[0103] The above-mentioned classified objects refer to personnel objects classified by experts. The above-mentioned classifications can be, for example, electronic enthusiasts, small animal lovers, photography enthusiasts, etc. The specific classifications can be determined according to different needs. The labels of the above-mentioned classified objects correspond to each classification. It should be noted that the number of the above-mentioned classified objects should not be less than the total number of categories, that is, if the number of categories is F, then Y is greater than or equal to F. Furthermore, the above-mentioned Y classified objects should include classified objects of all categories.
[0104] The above-mentioned peer relationship refers to two or more person objects appearing together in a certain place. In the image data, the peer person objects can be analyzed by time information and space information. For example, in two image files, under the same image acquisition device, the acquisition time point difference of two face images is less than the preset time, such as 2 seconds or 3 seconds, which is regarded as one peer relationship. When the number of peer encounters in two image files is greater than or equal to the preset number, it can be determined that the person objects corresponding to the two image files have a peer relationship.
[0105] The above adjacency matrix refers to the adjacency relationship between every two personnel objects in M objects. If there is a peer relationship between two personnel objects, the adjacency relationship between the two personnel objects is a peer relationship. If the adjacency relationship between the two personnel objects is not a peer relationship, it can be represented by a preset value, such as 1. If there is no peer relationship between the two personnel objects, it can be represented by another preset value, such as 0. For M objects, an M*M adjacency matrix A can be constructed, and the adjacency matrix A is a second-order tensor.
[0106] 502. Extract original spatiotemporal data tensors of M objects according to the image data.
[0107] The above-mentioned original spatiotemporal data tensor includes data of time dimension and space dimension.
[0108] The original spatiotemporal data tensor of the M personnel objects mentioned above can be expressed as a third-order tensor. Assuming that the time dimension is T and the space dimension is L, the original spatiotemporal data tensor of the M personnel objects is expressed as a third-order tensor of M*T*L. The third-order tensor expresses the image acquisition data of the M personnel objects in T time periods under L image acquisition devices.
[0109] 503. Perform a convolution operation on the original spatiotemporal data tensors of the M objects through a preset first convolution kernel to extract spatial features of the original spatiotemporal data tensors of the M objects in the time dimension.
[0110] In this step, the number of the first convolution kernels may be a preset number, and each first convolution kernel slides along the time dimension of the original spatiotemporal data tensor to extract spatial features in different time periods.
[0111] Specifically, the size of the first convolution kernel can be s t *L, indicating that each first convolution kernel is an s t *L is a second-order tensor. Among them, the above s t is the first time dimension of the first convolution kernel, and L is the first space dimension of the first convolution kernel. The first space dimension L is equal to the space dimension L in the original spatiotemporal data tensor. The first time dimension s t is smaller than the time dimension T in the original spatiotemporal data tensor, i.e., s t <T, so that sliding space can be left in the time dimension of the original spatiotemporal data tensor, so that the first convolution kernel can slide on the time dimension of the original spatiotemporal data tensor. Furthermore, after each first convolution kernel performs a convolution operation on the time dimension of the above original spatiotemporal data tensor with a preset convolution step size, a spatial feature can be obtained, and K1 first convolution kernels can obtain K1 spatial features. The dimension of the spatial feature is related to the number of sliding times of the first convolution kernel, which can be specifically the number of sliding times + 1. Among the K1 first convolution kernels, the first time dimension s of each first convolution kernel is t can be the same or different, but the first spatial dimension L of each first convolution kernel is the same, that is, since there is a first time dimension s in the K1 first convolution kernels t Different situations make the dimensions of the K1 spatial features obtained after convolution different. At this time, the K1 spatial features can be downsampled or linearly transformed to obtain spatial features of preset dimensions, which can be the dimensions T-s of different spatial features. t A common divisor of , such as 1.
[0112] 504. Perform a convolution operation on the original spatiotemporal data tensors of the M objects through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensors of the M objects in the spatial dimension.
[0113] In this step, the number of the second convolution kernels may be a preset number, and each second convolution kernel slides along the spatial dimension direction of the original spatiotemporal data tensor to extract temporal features in different spaces.
[0114] Specifically, the size of the second convolution kernel can be T*s l , indicating that each second convolution kernel is a T*s l The second-order tensor of . Among them, the above s lis the second spatial dimension of the second convolution kernel, and the above T is the second time dimension of the second convolution kernel. The above second time dimension T is equal to the time dimension T in the original spatiotemporal data tensor. The above second spatial dimension s l is smaller than the spatial dimension L of the original spatiotemporal data tensor, i.e., s l <L, in this way, sliding space can be left in the spatial dimension of the original spatiotemporal data tensor, so that the second convolution kernel can slide on the spatial dimension of the original spatiotemporal data tensor.
[0115] After each second convolution kernel performs a convolution operation on the spatial dimension of the original spatiotemporal data tensor with a preset convolution step size, a time feature can be obtained. K2 second convolution kernels can obtain K2 time features. The dimension of the time feature is related to the number of sliding times of the second convolution kernel, which can be specifically the number of sliding times + 1. Among the K2 second convolution kernels, the second spatial dimension s of each second convolution kernel is l can be the same or different, but the second time dimension T of each second convolution kernel is the same, that is, since there is a second spatial dimension s in the K2 second convolution kernels l In different situations, the dimensions of the K2 time features obtained after convolution are also different. At this time, the K2 time features can be downsampled or linearly transformed to obtain time features of preset dimensions, which can be the dimensions L-s of different time features. l A common divisor of , such as 1.
[0116] It should be noted that the above-mentioned K1 and the above-mentioned K2 can be the same or different, that is, the number of the first convolution kernels and the number of the second convolution kernels can be the same or different, and there is no necessary relationship between the number of the first convolution kernels and the number of the second convolution kernels.
[0117] 505. Determine the spatiotemporal data tensors of the M objects according to the spatial features and the temporal features.
[0118] In this step, the above-mentioned spatial features and temporal features can be spliced to obtain the spatiotemporal data tensor of each object, and then the spatiotemporal data tensor of M objects can be obtained. The K1-dimensional spatial features and the K2-dimensional temporal features can be spliced to obtain the K1+K2-dimensional spatiotemporal data tensor, and the dimension of the spatiotemporal data tensor of M objects is M*(K1+K2). Of course, when K1=K2=K, the dimension of the spatiotemporal data tensor of M objects is M*2K.
[0119] 506. Input the adjacency matrix and the spatiotemporal data tensor of the M objects into the graph convolutional neural network to be trained for classification prediction, and obtain the classification results of the M objects.
[0120] Among them, the graph convolutional neural network to be trained includes at least two weight matrices, each weight matrix corresponds to a layer of graph convolutional neural network, namely the first weight matrix and the second weight matrix. Further, the first weight matrix includes first category distribution information, which can be understood as an intermediate implicit category. Similarly, the first result matrix can also be understood as an intermediate implicit result. The second weight matrix includes second category distribution information. If there are only two layers in the graph convolutional neural network, corresponding to the first weight matrix and the second weight matrix, the second category distribution information can be understood as the final category distribution information. For example, assuming that the first weight matrix is W0, the tensor dimension of W0 is 2K*C, the second weight matrix is W1, and the tensor dimension of W1 is C*F, then at this time, the first weight matrix W0 is the weight matrix from the input layer to the hidden layer, and the first weight matrix W0 has C categories of feature mapping (indicating that there are C intermediate implicit categories), and the second weight matrix W1 is the weight matrix from the hidden layer to the output layer, and the second weight matrix W1 has F categories of feature mapping (indicating that there are F categories). At the same time, assuming that the tensor dimension of the adjacency matrix is M*M, and the dimension of the spatiotemporal data tensor is M*2K, then in the first layer of tensor operations of the graph convolutional neural network, M*M·M*2K=M*2K, M*2K·2K*C=M*C, where M*C is the first result matrix. In the second layer of tensor operations of the graph convolutional neural network, M*M·M*C=M*C, M*C·C*F=M*F, and finally a M*F category distribution matrix is obtained. For a certain person object among the M person objects, the category with the largest corresponding F value is the category predicted for that person object.
[0121] 507. Extract classification results of Y classified objects from the classification results of the M objects, and calculate the loss of the classification results of the Y classified objects and the labels corresponding to the Y classified objects.
[0122] After the category distribution matrix Z is obtained in step 506, the classification results of the Y classified objects can be extracted from the category distribution matrix Z, thereby calculating the loss of the classification results of the Y classified objects and the corresponding classification labels. The loss function L is as follows:
[0123] Among them, the above y L represents the label set of Y classified objects, F represents the total number of categories, and Y lf Represents the classification label corresponding to the classified object, Z lf Represents the classification results corresponding to the above classified objects in the category distribution matrix Z.
[0124] 508. Back-propagation is performed on the graph convolutional neural network to be trained according to the loss to adjust the graph convolutional neural network to be trained.
[0125] After the error loss is calculated, back propagation is performed through the gradient descent method to adjust the weight matrix of the graph convolutional neural network to be trained. In an embodiment of the present invention, the first weight matrix W0 and the second weight matrix W1 are adjusted through back propagation.
[0126] In a possible embodiment, the first convolution kernel and the second convolution kernel mentioned above can be added for joint training, and the first convolution kernel and the second convolution kernel are also adjusted during back propagation. In this way, the training samples can be reduced. At the same time, since the first convolution kernel and the second convolution kernel are adjusted at the same time through the same training data, the fitting effect of the graph convolutional neural network can be improved, thereby improving the accuracy of classification.
[0127] 509. Iteratively train the graph convolutional neural network to be trained until the loss converges or reaches a preset number of iterations to obtain a trained graph convolutional neural network.
[0128] In this step, the timing of the end of iterative training may be when the number of iterations reaches a preset number, such as when the iterations reach 200 times. The timing of the end of iterative training may also be when the loss converges, such as when the loss value does not change for multiple consecutive times.
[0129] The trained graph convolutional neural network includes a trained weight matrix, and further includes a trained first weight matrix W0 and a second weight matrix W1. The trained graph convolutional neural network can be used for Figure 1 In the embodiment, the above N objects are classified.
[0130] After the graph convolutional neural network is trained, the M objects mentioned above can be classified through the graph convolutional neural network, or the classification result obtained in the last iteration can be used as the training result of the M objects mentioned above.
[0131] See also Figure 6 , Figure 6 is a schematic diagram of the structure of an object classification device provided by an embodiment of the present invention, such as Figure 6 As shown, the device comprises:
[0132] A first extraction module 601 is used to obtain image data of N objects, and extract the peer relationships of the N objects according to the image data to construct an adjacency matrix;
[0133] A second extraction module 602 is used to extract the original spatiotemporal data tensors of the N objects according to the image data, wherein the original spatiotemporal data tensors include data of time dimension and space dimension;
[0134] A third extraction module 603 is used to perform a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension;
[0135] A fourth extraction module 604 is used to perform a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time feature of the original spatiotemporal data tensor in the spatial dimension;
[0136] A first processing module 605 is used to determine the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features;
[0137] The first classification module 606 is used to input the adjacency matrix and the spatiotemporal data tensor into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects.
[0138] Optional, such as Figure 7 As shown, the third extraction module 603 includes:
[0139] A first acquisition submodule 6031 is used to pre-train K1 first convolution kernels, wherein the first convolution kernel includes a first time dimension and a first space dimension, the first time dimension and the time dimension of the original spatiotemporal data tensor are in the same direction, the first space dimension and the space dimension of the original spatiotemporal data tensor are in the same direction, and the first time dimension is smaller than the time dimension of the original spatiotemporal data tensor, and the first space dimension is equal to the space dimension of the original spatiotemporal data tensor;
[0140] A first calculation submodule 6032 is used to perform a convolution operation on the time dimension of the original spatiotemporal data tensor with a preset convolution step size through the K1 first convolution kernels to obtain K1 spatial features;
[0141] The second calculation submodule 6033 is used to calculate the spatial features of the original spatiotemporal data tensor in the time dimension based on the K1 spatial features.
[0142] Optional, such as Figure 8 As shown, the fourth extraction module 604 includes:
[0143] A second acquisition submodule 6041 is used to acquire pre-trained K2 second convolution kernels, wherein the second convolution kernel includes a second time dimension and a second space dimension, the second time dimension is in the same direction as the time dimension of the original spatiotemporal data tensor, the second space dimension is in the same direction as the space dimension of the original spatiotemporal data tensor, and the second time dimension is equal to the time dimension of the original spatiotemporal data tensor, and the second space dimension is smaller than the space dimension of the original spatiotemporal data tensor;
[0144] A third calculation submodule 6042 is used to perform a convolution operation on the spatial dimension of the original spatiotemporal data tensor with a preset convolution step size through the K2 second convolution kernels to obtain K2 time features;
[0145] The fourth calculation submodule 6043 is used to calculate the time feature of the original spatiotemporal data tensor in the spatial dimension based on the K2 time features.
[0146] Optional, such as Fig. 9 As shown, the first processing module 605 includes:
[0147] An alignment submodule 6051 is used to align the spatial feature with the temporal feature in dimension to obtain aligned spatial feature and temporal feature;
[0148] The splicing submodule 6052 is used to splice the aligned spatial features and temporal features to obtain the spatiotemporal data tensors of the N objects.
[0149] Optional, such as Fig.10 As shown, the graph convolutional neural network includes at least a pre-trained first weight matrix and a second weight matrix, the first weight matrix and the second weight matrix include category distribution information, and the first classification module 606 includes:
[0150] A fifth calculation submodule 6061 is used to calculate a first result matrix according to the adjacency matrix, the spatiotemporal data tensor and the first weight matrix;
[0151] A sixth calculation submodule 6062 is used to calculate a category distribution matrix according to the adjacency matrix, the first result matrix and the second weight matrix;
[0152] The determination submodule 6063 is used to determine the classification results of the N objects according to the category distribution matrix.
[0153] Optionally, the fifth calculation submodule 6061 is further used to preprocess the adjacency matrix, and calculate a first result matrix according to the adjacency matrix obtained after the preprocessing, the spatiotemporal data tensor and the first weight matrix;
[0154] The sixth calculation submodule 6062 is further used to calculate a category distribution matrix according to the adjacency matrix obtained after the preprocessing, the first result matrix and the second weight matrix.
[0155] Optional, such as Fig.11 As shown, the device also includes a training device, and the training device includes:
[0156] A fifth extraction module 607 is used to obtain sample image data of M objects, and extract peer relationships of the M objects according to the image data to construct an adjacency matrix, wherein the M objects include Y classified objects, M is greater than Y, and the Y classified objects include corresponding labels;
[0157] A sixth extraction module 608, configured to extract original spatiotemporal data tensors of the M objects according to the image data, wherein the original spatiotemporal data tensors include data of a time dimension and a space dimension;
[0158] A seventh extraction module 609 is used to perform a convolution operation on the original spatiotemporal data tensors of the M objects through a preset first convolution kernel to extract spatial features of the original spatiotemporal data tensors of the M objects in the time dimension;
[0159] An eighth extraction module 610, configured to perform a convolution operation on the original spatiotemporal data tensors of the M objects through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensors of the M objects in the spatial dimension;
[0160] A second processing module 611 is used to determine the spatiotemporal data tensors of the M objects according to the spatial features and the temporal features;
[0161] A second classification module 612 is used to input the adjacency matrix and the spatiotemporal data tensors of the M objects into the graph convolutional neural network to be trained for classification prediction, so as to obtain classification results of the M objects;
[0162] A third calculation module 613 is used to extract the classification results of Y classified objects from the classification results of the M objects, and calculate the loss of the classification results of the Y classified objects and the labels corresponding to the Y classified objects;
[0163] An adjustment module 614, configured to perform back propagation on the graph convolutional neural network to be trained according to the loss, so as to adjust the graph convolutional neural network to be trained;
[0164] The iteration module 615 is used to iteratively train the graph convolutional neural network to be trained until the loss converges or reaches a preset number of iterations to obtain a trained graph convolutional neural network.
[0165] It should be noted that the object classification device provided in the embodiment of the present invention can be applied to devices such as mobile phones, monitors, computers, servers, computer peripherals, etc. that can perform object classification.
[0166] The object classification device provided in the embodiment of the present invention can implement each process implemented by the object classification method in the above method embodiment and can achieve the same beneficial effect. To avoid repetition, it will not be described here.
[0167] See also Fig.12 , Fig.12 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.12 As shown, it includes: a memory 1202, a processor 1201, and a computer program stored in the memory 1202 and executable on the processor 1201, wherein:
[0168] The processor 1201 is used to call the computer program stored in the memory 1202 and execute the following steps:
[0169] Acquire image data of N objects, and extract peer relationships of the N objects according to the image data to construct an adjacency matrix;
[0170] Extracting original spatiotemporal data tensors of the N objects according to the image data, wherein the original spatiotemporal data tensors include data of time dimension and space dimension;
[0171] Performing a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension;
[0172] Performing a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time feature of the original spatiotemporal data tensor in the spatial dimension;
[0173] Determine the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features;
[0174] The adjacency matrix and the spatiotemporal data tensor are input into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects.
[0175] Optionally, the processor 1201 performs the convolution operation on the original spatiotemporal data tensor by using a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension, including:
[0176] Obtain K1 pre-trained first convolution kernels, wherein the first convolution kernel includes a first time dimension and a first space dimension, the first time dimension and the time dimension of the original spatiotemporal data tensor are in the same direction, the first space dimension and the space dimension of the original spatiotemporal data tensor are in the same direction, and the first time dimension is smaller than the time dimension of the original spatiotemporal data tensor, and the first space dimension is equal to the space dimension of the original spatiotemporal data tensor;
[0177] Performing a convolution operation on the time dimension of the original spatiotemporal data tensor with a preset convolution step size through the K1 first convolution kernels to obtain K1 spatial features;
[0178] Based on the K1 spatial features, the spatial features of the original spatiotemporal data tensor in the time dimension are calculated.
[0179] Optionally, the processor 1201 performs the convolution operation on the original spatiotemporal data tensor by using a preset second convolution kernel to extract the time feature of the original spatiotemporal data tensor in the spatial dimension, including:
[0180] Obtain pre-trained K2 second convolution kernels, wherein the second convolution kernel includes a second time dimension and a second space dimension, the second time dimension and the time dimension of the original spatiotemporal data tensor are in the same direction, the second space dimension and the space dimension of the original spatiotemporal data tensor are in the same direction, and the second time dimension is equal to the time dimension of the original spatiotemporal data tensor, and the second space dimension is smaller than the space dimension of the original spatiotemporal data tensor;
[0181] Performing a convolution operation on the spatial dimension of the original spatiotemporal data tensor with a preset convolution step size through the K2 second convolution kernels to obtain K2 time features;
[0182] Based on the K2 time features, the time features of the original spatiotemporal data tensor in the spatial dimension are calculated.
[0183] Optionally, the determining of the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features performed by the processor 1201 includes:
[0184] Aligning the spatial feature with the temporal feature in dimension to obtain aligned spatial feature and temporal feature;
[0185] The aligned spatial features and temporal features are concatenated to obtain the spatiotemporal data tensor of the N objects.
[0186] Optionally, the graph convolutional neural network includes at least a pre-trained first weight matrix and a second weight matrix, the first weight matrix and the second weight matrix include category distribution information, and the processor 1201 inputs the adjacency matrix and the spatiotemporal data tensor into the pre-trained graph convolutional neural network for classification prediction to obtain the classification results of the N objects, including:
[0187] Calculate a first result matrix according to the adjacency matrix, the spatiotemporal data tensor and the first weight matrix;
[0188] Calculate a category distribution matrix according to the adjacency matrix, the first result matrix and the second weight matrix;
[0189] According to the category distribution matrix, the classification results of the N objects are determined.
[0190] Optionally, the processor 1201 calculates a first result matrix according to the adjacency matrix, the spatiotemporal data tensor, and the first weight matrix, including:
[0191] Preprocessing the adjacency matrix, and calculating a first result matrix according to the adjacency matrix obtained after the preprocessing, the spatiotemporal data tensor, and the first weight matrix;
[0192] The calculating the category distribution matrix according to the adjacency matrix obtained after preprocessing, the first result matrix and the second weight matrix includes:
[0193] A category distribution matrix is calculated based on the adjacency matrix obtained after preprocessing, the first result matrix, and the second weight matrix.
[0194] Optionally, before inputting the adjacency matrix and the spatiotemporal data tensor into a pre-trained graph convolutional neural network for classification prediction to obtain the classification results of the N objects, the processor 1201 further executes the following steps:
[0195] Obtain sample image data of M objects, and extract peer relationships of the M objects according to the image data to construct an adjacency matrix, wherein the M objects include Y classified objects, M is greater than Y, and the Y classified objects include corresponding labels;
[0196] Extracting original spatiotemporal data tensors of the M objects according to the image data, wherein the original spatiotemporal data tensors include data of time dimension and space dimension;
[0197] Performing a convolution operation on the original spatiotemporal data tensors of the M objects through a preset first convolution kernel to extract spatial features of the original spatiotemporal data tensors of the M objects in the time dimension;
[0198] Performing a convolution operation on the original spatiotemporal data tensors of the M objects through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensors of the M objects in the spatial dimension;
[0199] Determine the spatiotemporal data tensors of the M objects according to the spatial features and the temporal features;
[0200] Inputting the adjacency matrix and the spatiotemporal data tensors of the M objects into the graph convolutional neural network to be trained for classification prediction, and obtaining classification results of the M objects;
[0201] Extracting classification results of Y classified objects from the classification results of the M objects, and calculating the loss of the classification results of the Y classified objects and the labels corresponding to the Y classified objects;
[0202] Performing back propagation on the graph convolutional neural network to be trained according to the loss to adjust the graph convolutional neural network to be trained;
[0203] The graph convolutional neural network to be trained is iteratively trained until the loss converges or reaches a preset number of iterations to obtain a trained graph convolutional neural network.
[0204] It should be noted that the above-mentioned electronic device may be a mobile phone, monitor, computer, server, computer peripheral device, etc. that can be used for object classification.
[0205] The electronic device provided by the embodiment of the present invention can implement each process implemented by the object classification method in the above method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.
[0206] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the object classification method provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0207] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0208] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for object classification, characterized in that: The following steps are involved: Acquire image data of N objects, and extract peer relationships of the N objects according to the image data to construct an adjacency matrix; Extracting original spatiotemporal data tensors of the N objects according to the image data, wherein the original spatiotemporal data tensors include data of time dimension and space dimension; Performing a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension; Performing a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract the time feature of the original spatiotemporal data tensor in the spatial dimension; Determine the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features; Inputting the adjacency matrix and the spatiotemporal data tensor into a preset graph convolutional neural network for classification prediction to obtain classification results of the N objects; the graph convolutional neural network includes at least a pre-trained first weight matrix and a second weight matrix, and the first weight matrix and the second weight matrix include category distribution information; The step of inputting the adjacency matrix and the spatiotemporal data tensor into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects includes: Calculate a first result matrix according to the adjacency matrix, the spatiotemporal data tensor and the first weight matrix; Calculate a category distribution matrix according to the adjacency matrix, the first result matrix and the second weight matrix; According to the category distribution matrix, the classification results of the N objects are determined.
2. The method according to claim 1, characterized in that The step of performing a convolution operation on the original spatiotemporal data tensor by using a preset first convolution kernel to extract the spatial features of the original spatiotemporal data tensor in the time dimension includes: Obtain K1 pre-trained first convolution kernels, wherein the first convolution kernel includes a first time dimension and a first space dimension, the first time dimension and the time dimension of the original spatiotemporal data tensor are in the same direction, the first space dimension and the space dimension of the original spatiotemporal data tensor are in the same direction, and the first time dimension is smaller than the time dimension of the original spatiotemporal data tensor, and the first space dimension is equal to the space dimension of the original spatiotemporal data tensor; Performing a convolution operation on the time dimension of the original spatiotemporal data tensor with a preset convolution step size through the K1 first convolution kernels to obtain K1 spatial features; Based on the K1 spatial features, the spatial features of the original spatiotemporal data tensor in the time dimension are calculated.
3. The method according to claim 2, characterized in that The convolution operation is performed on the original spatiotemporal data tensor by using a preset second convolution kernel to extract the time feature of the original spatiotemporal data tensor in the spatial dimension, including: Obtain K2 pre-trained second convolution kernels, wherein the second convolution kernel includes a second time dimension and a second space dimension, the second time dimension and the time dimension of the original spatiotemporal data tensor are in the same direction, the second space dimension and the space dimension of the original spatiotemporal data tensor are in the same direction, and the second time dimension is equal to the time dimension of the original spatiotemporal data tensor, and the second space dimension is smaller than the space dimension of the original spatiotemporal data tensor; Performing a convolution operation on the spatial dimension of the original spatiotemporal data tensor with a preset convolution step size through the K2 second convolution kernels to obtain K2 time features; Based on the K2 time features, the time features of the original spatiotemporal data tensor in the spatial dimension are calculated.
4. The method according to claim 1, characterized in that Determining the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features includes: Aligning the spatial feature with the temporal feature in dimension to obtain aligned spatial feature and temporal feature; The aligned spatial features and temporal features are concatenated to obtain the spatiotemporal data tensor of the N objects.
5. The method according to claim 1, characterized in that The step of calculating a first result matrix according to the adjacency matrix, the spatiotemporal data tensor and the first weight matrix includes: Preprocessing the adjacency matrix, and calculating a first result matrix according to the adjacency matrix obtained after the preprocessing, the spatiotemporal data tensor, and the first weight matrix; The calculating the category distribution matrix according to the adjacency matrix obtained after preprocessing, the first result matrix and the second weight matrix includes: A category distribution matrix is calculated based on the adjacency matrix obtained after preprocessing, the first result matrix, and the second weight matrix.
6. The method according to claim 1, characterized in that Before inputting the adjacency matrix and the spatiotemporal data tensor into a preset graph convolutional neural network for classification prediction to obtain classification results of the N objects, the method further includes: Obtain sample image data of M objects, and extract peer relationships of the M objects according to the image data to construct an adjacency matrix, wherein the M objects include Y classified objects, M is greater than Y, and the Y classified objects include corresponding labels; Extracting original spatiotemporal data tensors of the M objects according to the image data, wherein the original spatiotemporal data tensors include data of time dimension and space dimension; Performing a convolution operation on the original spatiotemporal data tensors of the M objects through a preset first convolution kernel to extract spatial features of the original spatiotemporal data tensors of the M objects in the time dimension; Performing a convolution operation on the original spatiotemporal data tensors of the M objects through a preset second convolution kernel to extract the time features of the original spatiotemporal data tensors of the M objects in the spatial dimension; Determine the spatiotemporal data tensors of the M objects according to the spatial features and the temporal features; Inputting the adjacency matrix and the spatiotemporal data tensors of the M objects into the graph convolutional neural network to be trained for classification prediction, and obtaining classification results of the M objects; Extracting classification results of Y classified objects from the classification results of the M objects, and calculating the loss of the classification results of the Y classified objects and the labels corresponding to the Y classified objects; Performing back propagation on the graph convolutional neural network to be trained according to the loss to adjust the graph convolutional neural network to be trained; The graph convolutional neural network to be trained is iteratively trained until the loss converges or a preset number of iterations is reached to obtain a trained graph convolutional neural network.
7. An object classification device, characterized in that: The device comprises: A first extraction module is used to obtain image data of N objects, and extract the peer relationship of the N objects according to the image data to construct an adjacency matrix; A second extraction module, configured to extract original spatiotemporal data tensors of the N objects according to the image data, wherein the original spatiotemporal data tensors include data of a time dimension and a space dimension; A third extraction module, configured to perform a convolution operation on the original spatiotemporal data tensor through a preset first convolution kernel to extract spatial features of the original spatiotemporal data tensor in a time dimension; a fourth extraction module, configured to perform a convolution operation on the original spatiotemporal data tensor through a preset second convolution kernel to extract a time feature of the original spatiotemporal data tensor in a spatial dimension; A first processing module, used for determining the spatiotemporal data tensors of the N objects according to the spatial features and the temporal features; A first classification module inputs the adjacency matrix and the spatiotemporal data tensor into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects; the graph convolutional neural network includes at least a pre-trained first weight matrix and a second weight matrix, and the first weight matrix and the second weight matrix include category distribution information; the input of the adjacency matrix and the spatiotemporal data tensor into a pre-trained graph convolutional neural network for classification prediction to obtain classification results of the N objects includes: calculating a first result matrix according to the adjacency matrix, the spatiotemporal data tensor and the first weight matrix; calculating a category distribution matrix according to the adjacency matrix, the first result matrix and the second weight matrix; determining the classification results of the N objects according to the category distribution matrix.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the object classification method according to any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the object classification method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Crowd video analysis method and system
CN106203458A
Peer user identification method and device, server and storage medium
CN111191601A
Peer identification method, device and system, processing server, storage medium and system
CN111209776A
Video classification method and device, storage medium and computer equipment
CN111382616A
