A method for image feature extraction for machine vision

Abstract

The present invention relates to the technical field of video feature extraction, and in particular to an image feature extraction method for machine vision. The method obtains the behavior trajectory index of the individual student based on the difference in the position of the bounding box of the same individual student in the grayscale image of the student behavior in adjacent frames and the grayscale distribution in the bounding box; obtains the interaction index of the two individual students based on the grayscale change area of ​​the pixel points in the bounding box of the individual student in the interaction area between any two individual students; obtains the similarity measurement index by combining the difference in the distance between any two individual students and the behavior trajectory index and the interaction index, and uses it to cluster the individual students in the grayscale image of the student behavior, and uses a convolutional neural network to extract the behavior characteristics of the individual students in the analysis cluster. The present invention extracts features from individual students in the analysis cluster that contains relatively rich behavior information in the image, thereby improving the efficiency of extracting student behavior features.

Description

Technical Field

[0001] The present invention relates to the technical field of video feature extraction, and in particular to an image feature extraction method for machine vision. Background Art

[0002] By extracting students' behavioral features from student behavior images stored in the smart campus platform, we can obtain students' learning status or exercise status in class, helping education experts conduct research and discussions on students' learning status and physical health.

[0003] Convolutional neural networks are usually used to extract the behavioral features of individual students in campus activity images. However, since there are a large number of students in areas such as campus playgrounds during recess, directly extracting the behavioral features of all individual students in the entire image results in a large amount of neural network computation and reduces the efficiency of feature extraction. Summary of the Invention

[0004] In order to solve the technical problem of reducing the efficiency of feature extraction when using convolutional neural networks to extract features of all individual student behaviors in an image, the present invention aims to provide an image feature extraction method for machine vision. The technical solution adopted is as follows:

[0005] The present invention proposes an image feature extraction method for machine vision, the method comprising:

[0006] Acquire at least two frames of grayscale images of student behavior;

[0007] Track and obtain the bounding box of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory index of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels within the bounding box;

[0008] Obtaining an interaction area between any two individual students in each frame of the student behavior grayscale image; obtaining an interaction index between any two individual students in each frame of the student behavior grayscale image based on a grayscale change trend of pixel points within a bounding box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image;

[0009] Combining the distance between the bounding boxes of any two individual students in each frame of the student behavior grayscale image, the difference between the behavior trajectory indicators, and the interaction indicator, obtaining a similarity measurement indicator between any two individual students in each frame of the student behavior grayscale image;

[0010] The individual students in each frame of the student behavior grayscale image are clustered according to the similarity measurement index to obtain different clustering clusters, and the analysis cluster is screened out based on the similarity measurement index between any two individual students in each cluster; the behavioral characteristics of the individual students in the analysis cluster of each frame of the student behavior grayscale image are extracted using a convolutional neural network.

[0011] Furthermore, the method for obtaining the behavior trajectory index of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and the grayscale difference of the pixels within the bounding box includes:

[0012] For each student in each frame of the student behavior grayscale image, the centroid of the student's bounding box is obtained; the grayscale values ​​of the pixels in the student's bounding box are arranged in sequence according to a preset order to obtain a grayscale sequence of the student;

[0013] Calculate the Euclidean distance between the centroid of the bounding box of the same student in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image as the movement distance of each student in each frame of the student behavior grayscale image;

[0014] Based on the difference between the elements at the same position in the grayscale sequence of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image, and the movement distance of each student individual in each frame of the student behavior grayscale image, the behavior trajectory index of each student individual in each frame of the student movement is obtained.

[0015] Furthermore, the calculation formula of the behavior trajectory index of each individual student in each frame of student movement is as follows:

[0016] Where, is the behavior trajectory indicator of the i-th student in each frame of the student behavior grayscale image; is the moving distance of the i-th student in each frame of the student behavior grayscale image; V is the total number of elements in the grayscale sequence of each student in each frame of the student behavior grayscale image; is the vth element in the grayscale sequence of the i-th student in each frame of the student behavior grayscale image; For the i-th student individual in each frame of student behavior grayscale image, the v-th element in the grayscale sequence corresponding to the student individual in the adjacent previous frame of student behavior grayscale image of each frame of student behavior grayscale image.

[0017] Furthermore, the method for obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image includes:

[0018] In each frame of the student behavior grayscale image, any two student individuals are selected as analysis individuals, and the line segment between the centroids of the bounding boxes connecting the two analysis individuals is recorded as the analysis segment between the two analysis individuals; the circular area with the midpoint of the analysis segment as the center and half the length of the analysis segment as the radius is used as the interaction area between the two analysis individuals.

[0019] Furthermore, the method for obtaining the interaction index between any two individual students in each frame of the student behavior grayscale image based on the grayscale change trend of the pixel points within the boundary box of the individual student in the interaction area between any two individual students in each frame of the student behavior grayscale image includes:

[0020] In each frame of the student behavior grayscale image, the student individuals whose pixels are within the interaction area between any two student individuals are regarded as the interaction objects of the any two student individuals; the line segment between the centroid of the bounding box of each interaction object of any two student individuals and the midpoint of the analysis line segment between any two student individuals is recorded as the reference line segment of each interaction object of any two student individuals; the angle between the reference line segment and the preset direction is regarded as the reference angle of each interaction object of any two student individuals;

[0021] The optical flow algorithm is used to obtain the optical flow vector of each pixel in each frame of the student behavior grayscale image; the angle between the optical flow vector of the pixel within the bounding box of the individual student in each frame of the student behavior grayscale image and the preset direction is calculated as the characteristic angle of the corresponding pixel;

[0022] The interaction index of any two individual students in each frame of the student behavior grayscale image is obtained based on the difference between the characteristic angle of each pixel point in the boundary box of the interaction object of any two individual students in each frame of the student behavior grayscale image and the reference angle.

[0023] Furthermore, the calculation formula of the interaction index of any two individual students in each frame of the student behavior grayscale image is as follows:

[0024] Where, is the interaction index of any two individual students in each frame of student behavior grayscale image; N is the total number of interaction objects of any two individual students in each frame of student behavior grayscale image; is the total number of pixels within the bounding box of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; The characteristic angle of the mth pixel point in the bounding box of the nth interaction object between any two individual students in each frame of the student behavior grayscale image; is the reference angle of the nth interaction object between any two individual students in each frame of the student behavior grayscale image; exp is an exponential function with the natural constant e as the base; cos is the cosine function.

[0025] Furthermore, the method for obtaining a similarity metric between any two individual students in each frame of the student behavior grayscale image includes:

[0026] In each frame of the student behavior grayscale image, the absolute value of the difference between the behavior trajectory indicators of any two individual students is calculated as the trajectory difference value between the two individual students;

[0027] Based on the Euclidean distance between the centroids of the bounding boxes of any two student individuals, and the interaction index and the trajectory difference value of any two student individuals, a similarity measurement index between any two student individuals in each frame of the student behavior grayscale image is obtained; the trajectory difference value and the interaction index are both positively correlated with the similarity measurement index, and the Euclidean distance is negatively correlated with the similarity measurement index.

[0028] Furthermore, the method of clustering individual students in each frame of the student behavior grayscale image according to the similarity measurement index to obtain different clusters includes:

[0029] In each frame of student behavior grayscale image, the similarity metric between each student individual and the rest of the student individuals is used as the improved transfer probability from each student individual to the rest of the student individuals in each frame of student behavior grayscale image;

[0030] The kernel density estimation algorithm is used to obtain the probability density value of each student in each frame of the student behavior grayscale image, and the density contour map of each frame of the student behavior grayscale image is obtained through the contour generation algorithm;

[0031] The student individuals corresponding to the maximum points of the density contour map are used as seed points. Based on the improved transition probability, the Markov clustering algorithm is used to cluster the student individuals in each frame of the student behavior grayscale image to obtain the cluster clusters of each frame of the student behavior grayscale image.

[0032] Furthermore, the method for selecting analysis clusters based on the similarity metric between any two individual students in each cluster includes:

[0033] For each cluster in the grayscale image of student behavior, the mean of the similarity measurement index of each pair of students in each cluster is normalized and used as the judgment index of each cluster; the cluster corresponding to the judgment index greater than the preset judgment threshold is selected as the analysis cluster.

[0034] Furthermore, the method for tracking and obtaining the bounding box of each student in each frame of the student behavior grayscale image includes:

[0035] A convolutional neural network is used to identify individual students in each frame of student behavior grayscale image; a multi-target tracking method is used to track individual students in at least two frames of behavior grayscale images to obtain the bounding box of each individual student in each frame of student behavior grayscale image.

[0036] The present invention has the following beneficial effects:

[0037] In an embodiment of the present invention, if a student does not move within a certain period of time, the position of the bounding box of the same student individual in the student behavior grayscale image of adjacent frames is the same and the grayscale of the pixels in the bounding box is basically the same. The student individual movement information is analyzed in terms of the position of the student individual's bounding box and the grayscale information, so that the behavior trajectory indicator accurately reflects the student movement information; the grayscale change trend of the pixel points in the bounding box of the student individual presents the displacement of the student individual in time, and the grayscale change trend of the pixel points in the bounding box of the student individual in the interaction area of ​​any two student individuals is used to reflect the behavioral interaction of any two student individuals, and obtain the interaction index; the difference between the behavior trajectory indicators presents the difference in the displacement change of the student individual between the student behavior grayscale images of adjacent frames, and the interaction index presents the two student individuals The degree of behavioral interaction between two students comprehensively reflects the richness of behavioral information in the interaction area of ​​the two students, and the richness of behavioral information in the interaction area is further determined by the distance between the bounding boxes of the two students, so that the similarity measurement index between the two students can accurately present the richness of behavioral feature information in the interaction area of ​​the students; the area with richer behavioral information is the focus of feature extraction, and the behavioral information of the students in the analysis cluster obtained by clustering using similarity measurement index is relatively rich, and the behavioral characteristics of the students in the analysis cluster basically include the behavioral characteristics of the students in the entire image; only the convolutional neural network is used to extract features for the students in the analysis cluster, which can reduce the amount of calculation while ensuring the integrity of the behavioral information in the image, thereby improving the efficiency of student behavior recognition using the convolutional neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flowchart of an image feature extraction method for machine vision provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0040] To further illustrate the technical means and effectiveness of the present invention in achieving its intended objectives, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of an image feature extraction method for machine vision proposed in accordance with the present invention. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0041] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0042] The specific scenario targeted by this invention is that the smart campus platform stores photos of different school areas, such as playgrounds and classrooms during recess, and extracts features from these photos to analyze student behavior information. The conventional method is to extract features from the target images using a convolutional neural network model and analyze student behavior information. When a large number of people are performing feature extraction and analysis, the excessive amount of data will cause excessive computational effort, which will put a load on the system operation, resulting in lower overall system efficiency and affecting the stability of the smart campus platform.

[0043] The following describes in detail a specific solution of an image feature extraction method for machine vision provided by the present invention with reference to the accompanying drawings.

[0044] See also Figure 1 , which shows a method flow chart of an image feature extraction method for machine vision provided by one embodiment of the present invention, the method comprising:

[0045] Step S1: Acquire at least two frames of student behavior grayscale images.

[0046] Specifically, high-definition cameras are installed in public places such as the playground and classrooms on campus, and continuous video frame images of the playground and classroom during recess are collected by the cameras. These video frame images are recorded as student behavior images; the collected student behavior images are uploaded to the smart campus platform. In order to improve the accuracy of subsequent analysis results, each frame of the student behavior image is grayscaled and denoised, and image enhancement processing is performed to obtain a grayscale image of each frame of the student behavior. The smart campus platform uses a convolutional neural network to extract features from the pre-processed student behavior grayscale image and analyze the student behavior feature information. Since the amount of information in the image is too large, that is, the number of students is large, the embodiment of the present invention clusters the students in the image into groups, and reduces the objects of feature extraction to achieve the purpose of reducing the amount of calculation.

[0047] It should be noted that in the embodiment of the present invention, the camera has a frame rate of 30 frames per second. The continuous frames of student behavior images captured by the camera every ten seconds are used as a group of images for subsequent analysis. The implementer can set them according to the specific situation; the weighted average grayscale algorithm is selected for grayscale processing, Gaussian filtering is used for denoising, and the grayscale histogram algorithm is selected for image enhancement. The specific methods are not introduced here, and they are all technical means well known to those skilled in the art. In other embodiments of the present invention, other image acquisition devices and image preprocessing algorithms can also be selected. Image acquisition and image preprocessing algorithms are both technical means well known to those skilled in the art and are not limited here.

[0048] Step S2: Track and obtain the bounding box of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory index of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box.

[0049] Specifically, in this embodiment of the present invention, a convolutional neural network is used to identify individual students within each frame of grayscale images of student behavior. Multi-Object Tracking (MOT) technology is then used to track the individual students within each set of grayscale images, obtaining a bounding box for each individual student within each frame. Both convolutional neural networks and multi-object tracking technology are well known to those skilled in the art and will not be further described here.

[0050] Multi-target tracking technology tracks and analyzes objects in consecutive video frames. It uses this technology to obtain the bounding boxes of individual students in the grayscale image corresponding to each frame of the student behavior image in each image group. The bounding boxes of the same student in the grayscale image corresponding to each frame of the student behavior image in each image group correspond one-to-one. In this embodiment, the bounding boxes of the individual students are rectangular.

[0051] It's important to note that because the height and weight of students in a school are similar, and the camera is far from the student activity area, the sizes of individual students in the student behavior grayscale image are similar. Therefore, in this embodiment of the present invention, the bounding boxes of different individual students in the student behavior grayscale image are assumed to have the same scale. In other embodiments of the present invention, a Fourier transform-based target tracking algorithm (Minimum Output Sum of Squared Error, MOSSE) can also be used to obtain the bounding box of each individual student in each frame of the student behavior grayscale image. This algorithm typically uses a fixed-size template to represent the target object, so the bounding boxes of individual students have the same scale.

[0052] If a student does not move within a certain period of time, the bounding box position of the same student individual in the student behavior grayscale image of adjacent frames is the same, and the pixel grayscale within the bounding box is basically the same. The above analysis of the student individual movement information is carried out from the perspective of the position of the student individual's bounding box and the grayscale information, so that the behavior trajectory indicator accurately reflects the student's movement information.

[0053] Preferably, the specific method for obtaining the behavior trajectory indicator is: for each individual student in each frame of the student behavior grayscale image, obtain the centroid of the individual student's bounding box; arrange the grayscale values ​​of the pixel points in the individual student's bounding box in sequence according to a preset order to obtain the grayscale sequence of the individual student; calculate the Euclidean distance between the centroid of the bounding box of each frame of the student behavior grayscale image and the same individual student in the adjacent previous frame of the student behavior grayscale image as the movement distance of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory indicator of each individual student in each frame of student movement based on the difference between the elements at the same position in the grayscale sequence of the same individual student in each frame of the student behavior grayscale image and the adjacent previous frame of the student behavior grayscale image, as well as the movement distance of each individual student in each frame of the student behavior grayscale image.

[0054] It should be noted that the preset order in this embodiment of the present invention is from left to right and from top to bottom; the grayscale values ​​of the pixels within the bounding boxes of the individual students are arranged in this order from left to right and from top to bottom to obtain a grayscale sequence, which can be customized by the implementer according to the specific situation. This embodiment calculates the Euclidean distance between the centroids of the bounding boxes of two individual students based on the pixel coordinates of the centroids of the bounding boxes of the individual students. Pixel coordinates are well known in the art and will not be described here.

[0055] The calculation formula for the behavior trajectory index of each student in each frame of student movement is as follows:

[0056]

[0057] Where, is the behavior trajectory indicator of the i-th student in each frame of the student behavior grayscale image; is the moving distance of the i-th student in each frame of the student behavior grayscale image; V is the total number of elements in the grayscale sequence of each student in each frame of the student behavior grayscale image; is the vth element in the grayscale sequence of the i-th student in each frame of the student behavior grayscale image; For the i-th student individual in each frame of student behavior grayscale image, the v-th element in the grayscale sequence of the corresponding student individual in the adjacent previous frame of student behavior grayscale image of each frame of student behavior grayscale image.

[0058] It should be noted that when the moving distance When the value is larger, the displacement change of the i-th student individual between the grayscale images of the student behavior in adjacent frames is greater, the movement trajectory of the student individual is more obvious, and the behavior trajectory index is The bigger. The grayscale difference of the corresponding pixels in the bounding box of the same student individual in the grayscale image of the student behavior in adjacent frames is presented. When the value is larger, the grayscale difference of the bounding box of the same student individual in the adjacent frames of the student behavior grayscale image is greater, and the displacement change of the i-th student individual between the adjacent frames of the student behavior grayscale image is greater, and the motion trajectory of the student individual is more obvious, and the behavior trajectory index is The bigger.

[0059] Step S3: Obtain the interaction area between any two student individuals in each frame of the student behavior grayscale image; obtain the interaction index of any two student individuals in each frame of the student behavior grayscale image based on the grayscale change trend of the pixel points within the boundary box of the student individuals in the interaction area between any two student individuals in each frame of the student behavior grayscale image.

[0060] Specifically, when there is strong interaction between two student individuals, the temporal displacement changes of all student individuals in a certain area between the two student individuals are close to the same direction. The interaction area between any two student individuals is obtained in order to analyze the temporal displacement changes of the student individuals in the interaction area.

[0061] Preferably, the method for obtaining the interaction area is: in each frame of the student behavior grayscale image, any two student individuals are selected as analysis individuals, and the line segment between the centroids of the bounding boxes connecting the two analysis individuals is recorded as the analysis segment between the two analysis individuals; the circular area with the midpoint of the analysis segment as the center and half the length of the analysis segment as the radius is used as the interaction area between the two analysis individuals.

[0062] The grayscale change trend of the pixel points within the bounding box of the student individual shows the displacement of the student individual in time. The grayscale change trend of the pixel points within the bounding box of the student individual in the interaction area between any two student individuals reflects the behavioral interaction between any two student individuals, and then obtains the interaction index of any two student individuals.

[0063] Preferably, the method for obtaining the interaction index is as follows: in each frame of the student behavior grayscale image, the student individual whose pixel point is within the interaction area between any two student individuals is regarded as the interaction object of any two student individuals; the line segment between the centroid of the bounding box connecting each interaction object of any two student individuals and the midpoint of the analysis line segment between any two student individuals is recorded as the baseline line segment of each interaction object of any two student individuals; the angle between the baseline line segment and the preset direction is regarded as the baseline angle of each interaction object of any two student individuals; the optical flow algorithm is used to obtain the optical flow vector of each pixel point in each frame of the student behavior grayscale image; the angle between the optical flow vector of the pixel point in the bounding box of the student individual in each frame of the student behavior grayscale image and the preset direction is calculated as the characteristic angle of the corresponding pixel point; the interaction index of any two student individuals in each frame of the student behavior grayscale image is obtained based on the difference between the characteristic angle of each pixel point in the bounding box of the interaction object of any two student individuals in each frame of the student behavior grayscale image and the baseline angle. Among them, the optical flow algorithm is a well-known technology for those skilled in the art and will not be introduced here.

[0064] It should be noted that, in the embodiment of the present invention, the preset direction is the horizontal right direction, and the angle between the optical flow vector of the pixel point and the preset direction is equal to the angle between the optical flow vector and the unit vector The process of obtaining the reference angle of each interaction object of any two student individuals is as follows: the pixel coordinates of the center of mass of each interaction object of any two student individuals are subtracted from the pixel coordinates of the midpoint of the analysis line segment between the two student individuals to obtain the direction vector of each interaction object, and the direction vector is added to the unit vector The angle between them is used as the reference angle of each interactive object, and the implementer can set it according to the specific situation. Among them, the pixel coordinates are well known to those skilled in the art and will not be described in detail here.

[0065] The calculation formula for the interaction index of any two students in each frame of the student behavior grayscale image is as follows:

[0066]

[0067] Where, is the interaction index of any two individual students in each frame of student behavior grayscale image; N is the total number of interaction objects of any two individual students in each frame of student behavior grayscale image; is the total number of pixels within the bounding box of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; is the feature angle of the mth pixel point in the bounding box of the nth interaction object between any two individual students in each frame of the student behavior grayscale image; is the reference angle of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; exp is the exponential function with the natural constant e as the base; cos is the cosine function.

[0068] It should be noted that there is a strong interaction between the two student individuals, and the displacement changes of the interaction objects of the two student individuals in time are close to the same direction. The smaller the value is, the closer the displacement change of the nth interaction object of any two student individuals in time is to the center, the greater the possibility that the two student individuals will participate in the same activity, the stronger the interaction behavior of the two student objects, and the larger the interaction index W. The interaction index is used to predict whether two student individuals are about to have an interaction behavior. For example, when two students are playing basketball, the distance between them is far. If the interaction between the two student individuals is strong, The smaller the distance, the more likely it is that two students can participate in the same activity even though they are far apart.

[0069] Step S4: Combining the distance between the bounding boxes of any two individual students in each frame of the student behavior grayscale image, the difference between the behavior trajectory indicators, and the interaction indicator, obtain the similarity measurement indicator between any two individual students in each frame of the student behavior grayscale image.

[0070] The difference between the behavioral trajectory indicators presents the difference in displacement changes between the grayscale images of student behavior in adjacent frames. The interaction indicator presents the degree of behavioral interaction between two student individuals, comprehensively reflecting the richness of behavioral information in the interaction area of ​​the two student individuals. The richness of behavioral information in the interaction area is further determined by the distance between the bounding boxes of the two student individuals, and the similarity measurement indicator between the two student individuals is obtained.

[0071] Preferably, the specific method for obtaining the similarity measurement index is: in each frame of the student behavior grayscale image, calculate the absolute value of the difference between the behavior trajectory indicators of any two student individuals as the trajectory difference value between any two student individuals; based on the Euclidean distance between the centroids of the bounding boxes of any two student individuals, and the interaction index and trajectory difference value of any two student individuals, obtain the similarity measurement index between any two student individuals in each frame of the student behavior grayscale image; the trajectory difference value and the interaction index are positively correlated with the similarity measurement index, and the Euclidean distance is negatively correlated with the similarity measurement index.

[0072] It should be noted that this embodiment calculates the Euclidean distance between the centroids of the bounding boxes of any two individual students based on the pixel coordinates of the centroids of the bounding boxes of the two individual students. Pixel coordinates are well known in the art and will not be described here.

[0073] The specific example of the calculation formula for the similarity metric between any two individual students in each frame of the student behavior grayscale image is as follows:

[0074]

[0075] Where, is the similarity metric between the ath student individual and the bth student individual in each frame of the student behavior grayscale image; is the behavior trajectory indicator of the ath student in each frame of the student behavior grayscale image; is the behavior trajectory indicator of the bth student in each frame of the student behavior grayscale image; is the interaction index between the ath student individual and the bth student individual in each frame of the student behavior grayscale image; is the Euclidean distance between the centroids of the bounding boxes of the ath student individual and the bth student individual in each frame of the student behavior grayscale image; is the trajectory difference between the ath student and the bth student in each frame of the student behavior grayscale image; is the absolute value function; e is a natural constant.

[0076] In another embodiment of the present invention, the product or sum of the trajectory difference value between any two student individuals and the interaction index can be used as the numerator, and the Euclidean distance between the center of mass of the bounding box of any two student individuals and the b-th student individual can be used as the denominator to obtain the ratio as the similarity measurement index between any two student individuals; in other embodiments of the present invention, other basic mathematical operations can be selected to form corresponding positive correlation and negative correlation relationships, which are not limited here.

[0077] It should be noted that when the trajectory difference value When the value is larger, the difference in displacement between the grayscale images of the student behavior of the a-th student and the b-th student in adjacent frames is greater, indicating that the difference in the behavior trajectories of the two student individuals is more obvious, the behavioral information of the two student individuals is richer, and the similarity metric is more accurate. The larger the interaction index When the value is larger, the possibility of two students participating in the same activity is greater, and the interaction between the two students is stronger. At the same time, the strength of the interaction between the two students is reflected by the interaction objects of the two students. When the value is larger, when two students participate in the same activity, the difference in the behavioral trajectories of the two students is more obvious, which makes the difference in the behavioral trajectories of the interaction objects of the two students more obvious, indicating that the behavioral feature information in the interaction area of ​​the two students is richer, and the similarity metric index is The richness of behavioral information in the local area When unchanged, Smaller means the interaction area between two students is smaller, compared with larger interaction area between two students. Larger, smaller This shows that the richer the behavioral information in the interaction area between two students, the higher the similarity metric The larger it is, the richer the behavioral information in the interaction area between two individual students.

[0078] Step S5: Cluster the individual students in each frame of the student behavior grayscale image based on the similarity measurement index to obtain different clusters, and select the analysis cluster based on the similarity measurement index between any two individual students in each cluster; use the convolutional neural network to extract the behavioral characteristics of the individual students in the analysis cluster of each frame of the student behavior grayscale image.

[0079] In each frame of the student behavior grayscale image, the similarity metric between each individual student and the remaining individual students is used as the improved transition probability from each individual student to the remaining individual students in each frame of the student behavior grayscale image. A kernel density estimation algorithm is used to obtain the probability density of individual students in each frame of the student behavior grayscale image, and a density contour map of each frame of the student behavior grayscale image is generated using a contour generation algorithm. The individual students corresponding to the maximum points of the density contour map are used as seed points, and based on the improved transition probability, a Markov clustering algorithm is used to cluster the individual students in each frame of the student behavior grayscale image, resulting in clusters for each frame of the student behavior grayscale image. The interaction regions between the individual students corresponding to the maximum points of the density contour map and the remaining pairs of students have relatively rich behavioral feature information. These students are selected as seed points to initiate clustering, enabling the clustering algorithm to converge to a stable clustering result more quickly and ensuring that the richness of behavioral feature information in the interaction regions between individual students within the same cluster is relatively similar.

[0080] It should be noted that, when clustering individual students in the student behavior grayscale image using the Markov clustering algorithm, a transfer matrix is ​​constructed. Each element of the transfer matrix, i.e., the probability that the behavior of one student individual is transferred to another student individual, is equal to the similarity metric between the two student individuals. In the embodiment of the present invention, the contour line generation algorithm can use methods such as interpolation and triangulation. Among them, the kernel density estimation algorithm, the Markov clustering algorithm and the contour line generation algorithm are all well-known technologies to those skilled in the art and will not be described in detail here. In other embodiments of the present invention, individual students in each frame of the student behavior grayscale image can also be randomly selected as seed points for the Markov clustering algorithm.

[0081] For each cluster in the grayscale image of student behavior, the normalized mean of the similarity metrics of each pair of students in each cluster is used as the judgment index of each cluster. The calculation formula of the judgment index of each cluster is as follows:

[0082]

[0083] Where, P is the judgment index of each cluster; A is the total number of individual students in each cluster; is the similarity metric between the ath student individual and the bth student individual in each frame of the student behavior grayscale image; Norm is the normalization function.

[0084] It should be noted that when the similarity metric When is larger, the local area behavior information of the two individual students is richer, and the larger the judgment index P is, the richer the behavior information in the interaction area of ​​any two individual students in each cluster is.

[0085] The area with richer behavioral information is the focus of feature extraction. The cluster corresponding to the judgment index greater than the preset judgment threshold is selected as the analysis cluster. The behavioral information of individual students in the analysis cluster is relatively rich and can basically represent the behavioral characteristics of individual students in the entire image. The behavioral characteristics of individual students in the analysis cluster of each frame of the student behavior grayscale image are extracted using a convolutional neural network. While reducing the amount of calculation, the integrity of the behavioral information in the image can be ensured, and the efficiency of the convolutional neural network in extracting student behavior features from the student behavior grayscale image can be effectively improved. In the embodiment of the present invention, the preset judgment threshold takes the empirical value of 0.5, and the implementer can set it according to the specific situation. Among them, the use of a convolutional neural network to extract the feature information of the target object in the image is a well-known technology and will not be repeated here.

[0086] So far, the present invention is completed.

[0087] In summary, in an embodiment of the present invention, the behavior trajectory index of the student individual is obtained based on the position of the bounding box of the same student individual in the student behavior grayscale image of adjacent frames and the difference in the grayscale distribution within the bounding box; the interaction index of the two student individuals is obtained based on the grayscale change area of ​​the pixel points within the bounding box of the student individual in the interaction area between any two student individuals; the similarity measurement index is obtained by combining the distance between any two student individuals with the difference in the behavior trajectory index and the interaction index, and the similarity measurement index is used to cluster the student individuals in the student behavior grayscale image, and the convolutional neural network is used to extract the behavior characteristics of the student individuals in the cluster cluster obtained by clustering the student behavior grayscale image. The present invention clusters the student individuals based on the richness of the behavior information, selects some cluster clusters for behavior feature extraction, and improves the efficiency of student behavior feature extraction.

[0088] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A method for extracting image features for machine vision, characterized in that: The method includes: Acquire at least two frames of grayscale images of student behavior; Track and obtain the bounding box of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory index of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box; Obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image; obtaining the interaction index between any two individual students in each frame of the student behavior grayscale image according to the grayscale change trend of the pixel points in the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image; Combining the distance between the bounding boxes of any two individual students in each frame of the student behavior grayscale image, the difference between the behavior trajectory indicators, and the interaction indicator, obtaining a similarity measurement indicator between any two individual students in each frame of the student behavior grayscale image; Clustering individual students in each frame of the student behavior grayscale image according to the similarity measurement index to obtain different clustering clusters, and screening out analysis clusters based on the similarity measurement index between any two individual students in each clustering cluster; using a convolutional neural network to extract the behavior characteristics of individual students in the analysis cluster of each frame of the student behavior grayscale image; The calculation formula of the interaction index of any two individual students in each frame of the student behavior grayscale image is as follows: ; In the formula, is the interaction index of any two individual students in each frame of the student behavior grayscale image; N is the total number of interaction objects of any two individual students in each frame of the student behavior grayscale image; is the total number of pixels in the bounding box of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; is the characteristic angle of the mth pixel point in the bounding box of the nth interaction object between any two individual students in each frame of the student behavior grayscale image; is the reference angle of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; exp is an exponential function with the natural constant e as the base; cos is the cosine function.

2. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for obtaining the behavior trajectory index of each individual student in each frame of the student behavior grayscale image according to the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box includes: For each individual student in each frame of the student behavior grayscale image, obtain the centroid of the bounding box of the individual student; arrange the grayscale values ​​of the pixel points in the bounding box of the individual student in sequence according to a preset order to obtain a grayscale sequence of the individual student; Calculate the Euclidean distance between the centroid of the bounding box of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image as the moving distance of each student individual in each frame of the student behavior grayscale image; According to the difference between the elements at the same position in the grayscale sequence of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image, and the moving distance of each student individual in each frame of the student behavior grayscale image, the behavior trajectory index of each student individual in each frame of the student movement is obtained.

3. The image feature extraction method for machine vision according to claim 2, characterized in that: The calculation formula of the behavior trajectory index of each student in each frame of student movement is as follows: ; In the formula, is the behavior trajectory indicator of the i-th student individual in each frame of the student behavior grayscale image; is the moving distance of the i-th student in each frame of the student behavior grayscale image; V is the total number of elements in the grayscale sequence of each student in each frame of the student behavior grayscale image; is the vth element in the grayscale sequence of the i-th student individual in each frame of the student behavior grayscale image; For the i-th student individual in each frame of student behavior grayscale image, the v-th element in the grayscale sequence corresponding to the student individual in the previous frame of student behavior grayscale image adjacent to each frame of student behavior grayscale image.

4. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image includes: In each frame of the student behavior grayscale image, any two student individuals are selected as analysis individuals, and the line segment between the centroids of the bounding boxes connecting the two analysis individuals is recorded as the analysis segment between the two analysis individuals; the circular area with the midpoint of the analysis segment as the center and half the length of the analysis segment as the radius is taken as the interaction area between the two analysis individuals.

5. The image feature extraction method for machine vision according to claim 4, characterized in that: The method for obtaining the interaction index of any two individual students in each frame of the student behavior grayscale image according to the grayscale change trend of the pixel points in the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image comprises: In each frame of the student behavior grayscale image, the student individual whose pixel point is within the interaction area between any two student individuals is taken as the interaction object of any two student individuals; the line segment between the centroid of the bounding box of each interaction object connecting any two student individuals and the midpoint of the analysis line segment between any two student individuals is recorded as the reference line segment of each interaction object of any two student individuals; the angle between the reference line segment and the preset direction is taken as the reference angle of each interaction object of any two student individuals; The optical flow algorithm is used to obtain the optical flow vector of each pixel in each frame of the student behavior grayscale image; the angle between the optical flow vector of the pixel in the bounding box of the individual student in each frame of the student behavior grayscale image and the preset direction is calculated as the characteristic angle of the corresponding pixel; The interaction index of any two individual students in each frame of the student behavior grayscale image is obtained based on the difference between the characteristic angle of each pixel point in the boundary box of the interaction object of any two individual students in each frame of the student behavior grayscale image and the reference angle.

6. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for obtaining a similarity metric between any two individual students in each frame of the student behavior grayscale image comprises: In each frame of the student behavior grayscale image, the absolute value of the difference between the behavior trajectory indicators of any two individual students is calculated as the trajectory difference value between any two individual students; According to the Euclidean distance between the centroids of the bounding boxes of any two individual students, and the interaction index and the trajectory difference value of any two individual students, a similarity measurement index between any two individual students in each frame of the student behavior grayscale image is obtained; the trajectory difference value and the interaction index are both positively correlated with the similarity measurement index, and the Euclidean distance is negatively correlated with the similarity measurement index.

7. The image feature extraction method for machine vision according to claim 1, characterized in that: The method of clustering individual students in each frame of the student behavior grayscale image according to the similarity measurement index to obtain different cluster clusters includes: In each frame of the student behavior grayscale image, the similarity metric between each individual student and the other individual students is used as the improved transfer probability from each individual student to the other individual students in each frame of the student behavior grayscale image; The kernel density estimation algorithm is used to obtain the probability density value of individual students in each frame of the student behavior grayscale image, and the density contour map of each frame of the student behavior grayscale image is obtained through the contour generation algorithm; The individual students corresponding to the maximum points of the density contour map are used as seed points, and based on the improved transition probability, the Markov clustering algorithm is used to cluster the individual students in each frame of the student behavior grayscale image to obtain the cluster clusters of each frame of the student behavior grayscale image.

8. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for selecting analysis clusters based on the similarity metric between any two individual students in each cluster includes: For each cluster in the grayscale image of student behavior, the mean of the similarity measurement index of each pair of students in each cluster is normalized and used as the judgment index of each cluster; the cluster corresponding to the judgment index greater than the preset judgment threshold is selected as the analysis cluster.

9. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for tracking and obtaining the bounding box of each individual student in each frame of the student behavior grayscale image includes: A convolutional neural network is used to identify individual students in each frame of student behavior grayscale image; a multi-target tracking method is used to track individual students in at least two frames of behavior grayscale images to obtain the bounding box of each individual student in each frame of student behavior grayscale image.

Images