Image feature extraction method for machine vision

Abstract

The invention relates to the technical field of video feature extraction, in particular to an image feature extraction method for machine vision. The method comprises the following steps: according to the position of a bounding box of the same student individual in adjacent frames of student behavior grayscale images and the grayscale distribution difference in the bounding box, obtaining a behavior trajectory index of the student individual; obtaining interaction indexes of the two student individuals according to the gray level change area of the pixel points in the bounding box of the student individuals in the interaction area between any two student individuals; a similarity measurement index is obtained by combining the distance between any two student individuals and the difference of the behavior trajectory indexes and the interaction index, the student individuals in the student behavior grayscale image are clustered by using the similarity measurement index, and the behavior characteristics of the student individuals in the cluster are extracted and analyzed by using the convolutional neural network. According to the method, feature extraction is carried out on the student individuals in the analysis clusters with rich behavior information in the images, and the student behavior feature extraction efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video feature extraction, and in particular to an image feature extraction method for machine vision. Background Art

[0002] By extracting the behavioral features of students from the student behavior images saved in the smart campus platform, we can obtain the students' learning status or exercise status in class, and help education experts conduct research and discussions on students' learning status and physical health.

[0003] Convolutional neural networks are usually used to extract the behavioral features of individual students in campus activity images. Since there are a large number of students in areas such as campus playgrounds during recess, directly extracting the features of the behaviors of all individual students in the entire image results in a large amount of neural network calculations, which reduces the efficiency of feature extraction. Summary of the invention

[0004] In order to solve the technical problem of reducing the efficiency of feature extraction by using convolutional neural networks to extract features of the behaviors of all individual students in an image, the purpose of the present invention is to provide an image feature extraction method for machine vision. The technical solution adopted is as follows: The present invention proposes an image feature extraction method for machine vision, the method comprising: Acquire at least two frames of grayscale images of student behavior; Track and obtain the bounding box of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory index of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box; Obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image; obtaining the interaction index between any two individual students in each frame of the student behavior grayscale image according to the grayscale change trend of the pixel points in the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image; Combining the distance between the bounding boxes of any two individual students in each frame of the student behavior grayscale image, the difference between the behavior trajectory indicators, and the interaction indicator, obtaining a similarity measurement indicator between any two individual students in each frame of the student behavior grayscale image; The individual students in each frame of the student behavior grayscale image are clustered according to the similarity measurement index to obtain different clustering clusters, and the analysis cluster is screened out based on the similarity measurement index between any two individual students in each clustering cluster; the behavioral characteristics of the individual students in the analysis cluster of each frame of the student behavior grayscale image are extracted using a convolutional neural network.

[0005] Furthermore, the method for obtaining the behavior trajectory index of each individual student in each frame of the student behavior grayscale image according to the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box includes: For each individual student in each frame of the student behavior grayscale image, obtain the centroid of the bounding box of the individual student; arrange the grayscale values ​​of the pixel points in the bounding box of the individual student in sequence according to a preset order to obtain a grayscale sequence of the individual student; Calculate the Euclidean distance between the centroid of the bounding box of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image as the moving distance of each student individual in each frame of the student behavior grayscale image; According to the difference between the elements at the same position in the grayscale sequence of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image, and the moving distance of each student individual in each frame of the student behavior grayscale image, the behavior trajectory index of each student individual in each frame of the student movement is obtained.

[0006] Furthermore, the calculation formula of the behavior trajectory index of each individual student in each frame of student movement is as follows: ; In the formula, is the behavior trajectory indicator of the i-th student individual in each frame of the student behavior grayscale image; is the moving distance of the i-th student in each frame of the student behavior grayscale image; V is the total number of elements in the grayscale sequence of each student in each frame of the student behavior grayscale image; is the vth element in the grayscale sequence of the i-th student individual in each frame of the student behavior grayscale image; For the i-th student individual in each frame of student behavior grayscale image, the v-th element in the grayscale sequence corresponding to the student individual in the previous frame of student behavior grayscale image adjacent to each frame of student behavior grayscale image.

[0007] Furthermore, the method for obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image includes: In each frame of the student behavior grayscale image, any two student individuals are selected as analysis individuals, and the line segment between the centroids of the bounding boxes connecting the two analysis individuals is recorded as the analysis segment between the two analysis individuals; the circular area with the midpoint of the analysis segment as the center and half the length of the analysis segment as the radius is taken as the interaction area between the two analysis individuals.

[0008] Furthermore, the method for obtaining the interaction index between any two individual students in each frame of the student behavior grayscale image according to the grayscale change trend of the pixel points in the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image comprises: In each frame of the student behavior grayscale image, the student individual whose pixel point is within the interaction area between any two student individuals is taken as the interaction object of any two student individuals; the line segment between the centroid of the bounding box of each interaction object connecting any two student individuals and the midpoint of the analysis line segment between any two student individuals is recorded as the reference line segment of each interaction object of any two student individuals; the angle between the reference line segment and the preset direction is taken as the reference angle of each interaction object of any two student individuals; The optical flow algorithm is used to obtain the optical flow vector of each pixel in each frame of the student behavior grayscale image; the angle between the optical flow vector of the pixel in the bounding box of the individual student in each frame of the student behavior grayscale image and the preset direction is calculated as the characteristic angle of the corresponding pixel; The interaction index of any two individual students in each frame of the student behavior grayscale image is obtained based on the difference between the characteristic angle of each pixel point in the boundary box of the interaction object of any two individual students in each frame of the student behavior grayscale image and the reference angle.

[0009] Furthermore, the calculation formula of the interaction index of any two individual students in each frame of the student behavior grayscale image is as follows: ; In the formula, is the interaction index of any two individual students in each frame of the student behavior grayscale image; N is the total number of interaction objects of any two individual students in each frame of the student behavior grayscale image; is the total number of pixels in the bounding box of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; is the characteristic angle of the mth pixel point in the bounding box of the nth interaction object of any two student individuals in each frame of the student behavior grayscale image; is the reference angle of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; exp is an exponential function with the natural constant e as the base; cos is the cosine function.

[0010] Furthermore, the method for obtaining a similarity metric between any two individual students in each frame of the student behavior grayscale image includes: In each frame of the student behavior grayscale image, the absolute value of the difference between the behavior trajectory indicators of any two individual students is calculated as the trajectory difference value between any two individual students; According to the Euclidean distance between the centroids of the bounding boxes of any two individual students, and the interaction index and the trajectory difference value of any two individual students, a similarity measurement index between any two individual students in each frame of the student behavior grayscale image is obtained; the trajectory difference value and the interaction index are both positively correlated with the similarity measurement index, and the Euclidean distance is negatively correlated with the similarity measurement index.

[0011] Furthermore, the method of clustering individual students in each frame of the student behavior grayscale image according to the similarity measurement index to obtain different cluster clusters includes: In each frame of the student behavior grayscale image, the similarity metric between each individual student and the other individual students is used as the improved transfer probability from each individual student to the other individual students in each frame of the student behavior grayscale image; The kernel density estimation algorithm is used to obtain the probability density value of individual students in each frame of the student behavior grayscale image, and the density contour map of each frame of the student behavior grayscale image is obtained through the contour generation algorithm; The individual students corresponding to the maximum points of the density contour map are used as seed points, and based on the improved transition probability, the Markov clustering algorithm is used to cluster the individual students in each frame of the student behavior grayscale image to obtain the cluster clusters of each frame of the student behavior grayscale image.

[0012] Furthermore, the method for selecting analysis clusters based on the similarity metric between any two individual students in each cluster includes: For each cluster in the grayscale image of student behavior, the mean of the similarity measurement index of each pair of students in each cluster is normalized and used as the judgment index of each cluster; the cluster corresponding to the judgment index greater than the preset judgment threshold is selected as the analysis cluster.

[0013] Furthermore, the method for tracking and obtaining the bounding box of each individual student in each frame of the student behavior grayscale image includes: A convolutional neural network is used to identify individual students in each frame of student behavior grayscale image; a multi-target tracking method is used to track individual students in at least two frames of behavior grayscale images to obtain the bounding box of each individual student in each frame of student behavior grayscale image.

[0014] The present invention has the following beneficial effects: In an embodiment of the present invention, if a student does not move within a certain period of time, the position of the bounding box of the same individual student in the grayscale image of the student behavior in adjacent frames is the same and the grayscale of the pixels in the bounding box is basically the same. The individual student movement information is analyzed in terms of the position of the individual student's bounding box and the grayscale information in turn, so that the behavior trajectory indicator accurately reflects the student movement information; the grayscale change trend of the pixel points in the bounding box of the individual student presents the displacement of the individual student in time, and the grayscale change trend of the pixel points in the bounding box of the individual student in the interaction area of ​​any two individual students is used to reflect the behavioral interaction of any two individual students, and obtain the interaction index; the difference between the behavior trajectory indicators presents the difference in displacement change of the individual student between the grayscale images of the student behavior in adjacent frames, and the interaction index presents the displacement change difference of the two individual students. The degree of behavioral interaction between two students comprehensively reflects the richness of behavioral information in the interaction area of ​​two students, and the richness of behavioral information in the interaction area is further determined by the distance between the bounding boxes of the two students, so that the similarity measurement index between the two students can accurately present the richness of behavioral feature information in the interaction area of ​​the students; the area with richer behavioral information is the focus of feature extraction, and the behavioral information of the students in the analysis cluster obtained by clustering with similarity measurement index is relatively rich, and the behavioral characteristics of the students in the analysis cluster basically include the behavioral characteristics of the students in the whole image; only the convolutional neural network is used to extract features for the students in the analysis cluster, which can reduce the amount of calculation while ensuring the integrity of the behavioral information in the image, and improve the efficiency of student behavior recognition by the convolutional neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A method flow chart of an image feature extraction method for machine vision provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the image feature extraction method for machine vision proposed by the present invention, its specific implementation, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0019] The specific scenario targeted by the present invention is as follows: the smart campus platform stores photos of different school areas, such as playgrounds, classrooms, etc., extracts features from these photos, and analyzes to obtain student behavior information. The conventional method is to extract features from the target image through a convolutional neural network model and analyze student behavior information. When there are a large number of people performing feature extraction and analysis, there will be too much data causing excessive computation, which will cause a load on the system operation, resulting in low overall system operation efficiency and affecting the stability of the smart campus platform.

[0020] The following is a detailed description of a specific scheme of an image feature extraction method for machine vision provided by the present invention in conjunction with the accompanying drawings.

[0021] See also Figure 1 , which shows a method flow chart of an image feature extraction method for machine vision provided by an embodiment of the present invention, the method comprising: Step S1: Acquire at least two frames of student behavior grayscale images.

[0022] Specifically, high-definition cameras are installed in public places such as playgrounds and classrooms on campus, and continuous video frame images of the campus playground and classrooms during recess are collected by the cameras, and these video frame images are recorded as student behavior images; the collected student behavior images are uploaded to the smart campus platform. In order to improve the accuracy of subsequent analysis results, each frame of student behavior image is grayed and denoised, and image enhancement is performed to obtain each frame of student behavior grayscale image. The smart campus platform uses a convolutional neural network to extract features from the pre-processed student behavior grayscale image and analyze the student behavior feature information. Since the amount of information in the image is too large, that is, the number of students is large, the embodiment of the present invention clusters the students in the image into groups, and reduces the objects of feature extraction to achieve the purpose of reducing the amount of calculation.

[0023] It should be noted that in the embodiment of the present invention, the frame rate of the camera is 30 frames per second, and the continuous frames of student behavior images collected by the camera every ten seconds are used as a group of images for subsequent analysis. The implementer can set it according to the specific situation; the weighted average grayscale algorithm is selected for grayscale processing, Gaussian filtering is used for denoising, and the grayscale histogram algorithm is selected for image enhancement. The specific methods are not introduced here, and they are all technical means well known to those skilled in the art. In other embodiments of the present invention, other image acquisition devices and image preprocessing algorithms can also be selected. Image acquisition and image preprocessing algorithms are both technical means well known to those skilled in the art and are not limited here.

[0024] Step S2: Track and obtain the bounding box of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory indicator of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box.

[0025] Specifically, in the embodiment of the present invention, a convolutional neural network is used to identify individual students in each frame of the student behavior grayscale image; a multi-object tracking technology (Multi-Object Tracking, MOT) is used to track individual students in the student behavior grayscale image of each group of images, and a bounding box of each individual student in each frame of the student behavior grayscale image is obtained. Among them, the convolutional neural network and the multi-object tracking technology are both well-known technologies to those skilled in the art, and will not be described in detail here.

[0026] The multi-target tracking technology tracks and analyzes objects in continuous video frames, and uses the multi-target tracking technology to obtain the bounding box of the individual student in the student behavior grayscale image corresponding to each frame of the student behavior image in each group of images, and the bounding box of the individual student in the student behavior grayscale image corresponding to each frame of the student behavior image in each group of images of the same student corresponds one to one. In this embodiment, the bounding box of the individual student is a rectangle.

[0027] It should be noted that, since the height and weight of the students in the school are similar, and the camera is far away from the student activity area, the sizes of different students in the student behavior grayscale image are similar. Therefore, in the embodiment of the present invention, it is considered that the scales of the bounding boxes of different students in the student behavior grayscale image are the same. In other embodiments of the present invention, the target tracking algorithm based on Fourier change (Minimum Output Sum of Squared Error, MOSSE) can also be used to obtain the bounding box of each student in each frame of the student behavior grayscale image. The algorithm usually uses a fixed-size template to represent the target object, so the scales of the bounding boxes of the individual students are the same.

[0028] If a student does not move within a certain period of time, the position of the bounding box of the same student individual in the grayscale image of the student behavior in adjacent frames is the same, and the grayscale of the pixels in the bounding box is basically the same. The above analysis of the student's individual movement information is carried out from the perspectives of the position of the student's individual bounding box and the grayscale information, so that the behavior trajectory indicator accurately reflects the student's movement information.

[0029] Preferably, the specific method for obtaining the behavior trajectory indicator is: for each individual student in each frame of the student behavior grayscale image, obtain the centroid of the individual student's bounding box; arrange the grayscale values ​​of the pixel points in the individual student's bounding box in sequence according to a preset order to obtain the individual student's grayscale sequence; calculate the Euclidean distance between the centroid of the bounding box of each frame of the student behavior grayscale image and the same individual student in the adjacent previous frame of the student behavior grayscale image as the movement distance of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory indicator of each individual student in each frame of student movement based on the difference between the elements at the same position in the grayscale sequence of the same individual student in each frame of the student behavior grayscale image and the adjacent previous frame of the student behavior grayscale image, as well as the movement distance of each individual student in each frame of the student behavior grayscale image.

[0030] It should be noted that the preset order in the embodiment of the present invention is from left to right and from top to bottom; the grayscale values ​​of the pixels in the bounding box of the individual students are arranged in the order from left to right and from top to bottom to obtain a grayscale sequence, which can be set by the implementer according to the specific situation. This embodiment calculates the Euclidean distance between the centroids of the bounding boxes of two individual students based on the pixel coordinates of the centroids of the bounding boxes of the individual students. Among them, the pixel coordinates are well-known technology and will not be introduced here.

[0031] The calculation formula for the behavior trajectory index of each student in each frame of student movement is as follows: In the formula, is the behavior trajectory indicator of the i-th student in each frame of the student behavior grayscale image; is the moving distance of the i-th student in each frame of the student behavior grayscale image; V is the total number of elements in the grayscale sequence of each student in each frame of the student behavior grayscale image; is the vth element in the grayscale sequence of the i-th student in each frame of the student behavior grayscale image; is the i-th student individual in each frame of student behavior grayscale image, and the v-th element in the grayscale sequence of the corresponding student individual in the adjacent previous frame of student behavior grayscale image of each frame of student behavior grayscale image.

[0032] It should be noted that when the moving distance When the value is larger, the displacement change of the i-th student individual between the grayscale images of the student behavior in adjacent frames is greater, the movement trajectory of the student individual is more obvious, and the behavior trajectory index is The bigger. The grayscale difference of corresponding pixels in the bounding box of the same student individual in the grayscale image of student behavior in adjacent frames is presented. When the value is larger, the grayscale difference of the bounding box of the same student individual in the grayscale images of the student behavior in adjacent frames is greater, and the displacement change of the i-th student individual between the grayscale images of the student behavior in adjacent frames is greater, and the motion trajectory of the student individual is more obvious, and the behavior trajectory index is The bigger.

[0033] Step S3: Obtain the interaction area between any two individual students in each frame of the student behavior grayscale image; obtain the interaction index between any two individual students in each frame of the student behavior grayscale image based on the grayscale change trend of the pixel points within the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image.

[0034] Specifically, when there is strong interaction between two student individuals, the temporal displacement changes of all student individuals in a certain area between the two student individuals are close to the same direction. The interaction area between any two student individuals is obtained in order to analyze the temporal displacement changes of the student individuals in the interaction area.

[0035] Preferably, the method for obtaining the interaction area is: in each frame of the student behavior grayscale image, select any two student individuals as analysis individuals, and record the line segment between the centroids of the bounding boxes connecting the two analysis individuals as the analysis segment between the two analysis individuals; and use the midpoint of the analysis segment as the center and the circular area with half the length of the analysis segment as the radius as the interaction area between the two analysis individuals.

[0036] The grayscale change trend of the pixel points within the bounding box of the student individual presents the displacement of the student individual in time. The grayscale change trend of the pixel points within the bounding box of the student individual in the interaction area between any two student individuals reflects the behavioral interaction between any two student individuals, and then obtains the interaction index of any two student individuals.

[0037] Preferably, the method for obtaining the interaction index is as follows: in each frame of the student behavior grayscale image, the student individual whose pixel point is in the interaction area between any two student individuals is taken as the interaction object of any two student individuals; the line segment between the centroid of the bounding box connecting each interaction object of any two student individuals and the midpoint of the analysis line segment between any two student individuals is recorded as the reference line segment of each interaction object of any two student individuals; the angle between the reference line segment and the preset direction is taken as the reference angle of each interaction object of any two student individuals; the optical flow algorithm is used to obtain the optical flow vector of each pixel point in each frame of the student behavior grayscale image; the angle between the optical flow vector of the pixel point in the bounding box of the student individual in each frame of the student behavior grayscale image and the preset direction is calculated as the characteristic angle of the corresponding pixel point; the interaction index of any two student individuals in each frame of the student behavior grayscale image is obtained according to the difference between the characteristic angle of each pixel point in the bounding box of the interaction object of any two student individuals in each frame of the student behavior grayscale image and the reference angle. Among them, the optical flow algorithm is a well-known technology for those skilled in the art and will not be introduced here.

[0038] It should be noted that in the embodiment of the present invention, the preset direction is the horizontal right direction, and the angle between the optical flow vector of the pixel point and the preset direction is equal to the angle between the optical flow vector and the unit vector The process of obtaining the reference angle of each interaction object of any two student individuals is as follows: the pixel coordinates of the center of mass of each interaction object of any two student individuals are subtracted from the pixel coordinates of the midpoint of the analysis line segment between the two student individuals to obtain the direction vector of each interaction object, and the direction vector is added to the unit vector The angle between is used as the reference angle of each interactive object, and the implementer can set it according to the specific situation. Among them, the pixel coordinate is a well-known technology for those skilled in the art, and will not be described here.

[0039] The calculation formula for the interaction index of any two individual students in each frame of the student behavior grayscale image is as follows: In the formula, is the interaction index of any two individual students in each frame of student behavior grayscale image; N is the total number of interaction objects of any two individual students in each frame of student behavior grayscale image; is the total number of pixels in the bounding box of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; is the characteristic angle of the mth pixel point in the bounding box of the nth interaction object between any two individual students in each frame of the student behavior grayscale image; is the reference angle of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; exp is an exponential function with the natural constant e as the base; cos is the cosine function.

[0040] It should be noted that there is a strong interaction between the two student individuals, and the displacement changes of the interaction objects of the two student individuals in time are close to the same direction. The smaller the value is, the closer the displacement change of the nth interaction object of any two student individuals in time is to the center, the greater the possibility that the two student individuals will participate in the same activity, the stronger the interaction behavior of the two student objects, and the larger the interaction index W. The interaction index is used to predict whether two student individuals are about to have an interaction behavior. For example, when two students are playing basketball, the distance between them is far. If the interaction between the two student individuals is strong, that is, The smaller the distance, the more likely it is that two students can participate in the same activity even though they are far apart.

[0041] Step S4: Combining the distance between the bounding boxes of any two individual students in each frame of the student behavior grayscale image, the difference between the behavior trajectory indicators, and the interaction indicator, obtain the similarity measurement indicator between any two individual students in each frame of the student behavior grayscale image.

[0042] The difference between the behavior trajectory indicators presents the difference in displacement changes between the student behavior grayscale images of adjacent frames. The interaction indicator presents the degree of behavioral interaction between two student individuals, comprehensively reflecting the richness of behavioral information in the interaction area of ​​the two student individuals. The richness of behavioral information in the interaction area is further determined by the distance between the bounding boxes of the two student individuals, and the similarity measurement indicator between the two student individuals is obtained.

[0043] Preferably, the specific method for obtaining the similarity measurement index is: in each frame of the student behavior grayscale image, the absolute value of the difference between the behavior trajectory indicators of any two student individuals is calculated as the trajectory difference value between any two student individuals; based on the Euclidean distance between the centroids of the bounding boxes of any two student individuals, and the interaction index and the trajectory difference value of any two student individuals, the similarity measurement index between any two student individuals in each frame of the student behavior grayscale image is obtained; the trajectory difference value and the interaction index are both positively correlated with the similarity measurement index, and the Euclidean distance is negatively correlated with the similarity measurement index.

[0044] It should be noted that this embodiment calculates the Euclidean distance between the centroids of the bounding boxes of any two individual students based on the pixel coordinates of the centroids of the bounding boxes of any two individual students. Pixel coordinates are well-known technologies and will not be introduced here.

[0045] The specific example of the calculation formula of the similarity metric between any two individual students in each frame of the student behavior grayscale image is as follows: In the formula, is the similarity metric between the ath student individual and the bth student individual in each frame of the student behavior grayscale image; is the behavior trajectory indicator of the ath student in each frame of the student behavior grayscale image; is the behavior trajectory indicator of the bth student in each frame of the student behavior grayscale image; is the interaction index between the ath student individual and the bth student individual in each frame of the student behavior grayscale image; is the Euclidean distance between the centroids of the bounding boxes of the ath student individual and the bth student individual in each frame of the student behavior grayscale image; is the trajectory difference between the ath student and the bth student in each frame of the student behavior grayscale image; is the absolute value function; e is a natural constant.

[0046] In another embodiment of the present invention, the product or sum of the trajectory difference value between any two student individuals and the interaction index can be used as the numerator, and the Euclidean distance between the center of mass of the bounding box of any two student individuals and the bth student individual can be used as the denominator to obtain the ratio as the similarity measurement index between any two student individuals; in other embodiments of the present invention, other basic mathematical operations can be selected to form corresponding positive correlation and negative correlation relationships, which are not limited here.

[0047] It should be noted that when the trajectory difference value When the value is larger, the difference in displacement between the grayscale images of the student behavior of the a-th student and the b-th student in the adjacent frames is greater, indicating that the difference in the behavior trajectories of the two student individuals is more obvious, the behavior information of the two student individuals is richer, and the similarity metric index is When the interaction index When is larger, the possibility of two students participating in the same activity is greater, and the interaction between the two students is stronger; at the same time, the strength of the interaction between the two students is reflected by the interaction objects of the two students; when When the value is larger, when two students participate in the same activity, the difference in the behavior trajectories of the two students is more obvious, which makes the difference in the behavior trajectories of the interaction objects of the two students more obvious, indicating that the behavioral feature information in the interaction area of ​​the two students is richer, and the similarity metric index is The richness of behavioral information in the local area When unchanged, Smaller means the interaction area between two students is smaller, compared to larger interaction area between two students. Bigger, smaller This shows that the richer the behavioral information in the interaction area between two students, the higher the similarity metric index. The larger it is, the richer the behavioral information in the interaction area between two individual students.

[0048] Step S5: Cluster the individual students in each frame of the student behavior grayscale image according to the similarity measurement index to obtain different clustering clusters, and select the analysis cluster based on the similarity measurement index between any two individual students in each clustering cluster; use the convolutional neural network to extract the behavioral characteristics of the individual students in the analysis cluster of each frame of the student behavior grayscale image.

[0049] In each frame of student behavior grayscale image, the similarity metric between each student individual and the rest of the student individuals is used as the improved transfer probability from each student individual to the rest of the student individuals in each frame of student behavior grayscale image; the kernel density estimation algorithm is used to obtain the probability density value of the student individual in each frame of student behavior grayscale image, and the density contour map of each frame of student behavior grayscale image is obtained by the contour generation algorithm; the student individuals corresponding to the maximum points of the density contour map are used as seed points, and based on the improved transfer probability, the Markov clustering algorithm is used to cluster the student individuals in each frame of student behavior grayscale image to obtain the cluster clusters of each frame of student behavior grayscale image. The interaction area between the student individuals corresponding to the maximum points of the density contour map and the rest of the student pairs is rich in behavioral feature information. These student individuals are selected as seed points to start clustering, so that the clustering algorithm converges to a stable clustering result faster, and the richness of behavioral feature information in the interaction area between student individuals in the same cluster is similar.

[0050] It should be noted that, in the process of clustering individual students in the student behavior grayscale image using the Markov clustering algorithm, a transfer matrix is ​​constructed, and each element of the transfer matrix, i.e., the probability that the behavior of one individual student is transferred to another individual student, is equal to the similarity metric between the two individual students. In the embodiment of the present invention, the contour line generation algorithm can select methods such as interpolation and triangulation, among which the kernel density estimation algorithm, the Markov clustering algorithm and the contour line generation algorithm are all well-known technologies to those skilled in the art and will not be repeated here. In other embodiments of the present invention, individual students in each frame of the student behavior grayscale image can also be randomly selected as seed points of the Markov clustering algorithm.

[0051] For each cluster in the grayscale image of student behavior, the result of normalizing the mean of the similarity metric index of each pair of students in each cluster is used as the judgment index of each cluster; the calculation formula of the judgment index of each cluster is as follows: In the formula, P is the judgment index of each cluster; A is the total number of individual students in each cluster; is the similarity metric between the ath student individual and the bth student individual in each frame of the student behavior grayscale image; Norm is the normalization function.

[0052] It should be noted that when the similarity metric When is larger, the local area behavior information of the two individual students is richer, and the larger the judgment index P is, the richer the behavior information in the interaction area of ​​any two individual students in each cluster is.

[0053] The area with richer behavioral information is the focus of feature extraction. The cluster corresponding to the judgment index greater than the preset judgment threshold is selected as the analysis cluster. The behavioral information of individual students in the analysis cluster is relatively rich, and can basically represent the behavioral characteristics of individual students in the entire image. The convolutional neural network is used to extract the behavioral characteristics of individual students in the analysis cluster of each frame of the student behavior grayscale image. While reducing the amount of calculation, the integrity of the behavioral information in the image can be ensured, and the efficiency of the convolutional neural network in extracting student behavior features in the student behavior grayscale image can be effectively improved. In the embodiment of the present invention, the preset judgment threshold takes the empirical value of 0.5, and the implementer can set it according to the specific situation. Among them, using a convolutional neural network to extract feature information of the target object in the image is a well-known technology and will not be repeated here.

[0054] So far, the present invention is completed.

[0055] In summary, in an embodiment of the present invention, the behavior trajectory index of the individual student is obtained according to the difference in the grayscale distribution within the bounding box and the position of the same individual student in the grayscale image of the student behavior in adjacent frames; the interaction index of the two individual students is obtained according to the grayscale change area of ​​the pixel points within the bounding box of the individual student in the interaction area between any two individual students; the similarity metric index is obtained by combining the distance between any two individual students with the difference in the behavior trajectory index and the interaction index, and the similarity metric index is used to cluster the individual students in the grayscale image of the student behavior, and the behavior characteristics of the individual students in the cluster cluster obtained by clustering the grayscale image of the student behavior are extracted using a convolutional neural network. The present invention clusters individual students based on the richness of the behavior information, selects some cluster clusters for behavior feature extraction, and improves the efficiency of extracting student behavior features.

[0056] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0057] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for extracting image features for machine vision, characterized in that: The method includes: Acquire at least two frames of grayscale images of student behavior; Track and obtain the bounding box of each individual student in each frame of the student behavior grayscale image; obtain the behavior trajectory index of each individual student in each frame of the student behavior grayscale image based on the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box; Obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image; obtaining the interaction index between any two individual students in each frame of the student behavior grayscale image according to the grayscale change trend of the pixel points in the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image; Combining the distance between the bounding boxes of any two individual students in each frame of the student behavior grayscale image, the difference between the behavior trajectory indicators, and the interaction indicator, obtaining a similarity measurement indicator between any two individual students in each frame of the student behavior grayscale image; The individual students in each frame of the student behavior grayscale image are clustered according to the similarity measurement index to obtain different clustering clusters, and the analysis cluster is screened out based on the similarity measurement index between any two individual students in each clustering cluster; the behavioral characteristics of the individual students in the analysis cluster of each frame of the student behavior grayscale image are extracted using a convolutional neural network.

2. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for obtaining the behavior trajectory index of each individual student in each frame of the student behavior grayscale image according to the position distribution of the bounding box of the same individual student in each frame of the student behavior grayscale image and its adjacent frame of the student behavior grayscale image and the grayscale difference of the pixels in the bounding box includes: For each individual student in each frame of the student behavior grayscale image, obtain the centroid of the bounding box of the individual student; arrange the grayscale values ​​of the pixel points in the bounding box of the individual student in sequence according to a preset order to obtain a grayscale sequence of the individual student; Calculate the Euclidean distance between the centroid of the bounding box of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image as the moving distance of each student individual in each frame of the student behavior grayscale image; According to the difference between the elements at the same position in the grayscale sequence of the same student individual in each frame of the student behavior grayscale image and its adjacent previous frame of the student behavior grayscale image, and the moving distance of each student individual in each frame of the student behavior grayscale image, the behavior trajectory index of each student individual in each frame of the student movement is obtained.

3. The image feature extraction method for machine vision according to claim 2, characterized in that: The calculation formula of the behavior trajectory index of each student in each frame of student movement is as follows: ; In the formula, is the behavior trajectory indicator of the i-th student individual in each frame of the student behavior grayscale image; is the moving distance of the i-th student in each frame of the student behavior grayscale image; V is the total number of elements in the grayscale sequence of each student in each frame of the student behavior grayscale image; is the vth element in the grayscale sequence of the i-th student individual in each frame of the student behavior grayscale image; For the i-th student individual in each frame of student behavior grayscale image, the v-th element in the grayscale sequence corresponding to the student individual in the previous frame of student behavior grayscale image adjacent to each frame of student behavior grayscale image.

4. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for obtaining the interaction area between any two individual students in each frame of the student behavior grayscale image includes: In each frame of the student behavior grayscale image, any two student individuals are selected as analysis individuals, and the line segment between the centroids of the bounding boxes connecting the two analysis individuals is recorded as the analysis segment between the two analysis individuals; the circular area with the midpoint of the analysis segment as the center and half the length of the analysis segment as the radius is taken as the interaction area between the two analysis individuals.

5. The image feature extraction method for machine vision according to claim 4, characterized in that: The method for obtaining the interaction index of any two individual students in each frame of the student behavior grayscale image according to the grayscale change trend of the pixel points in the boundary box of the individual students in the interaction area between any two individual students in each frame of the student behavior grayscale image comprises: In each frame of the student behavior grayscale image, the student individual whose pixel point is within the interaction area between any two student individuals is taken as the interaction object of any two student individuals; the line segment between the centroid of the bounding box of each interaction object connecting any two student individuals and the midpoint of the analysis line segment between any two student individuals is recorded as the reference line segment of each interaction object of any two student individuals; the angle between the reference line segment and the preset direction is taken as the reference angle of each interaction object of any two student individuals; The optical flow algorithm is used to obtain the optical flow vector of each pixel in each frame of the student behavior grayscale image; the angle between the optical flow vector of the pixel in the bounding box of the individual student in each frame of the student behavior grayscale image and the preset direction is calculated as the characteristic angle of the corresponding pixel; The interaction index of any two individual students in each frame of the student behavior grayscale image is obtained based on the difference between the characteristic angle of each pixel point in the boundary box of the interaction object of any two individual students in each frame of the student behavior grayscale image and the reference angle.

6. The image feature extraction method for machine vision according to claim 5, characterized in that: The calculation formula of the interaction index of any two individual students in each frame of the student behavior grayscale image is as follows: ; In the formula, is the interaction index of any two individual students in each frame of the student behavior grayscale image; N is the total number of interaction objects of any two individual students in each frame of the student behavior grayscale image; is the total number of pixels in the bounding box of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; is the characteristic angle of the mth pixel point in the bounding box of the nth interaction object of any two student individuals in each frame of the student behavior grayscale image; is the reference angle of the nth interaction object between any two student individuals in each frame of the student behavior grayscale image; exp is an exponential function with the natural constant e as the base; cos is the cosine function.

7. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for obtaining a similarity metric between any two individual students in each frame of the student behavior grayscale image comprises: In each frame of the student behavior grayscale image, the absolute value of the difference between the behavior trajectory indicators of any two individual students is calculated as the trajectory difference value between any two individual students; According to the Euclidean distance between the centroids of the bounding boxes of any two individual students, and the interaction index and the trajectory difference value of any two individual students, a similarity measurement index between any two individual students in each frame of the student behavior grayscale image is obtained; the trajectory difference value and the interaction index are both positively correlated with the similarity measurement index, and the Euclidean distance is negatively correlated with the similarity measurement index.

8. The image feature extraction method for machine vision according to claim 1, characterized in that: The method of clustering individual students in each frame of the student behavior grayscale image according to the similarity measurement index to obtain different cluster clusters includes: In each frame of the student behavior grayscale image, the similarity metric between each individual student and the other individual students is used as the improved transfer probability from each individual student to the other individual students in each frame of the student behavior grayscale image; The kernel density estimation algorithm is used to obtain the probability density value of individual students in each frame of the student behavior grayscale image, and the density contour map of each frame of the student behavior grayscale image is obtained through the contour generation algorithm; The individual students corresponding to the maximum points of the density contour map are used as seed points, and based on the improved transition probability, the Markov clustering algorithm is used to cluster the individual students in each frame of the student behavior grayscale image to obtain the cluster clusters of each frame of the student behavior grayscale image.

9. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for selecting analysis clusters based on the similarity metric between any two individual students in each cluster includes: For each cluster in the grayscale image of student behavior, the mean of the similarity measurement index of each pair of students in each cluster is normalized and used as the judgment index of each cluster; the cluster corresponding to the judgment index greater than the preset judgment threshold is selected as the analysis cluster.

10. The image feature extraction method for machine vision according to claim 1, characterized in that: The method for tracking and obtaining the bounding box of each individual student in each frame of the student behavior grayscale image includes: A convolutional neural network is used to identify individual students in each frame of student behavior grayscale image; a multi-target tracking method is used to track individual students in at least two frames of behavior grayscale images to obtain the bounding box of each individual student in each frame of student behavior grayscale image.

Images