Mouse characteristic behavior analysis method based on scene geometric constraints and deep learning

Abstract

A method for analyzing characteristic behavior of experimental mice based on scene geometric constraints and deep learning belongs to the field of machine vision technology, including configuring markers, extracting scene information, and constructing a structured scene space; collecting mouse video data and creating a data set; constructing a mouse semantic posture model with semantic shaping points; selecting a deep neural network, training and learning, and extracting dynamic information of the mouse semantic shaping points; based on the view geometry principle, mapping the mouse semantic shaping points of the video image to the real scene to achieve regression positioning of the shaping points; using a density clustering algorithm to analyze the regression-located shaping points and lock the key sequence frames; in different plane spaces, using field geometric constraints, analyzing the characteristics and differences of the behavior. Finally, through a structured spatial scene, scene geometric constraints are established to transform complex and high-dimensional behaviors into low-dimensional physical quantities, and non-interference fine-grained recognition and quantitative analysis of the behavior of experimental mice are achieved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning. Background Art

[0002] The characteristic behavior of mice is used to reveal the potential strategies of the brain to adapt to the environment and capture changes caused by genetic or neural manipulation. It is one of the behavioral methods to evaluate learning and memory ability, an indicator to measure medication, emotions, etc., and an important part of rodent behavior and pharmacology research. It plays an irreplaceable role and has broad research prospects and guiding significance. A recent trend in the study of laboratory mouse behavior is to capture activities and behaviors in a natural environment. However, the study of the characteristic behavior of laboratory mice is generally conducted through human eye observation or by attaching markers to mice, implanting sensors, and using infrared devices for measurement. These methods affect the normal activities of mice to a certain extent, keeping the movement of mice in an unnatural state. The measurement method of applying machine vision combined with scene geometric constraints can not only avoid interference with the experimental environment, but also keep the mice in a natural state of movement due to the non-intervention of scene construction. It can not only monitor the behavior of mice without contact and automatically through visual methods, but also identify the behavior of mice through scene geometric constraints, quantify the behavioral parameters and health status of mice, improve safety, and reduce labor costs. Summary of the invention

[0003] According to the technical problems raised above, a method for analyzing characteristic behavior of mice based on scene geometric constraints and deep learning is provided. The present invention mainly utilizes video analysis technology combined with scene geometric constraints to quantify the behavior of mice, so as to further analyze the behavioral changes of mice in multiple dimensions and achieve timely monitoring, so as to meet the laboratory's use of upright behavior as a measurement indicator in various studies on mice.

[0004] The technical means adopted by the present invention are as follows:

[0005] A method for analyzing characteristic behavior of mice based on scene geometric constraints and deep learning is provided, characterized in that it includes:

[0006] S1, analyze scene features, extract scene information, assist virtual markers, and construct scene space;

[0007] S2, collect video data of mouse movement, create a data set, analyze the semantic shaping points of mice, and build a semantic posture model of mice;

[0008] S3, select a deep neural network and use the data set for learning and training to extract the dynamic information of the mouse semantic image points;

[0009] S4. Based on the principle of single-view projective geometry in computer vision, the semantic shaping points of the experimental mouse in the image plane of the video sequence are mapped to the real scene to achieve regression positioning of the shaping points in the real scene;

[0010] S5. Use the density clustering algorithm to perform cluster analysis on the modeling points located by regression, and lock the subsequence frames of key motion behaviors.

[0011] S6. For the key behavior sequence frames, clarify the position information of the shaping points in different spatial planes, use the scene geometric constraints to analyze the position information of the shaping points in different planes, highlight the correlation between the shaping points and the scene geometric constraints in different spaces and the geometric characteristics between the shaping points, and analyze their characteristics and differences. Finally, determine the characteristic behavior and conduct quantitative analysis.

[0012] S7. Using the above research methods, establish strong geometric homography constraints in plane images and real scenes, qualitatively analyze the transformation of shaping points between different planes, highlight the differences in changes of different shaping points, determine upright behavior, and quantitatively analyze behavioral parameters such as the time, duration, and frequency of the behavior. Taking a homography constraint in scene geometric constraints as an example, clarify the shaping points on different planes, establish strong geometric homography constraints on the bottom surface of the image and the real scene, and highlight the changes that occur when the shaping points intersect the bottom surface, as well as highlight the differences in changes between different shaping points. Establish combined discrimination conditions, determine it as upright characteristic behavior, and quantitatively analyze behavioral parameters such as the time, duration, and frequency of upright behavior.

[0013] Furthermore, the scene features are analyzed, scene information is extracted, auxiliary markers are used, and scene space is constructed. According to the structural features of the experimental open field itself, the corner points and linear structural features of the open field are extracted; the position of the coding calibration plate is used to establish virtual structural features so that it can be superimposed on the real scene to normalize the scene space. Specifically, the structural features of the experimental open field are analyzed to extract the geometric structure of the open field; the markers are coded, including the selection and position arrangement of the markers, and the virtual structural information of the space is added so that it can be superimposed on the real scene to construct and supplement the geometric features of the scene. The positioning points of the scene include the structural features of the experimental open field itself and the added virtual structural features, and generally the corner points are used as positioning points.

[0014] Furthermore, the method collects video data of mouse movement, creates a data set, analyzes the semantic shaping points of the mouse, and constructs a semantic posture model of the mouse. Specifically, based on scene structuring, based on video data collection, adjusts camera parameters, fixes camera position and viewing angle, and performs calibration processing on the camera to monitor the movement of the mouse; detects the moving mouse based on the background difference method of the mixed Gaussian model, separates the foreground and background, and prepares the mouse data set using a segmentation algorithm; analyzes the movement characteristics of the mouse, replaces body parts with semantic shaping points, and constructs a semantic posture model of the mouse.

[0015] On the basis of scene structuring, based on video data acquisition, camera parameters are adjusted and the camera position and viewing angle are fixed, camera calibration is processed, and moving targets are monitored; video data acquisition of mice includes the use of a color camera, a bracket and a computer, the camera is mounted on the bracket, the camera is arranged at a certain angle to the horizontal direction, and the camera is arranged on the side of the open field for shooting, so as to achieve recording in the entire range of the open field; the computer is used to control the camera's shooting, recording, storage, lighting control, and data processing. The camera calibration uses a calibration plate to perform camera calibration processing to eliminate the distortion effect of the camera.

[0016] The segmentation algorithm segments mice and prepares a data set of mice, including: using a mixed Gaussian model to model the background in the video to achieve background elimination, generate a binary image of the mouse, extract the outline of the mouse and calculate its center of mass, generate a rectangular ROI area, and segment the binary image to obtain a part that only contains the rectangular ROI area to prepare a sub-image data set of the mouse.

[0017] The background difference method based on the mixed Gaussian model is used to detect the moving target mouse, and the moving target is segmented using the segmentation algorithm to prepare a mouse data set. The mixed Gaussian model is a background representation method based on the statistical information of pixel samples. It uses statistical information such as the probability density of a large number of sample values ​​of pixels over a long period of time to represent the background, and then uses statistical differences to judge the target pixel. K Gaussian models are established for each pixel in the image. The probability density function of the pixel point X at time t is:

[0018]

[0019]

[0020] Where: η represents a single Gaussian probability density function;

[0021] W n,t Represents the weight of the nth Gaussian model at time t, where 0≤W n,t ≤1;

[0022] μ n,tRepresents the mean vector of the nth Gaussian model at time t;

[0023] τ n,t represents the covariance vector of the nth Gaussian model at time t, where μ n,t =σ n,t I (I is the identity matrix), σ n,t is the standard deviation of the nth Gaussian model at time t;

[0024] K represents the number of Gaussian models.

[0025] First, the background is trained to initialize the parameters in the Gaussian model and find the corresponding preliminary parameters. Then, each pixel of each frame is calculated to see if it matches a certain model. If the value of the pixel is within 3 times the mean of its model, it matches, and it is included in the model and the parameters of the model are updated. If it does not match, a new Gaussian model is established with the pixel, the parameters are initialized, and the least likely model in the original model is replaced. Finally, each new pixel value X is compared with each mean of the current K models. If the difference is within 2 times the variance, it is considered to be the background, otherwise it is considered to be the foreground. Modeling of the foreground is achieved, the background is eliminated, a binary image of the mouse is generated, the outline of the mouse is extracted and its center of mass is calculated, a rectangular ROI area is generated, and the part containing only the rectangular ROI area is segmented in the binary image to form a sub-image data set of the mouse.

[0026] Furthermore, the method of replacing body parts with semantic shaping points to construct a semantic posture model of the mouse includes:

[0027] According to the morphological characteristics and movement characteristics of mice, the key body parts of mice are marked and annotated as semantic shaping points, and a simple semantic posture model of mice is established. This includes marking semantic shaping points to replace the key parts of mice according to the movement characteristics and body characteristics of mice, and simply establishing a semantic posture model of mice. The movement information of the shaping points is recorded and obtained, so that it can comprehensively display the body characteristics of mice and fully express the movement form of mice.

[0028] Furthermore, a deep neural network was selected and trained with the data set to extract dynamic information of the mouse semantic image points, including:

[0029] The network structure and performance of the rodent tracking model were compared and analyzed. The deep neural network model Deeplabcut was selected, and the subgraph dataset was used to train and learn the network model, adjust the model parameters, and correct the model offset. The trained network model was used to predict shaping points and extract the motion information of semantic shaping points.

[0030] The method is based on the single-view projective geometry principle of computer vision, mapping the semantic shaping points of the experimental mouse in the image plane of the video sequence to the real scene, and realizing the regression positioning of the shaping points in the real scene, including:

[0031] Based on the single-view projective geometry principle of computer vision, the modeling points in the 2D image plane are mapped to the 3D space through reverse projective transformation to achieve regression positioning of the modeling points and reproduce the motion information of the modeling points in the 3D space. The reverse projection transformation includes: in camera calibration, determining the intrinsic and extrinsic parameters of the camera, making the positioning points in the real scene correspond to the positioning points in the image one by one, and determining the projection matrix P of the camera. The projection matrix P is usually a 3×4 matrix, generally with 11 degrees of freedom, and the formula is as follows:

[0032] sx=PX=K[R,t]X=[P1,P2,P3,P4]X

[0033] Where s is a scalar, X represents the actual position of the i-th anchor point in space, x represents the position of the i-th scene anchor point in the pixel coordinate system, and P i Represents the i-th column of the projection matrix.

[0034] Secondly, according to the projection matrix P, under the pinhole camera model, the coordinates of the image point on the image plane are only related to the choice of the image coordinate system and have nothing to do with the world coordinate system. Therefore, an o-xy plane is established with the bottom surface π of the open field used in the experiment as the world coordinate system. The Z-axis coordinate of the modeling point on this plane is zero, and the following formula is deduced:

[0035]

[0036] (xy 1) T represents the homogeneous coordinates of the shaping point in the image plane, (XY 0 1) T represents the homogeneous coordinates of the shaping point on the open ground in the real scene, h ij represents the i-th row and j-th column of the H matrix, then the above formula can be abbreviated as x=HX, where H is a 3×3 non-singular homogeneous matrix with 8 degrees of freedom. Once the homography matrix of the image plane and the world plane is determined, the image point can be obtained through H -1 The reverse mapping is a point on the world plane, realizing the regression positioning of the statue point in the real scene. The positioning point of the scene structure includes the structural features of the experimental open field itself and the added virtual construction information, and generally the corner point is used as the positioning point.

[0037] Furthermore, the clustering analysis of the regressed and located shaping points by using a density clustering algorithm to lock the behavior subsequence frames of key movements includes:

[0038] The DBSCAN algorithm based on density clustering is used to cluster the motion information of the modeling points located by regression, and the closeness between the points is counted and divided into different categories. The clustering results are analyzed, and different sequences corresponding to different categories are locked. The sequences are screened and finally the key motion subgraph sequences are extracted.

[0039] The DBSCAN algorithm is a density-based clustering algorithm that uses the number of neighboring points in the ε domain of a point to measure the density of the space where the point is located. From the selected core point, it continuously expands to the density-reachable area to obtain a maximized area containing core points and boundary points, and any two points in the area are connected by density. For the input X={x (1) ,x (2) ,x (3) ,…,x (n)} data set is divided into K clusters and a noise point set, and the cluster marker array is introduced Finally, the marker array m is generated j , i = 1, 2, ..., N, where K is The number of mutually different non-negative numbers in . For the setting of parameters in the algorithm, there are mainly two parameters, Eps and MinPts, the former is the field radius, and the latter is the threshold of the core point.

[0040] Furthermore, the method for analyzing characteristic behavior of mice based on scene geometric constraints and deep learning is characterized in that, for the key behavior subsequence frames, based on the scene geometric constraints, the position information of the shaping points on different planes is clarified, and the position information of the shaping points on different planes is analyzed using the scene geometric constraints, highlighting the correlation between the shaping points and the scene geometric constraints and the geometric characteristics between the shaping points, and analyzing their characteristics and differences, and finally determining the characteristic behavior and conducting quantitative analysis. Including:

[0041] According to the movement characteristics of the mouse and the geometric characteristics of the plane scene in which it is located, the mapping changes of the shaping points between different planes are clarified, so that the physical meaning of the changes of the shaping points in different spatial scenes is more obvious, and the differences of the shaping points are observed and measured. The scene geometric constraints include the geometric constraint relationship between the shaping points and the points, lines and planes in the scene space, the constraint relationship between the vanishing lines and vanishing points in the scene structure space, and the intersection ratio between the shaping points. Through the geometric construction of the scene space, the scene geometric constraints are established to make the position information changes of the shaping points clearer, highlight the constraints of the geometric constraints on the shaping points, and under the geometric constraints, it is easier to analyze the changes of the shaping points between subsequence frames. Further, the scene geometric constraints can be used to measure the changes of the shaping points to quantitatively analyze the movement characteristics of the mouse.

[0042] The further described method for analyzing characteristic behavior of mice based on scene geometric constraints and deep learning is characterized in that, by utilizing the above-mentioned research method, taking a homography constraint in scene geometric constraints as an example, the shaping points on different planes are clearly identified, and a strong geometric homography constraint is established on the bottom surface of the image and the real scene with a homography matrix. A combined discrimination condition is established for the changes that occur when the shaping points intersect the bottom surface and for highlighting the differences in changes between different shaping points, to determine the characteristic upright behavior, and to quantitatively analyze the behavioral parameters such as the time, duration, and frequency of the upright behavior. Including,

[0043] A homography matrix is ​​selected in the scene geometry to establish a strong geometric homography constraint between the 2D pixel plane and the plane on which the mouse crawls in the real scene. The shaping points undergo mapping changes between different planes, making the physical significance of the changes of the shaping points on different planes more obvious, highlighting the differences in the changes. The establishment of the combined recognition conditions includes determining the shape vector composed of the key subsequence shaping points in the 2D and 3D planes, calculating the modulus of the vector, and analyzing the differences in the modulus of the shape vectors located in different spaces after the shaping points are mapped in the same frame under the constraints of the scene geometry, as the first combined criterion condition; calculating the angle between the shape vector of each frame of the subsequence and the reference vector in the 3D plane, and analyzing the change in the angle as the second criterion condition.

[0044] The threshold of amplitude change of the judgment condition is limited. For the change of the vector modulus length ratio in different spaces of the same frame, the amplitude threshold D is set. When the amplitude change value is higher than the D value, it is pre-determined that a special behavior exists. A double survey and analysis is performed, and the sequence with double-tone change is regarded as having characteristic behavior. For the change trend of the subsequence vector angle, a threshold T is set. When it is higher than the T value, it is pre-determined that a special behavior exists. A double survey and analysis is performed on the trend of angle change. The trend of double-tone change is used as the second condition for determining that the subsequence has characteristic behavior. After analyzing the subsequence using the combined judgment condition, it can be determined as an upright behavior sequence, and the duration of the sequence and the time node are recorded.

[0045] Furthermore, the video data collection includes a color camera, a bracket and a computer. The camera is mounted on the bracket so that the camera maintains a certain angle with the horizontal direction and shoots at the side of the open field to achieve recording of the entire open field. The computer is used to control the camera's shooting, recording, storage, lighting control, and data processing. The camera calibration uses a calibration plate to perform camera correction processing to eliminate the distortion of the camera. The open field refers to a rectangular parallelepiped with one side of the rectangular parallelepiped being transparent.

[0046] Furthermore, the segmentation algorithm segments mice and prepares a data set of mice, including using a mixed Gaussian model to model the background in the video, eliminating the background, generating a binary image, analyzing the binary image, extracting the outline of the mouse and calculating its center of mass, generating a rectangular ROI area, and segmenting the binary image to obtain a portion that only contains the rectangular ROI area to prepare a sub-image data set of mice.

[0047] Furthermore, the shape vector composed of the shaping points in the 2D and 3D spaces is calculated, and the difference in the modulus of the shape vector composed of the shaping points in different spaces after the shaping points are mapped under the scene geometric constraints in the same frame is analyzed, including calculating the shape vector composed of the two shaping points of the nose tip and the tail root in the 2D space as The vector composed of the two regression positioning shaping points of the nose tip and tail root in 3D space is Calculate the modulus of the vectors in different spaces of the same frame, the ratio is:

[0048] Furthermore, the reference vector includes a vector composed of two shaping points of a frame before the first frame of the key subsequence.

[0049] Due to the adoption of the above technical solution, the technical effects achieved by the present invention are as follows:

[0050] The mouse characteristic behavior analysis method based on scene geometric constraints and deep learning provided by the present invention constructs a scene space and adds a virtual scene structure to a real scene. Compared with an "implanted" device, it does not affect the activities of mice in a natural state, pays attention to the neglected scene space structure, and the scene construction is non-interference.

[0051] The mouse characteristic behavior analysis method based on scene geometric constraints and deep learning provided by the present invention can record, summarize, store and replay the mouse's motion sequence images, analyze and evaluate the mouse's upright behavior, and achieve traceability and review of video sequences, thus overcoming the limitations of traditional animal behavior recognition analysis that is recorded through human eye observation.

[0052] The mouse characteristic behavior analysis method based on scene geometric constraints and deep learning provided by the present invention utilizes a segmentation algorithm to prepare a mouse data set, thereby improving the efficiency of annotating and labeling the data set, reducing the complexity of training a network model, and improving efficiency.

[0053] The method for analyzing characteristic behavior of mice based on scene geometric constraints and deep learning provided by the present invention utilizes scene geometric constraints to regress and locate shaping points, adopts the homography strong geometric constraint relationship between planes to detect the characteristic behavior of mice in three-dimensional scenes, and detects changes in the behavior of mice. There is no need to give the mice any markers, thereby realizing the traceability of motion information in three-dimensional scenes.

[0054] The mouse characteristic behavior analysis method based on scene geometric constraints and deep learning provided by the present invention utilizes density clustering to lock motion subsequences, screens the sequences, saves storage space, highlights the sequences where characteristic behaviors occur, reduces the time of quantitative analysis, and improves efficiency.

[0055] The method for analyzing characteristic behavior of mice based on scene geometric constraints and deep learning provided by the present invention has a non-invasive video data acquisition method and can realize all-round recording of the activities of mice in an enclosure.

[0056] Based on the above reasons, the present invention can be widely promoted in fields such as machine vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the process of the present invention;

[0058] Figure 2 It is a visual structure diagram of the present invention;

[0059] Figure 3 This is a schematic diagram of the video acquisition data flow of the system of the present invention;

[0060] Figure 4 This is a schematic diagram of characteristic behavior detection of the present invention;

[0061] Figure 5 Schematic diagram of the motion parameter measurement of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] like Figure 1 As shown, the present invention provides a method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning, comprising:

[0065] S1, analyze scene features, extract scene information, assist with markers, and construct scene space;

[0066] In specific implementation, as a preferred embodiment of the present invention, the mouse characteristic behavior analysis method based on scene geometry and deep learning is characterized in that the scene characteristics are analyzed, scene information is extracted, auxiliary markers are used, and scene space is constructed, including: according to the structural characteristics of the experimental open field, the corner points and linear features of the open field are extracted; secondly, the position and grid structure of the calibration plate are encoded, and the size of the calibration plate grid is 20×20m 2 , and use the laser beam to maintain parallel illumination on the side walls and bottom surface to encode virtual structural features, superimpose them on the real scene, and structure the scene space.

[0067] S2, collect video data of mouse movement, create a data set, analyze the semantic shaping points of mice, and build a semantic posture model of mice;

[0068] In specific implementation, as a preferred embodiment of the present invention, the video data acquisition of the moving target includes a color camera, a bracket and a computer. The camera is installed on the bracket so that the camera shoots at the side of the experimental open field at an angle less than 45 degrees from the horizontal direction to achieve recording in the entire range of the open field, adjust the focal length, and calibrate the camera using a calibration plate to eliminate distortion; the computer is used to control the camera's shooting, recording, storage, lighting control, and data processing. The experimental open field refers to a rectangular parallelepiped with a size of 65×45×20m 3 , one side of the cuboid is transparent.

[0069] The background difference method based on the mixed Gaussian model is used to detect the moving target mouse, and the moving target is segmented using a segmentation algorithm to prepare a mouse data set.

[0070] During specific implementation, as a preferred embodiment of the present invention, the segmentation algorithm segments moving targets, prepares the data set of mice, including, first, training the background, initializing the parameters in the Gaussian model, and obtaining the preliminary parameters. Then, calculate each pixel of each frame to see whether it matches a certain model, if the value of the pixel is within 3 times of the difference of its model mean, then match, classify it into the model, and update the parameters of the model, if not match, then establish a new Gaussian model with the pixel, initialize the parameters, and replace the most unlikely model in the original model. Finally, each new pixel value X is compared with each mean of the current model, if the difference is within 2 times of the variance, then it is considered to be the background, otherwise it is considered to be the foreground. Utilize the mixed Gaussian model to model the background in the video, eliminate the background, generate the binary image of the mouse, extract the outline of the mouse and calculate its center of mass, generate the rectangular ROI area, segment the part that only contains the rectangular ROI area in the binary image, and form the sub-image data set of the mouse.

[0071] The method analyzes the semantic shaping points of mice, constructs a semantic posture model of mice, and marks and annotates the nose tip and tail root of mice as semantic shaping points instead of specific body parts according to the morphological characteristics and movement characteristics of mice, so as to establish a simple semantic posture model of mice.

[0072] S3. Select a deep neural network and use the data set for learning and training to extract the dynamic information of the mouse semantic image points.

[0073] In specific implementation, as a preferred embodiment of the present invention, the deep neural network is selected and trained with a data set to extract the dynamic information of the semantic shaping points of mice, including analyzing and comparing the network structure and performance of the tracking model of rodents, selecting the deep neural network model Deeplabcut, training and learning the model through the data set, adjusting the parameters of the model and optimizing the calibration processing, and using the trained model to predict the position of the semantic shaping points and extract the motion information of the shaping point coordinates.

[0074] S4. Based on the principle of single-view projective geometry in computer vision, the semantic shaping points of the experimental mouse in the image plane of the video sequence are mapped to the real scene to achieve regression positioning of the shaping points in the real scene;

[0075] In specific implementation, as a preferred embodiment of the present invention, based on the computer vision single view projective geometry principle, the semantic shaping points of the experimental mouse in the video sequence image plane are mapped to the real scene to achieve regression positioning of the shaping points in the real scene, including:

[0076] Based on the principle of single-view projective geometry in computer vision, the modeling points in the 2D image plane are mapped to the 3D space through reverse projective transformation to achieve regression positioning of the modeling points and reproduce the motion information of the modeling points in the 3D space. The reverse projection transformation includes: first, in camera calibration, the intrinsic and extrinsic parameters of the camera are determined so that the positioning points in the real scene correspond to the positioning points in the image one by one, and the projection matrix P of the camera is determined. The projection matrix P is usually a 3×4 matrix with 11 degrees of freedom. The formula is as follows:

[0077] sx=PX=K[R,t]X=[P1,P2,P3,P4]X

[0078] Where s is a scalar, X represents the actual position of the i-th anchor point in space, x represents the position of the i-th scene anchor point in the pixel coordinate system, and P i Represents the i-th column of the projection matrix.

[0079] Secondly, according to the projection matrix P, under the pinhole camera model, the coordinates of the image point on the image plane are only related to the choice of the image coordinate system and have nothing to do with the world coordinate system. Therefore, an o-xy plane is established with the bottom surface π of the open field used in the experiment as the world coordinate system. The Z-axis coordinate of the modeling point on this plane is zero, and the following formula is deduced:

[0080]

[0081] (xy 1) T represents the homogeneous coordinates of the shaping point in the image plane, (XY 0 1) T represents the homogeneous coordinates of the shaping point on the open ground in the real scene, h ij represents the i-th row and j-th column of the H matrix, then the above formula can be abbreviated as x=HX, where H is a 3×3 non-singular homogeneous matrix with 8 degrees of freedom. Once the homography matrix of the image plane and the world plane is determined, the image point can be obtained through H -1 The reverse mapping is performed to a point on the world plane to achieve the regression positioning of the statue point in the real scene.

[0082] S5. Use the density clustering algorithm to perform cluster analysis on the modeling points located by regression, and lock the behavior subsequence frames of key movements.

[0083] In specific implementation, as a preferred embodiment of the present invention, the density clustering algorithm is used to perform cluster analysis on the modeling points located by regression, lock the subsequence of characteristic behavior and screen it, and determine the key behavior sequence frame, including:

[0084] The DBSCAN algorithm based on density clustering is used to cluster the motion information of key points in 3D space. The algorithm's domain radius Eps is set to 12, and the core point threshold MinPts is set to 3. The closeness between each point is counted and divided into different categories. The clustering results are analyzed, and different sequences corresponding to different categories are locked. The sequences are screened and finally the key motion subgraph sequences are extracted.

[0085] S6. For the key behavior sequence frames, clarify the position information of the shaping points in different spatial planes, use the conditions of scene geometric constraints to analyze the position information of the shaping points in different planes, clarify the correlation between the shaping points in different spaces and the scene geometric constraints and the geometric characteristics between the shaping points, analyze their characteristics and differences, and finally, determine the characteristic behavior and conduct quantitative analysis.

[0086] In specific implementation, as a preferred embodiment of the present invention, the modeling points in different spaces are analyzed according to the density clustering algorithm, and the positions of the modeling points in different planes are clarified in the continuous sequence frames, and the geometric relationship between the modeling points and the scene space and the differences of the modeling points under the scene geometric constraints are analyzed. According to the characteristics of the mapping transformation of the modeling points in different spaces of the continuous frames, the characteristic behavior of the mouse is determined, and the specific behavioral parameters of the characteristic behavior are further measured by using the scene geometric constraints for quantitative analysis.

[0087] S7. Using the above research method, taking a homography constraint in the scene geometric constraint as an example, the shaping points on different planes are clearly identified, and strong geometric homography constraints are established on the bottom surface of the image and the real scene. The changes that occur when the shaping points intersect the bottom surface and the differences in changes between different shaping points are highlighted. Combined discrimination conditions are established to determine the upright characteristic behavior, and the behavioral parameters such as the time, duration, and frequency of the upright behavior are quantitatively analyzed.

[0088] During specific implementation, as a preferred embodiment of the present invention, a homography matrix is ​​selected in the scene geometry to establish a strong geometric homography constraint between the 2D pixel plane and the plane where the mouse crawls in the real scene. The shaping points are made to undergo mapping changes between planes, emphasizing the physical changes of the shaping points in different planes and highlighting their differences. The combination recognition condition is established to determine the shape vector composed of the key subsequence shaping points in the 2D and 3D planes, calculate the modulus of the vector, and analyze the differences in the modulus of the shape vectors located in different spaces after the shaping points are mapped in the same frame under the scene geometry constraints, as the first combination criterion condition; the angle between the shape vector of each frame of the subsequence and the reference vector is calculated in the 3D plane, and the change in angle is analyzed as the second criterion condition, including:

[0089] Using the scene geometry constraints, strong geometric homography constraints of the image plane and the bottom surface of the open field are established. The shaping points are mapped to form shape vectors in the 2D and 3D planes respectively. The difference in the vector modulus of the same frame in different planes under the scene geometry constraints is compared as the first combination criterion. The angle between the vector of each frame of the subsequence and the reference vector is calculated in the 3D plane as the second criterion.

[0090] The threshold of amplitude change of the limiting criterion condition is set. For the change of the modulus length ratio of different spaces in the same frame, the amplitude threshold D is set as the limit. Only when the amplitude change value is higher than the D value, it is pre-judged that the special behavior exists. The ratio change trend between sequences is analyzed, and a double investigation is performed. The existence of double-tonal changes is regarded as the judgment condition of the upright sequence. For the change trend of the subsequence vector angle, the threshold T is set as the limit. If it is higher than the T value, it is pre-judged that the special behavior exists. The trend of angle change is double-investigated. The existence of a continuous large change and a trend of maintaining double-tonal changes is used as the second condition for judging the subsequence upright. After analyzing the combined discrimination conditions, the sequence where the upright behavior occurs is determined, and the duration of the sequence and the time node are recorded.

[0091] In specific implementation, as a preferred embodiment of the present invention, the shape vector composed of the shaping points in the 2D and 3D spaces is calculated, and the difference in the modulus of the shape vector composed of the shaping points in different spaces after the shaping points are mapped under the scene geometric constraints in the same frame is analyzed, including calculating the shape vector composed of the two shaping points of the nose tip and the tail root in the 2D space as The vector composed of the two regression positioning shaping points of the nose tip and tail root in 3D space is Calculate the modulus of the vectors in different spaces of the same frame, the ratio is: The reference vector includes a vector of two shaping points in a frame before the first frame of the key subsequence.

[0092] The technical solution of the invention is further described below through specific embodiments.

[0093] As a further preferred embodiment of the present invention, Figure 2 As shown, the scene features are analyzed, scene information is extracted, auxiliary markers are used, and scene space is constructed, including extracting the corner points of the eight inner corners of the experimental open field and the linear features of the open field, using calibration plates to arrange the positions of the calibration plates on the bottom and sides of the open field, encoding the positions of the calibration plates and the corner points of the grid of the calibration plates. With the assistance of a laser beam, segmentation is performed parallel to the bottom and sides to create a virtual scene structure so that it can be superimposed in the scene, increase the hierarchical structure of the scene space, and standardize the scene space such as Figure 2 shown.

[0094] As a further preferred embodiment of the present invention, Figure 3As shown, the video data of mouse movement is collected, a data set is created, the semantic shaping points of the mouse are analyzed, and a semantic posture model of the mouse is constructed. The video data is collected on a 65×45×20m 3 The image was taken in a square open field with a transparent partition on one side to ensure consistent lighting during data collection. The camera was located on one side of the transparent partition, the viewing angle was adjusted to 45°, the focal length of the camera was adjusted, and a calibration plate was used to eliminate distortion. The computer mainly controls the camera to collect and process data, and transmits the processed data to the server for model training. The data set was prepared using a target tracking algorithm based on a mixed Gaussian model to separate the foreground and background, generate a binary image of the mouse, extract the outline of the mouse and calculate its center of mass, generate a rectangular ROI area, segment the binary image to include only the rectangular ROI area, and prepare a sub-image data set of mouse movement. The annotation tool was used to annotate the tail base and nose tip of the mouse as semantic shaping points to establish a semantic posture model. Figure 2 As shown, the trained model is used to predict the location of the mouse shaping points in the dataset and output the information of the semantic shaping points.

[0095] As a further implementation scheme of the present invention, the connection between the image and the bottom surface of the open field is established. First, the intrinsic and extrinsic parameters of the camera are obtained by the camera calibration process, and the known points in the world coordinate system are matched with the image plane points. The camera matrix P is obtained from the 8 positioning points in the open field and the center of the bottom open field. A world coordinate system is established on the bottom surface, and the coordinate in the Z-axis direction is made zero. The homography matrix H between the two planes is calculated. According to the reversibility of the homography matrix H, it can be reversely projected into the real scene, so as to realize the regression positioning of the model prediction shaping point data in the 2D image plane and the 3D open field bottom surface.

[0096] As a further embodiment of the present invention, Figure 2 As shown, the tail base shaping point is regressed and located, and the motion information of the regressed shaping points is clustered using the DBSCAN algorithm based on density clustering. The field radius Eps is set to 12, and the threshold MinPts of the core point is set to 3. The closeness between each point is counted and divided into different categories. The clustering results are analyzed, and a total of 18 different sequences corresponding to different categories are locked. The sequences are screened and finally the key motion subgraph sequences are extracted;

[0097] As a further embodiment of the present invention, Figure 4As shown in the figure, for the key behavior sequence frames, based on the scene geometric constraints, the shaping points on different planes are clearly defined. Under the strong geometric homography constraints established between the image and the bottom surface, the changes that occur when the shaping points intersect the bottom surface and the differences in the changes between different shaping points are highlighted. Combined discrimination conditions are established to detect upright characteristic behaviors, and the behavior parameters such as the moment, duration, and frequency of upright behaviors are quantitatively analyzed, including the regression mapping of the shaping points under the strong geometric constraints of the homography, determining the shape vectors composed of the nose tip and tail base shaping points of the key subsequences in different planes, comparing the differences in the vector modulus, and analyzing the modulus of the shape vectors in different spaces for the same frame under the scene geometric constraints as the first combined discrimination condition; the angle between the shape vector and the reference vector of each frame of the subsequence is calculated in the 3D plane, and the change in angle is analyzed as the second discrimination condition. For the changes in the vector modulus ratio in different spaces of the same frame, when the amplitude threshold is higher than 0.05, it is pre-judged as the existence of upright behavior, and a double survey analysis is performed to discriminate the sequence with double-tone changes as upright characteristic behavior. For the changing trend of the angle of the subsequence vector, when the change threshold is higher than 15°, it is pre-judged that upright behavior exists, and a double investigation and analysis is performed on the trend of angle change. The sequence with a double-tone change trend is judged as the condition of upright characteristic behavior. Two combined discrimination conditions are established. If they are met at the same time, it is determined to be upright behavior, and the duration of the sequence and the time node of the point are recorded.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning, characterized in that The following steps are involved: S1, analyze scene features, extract scene information, assist virtual markers, and construct scene space; S2, collect video data of mouse movement, create a data set, analyze the semantic shaping points of mice, and build a semantic posture model of mice; S3, select a deep neural network and use the data set for learning and training to extract the dynamic information of the mouse semantic image points; S4. Based on the principle of single-view projective geometry in computer vision, the semantic shaping points of the experimental mouse in the image plane of the video sequence are mapped to the real scene to achieve regression positioning of the shaping points in the real scene; S5. Perform cluster analysis on the modeling points located by regression using a density clustering algorithm to lock the subsequence frames of key motion behaviors; S6. For key behavior sequence frames, clarify the position information of shaping points in different spatial planes, use scene geometric constraints to analyze the position information of shaping points in different planes, highlight the correlation between shaping points in different spaces and scene structures, and the geometric characteristics between shaping points, analyze their characteristics and differences, determine characteristic behaviors and conduct quantitative analysis; S7. Using the above analysis method, establish strong geometric homography constraints in plane images and real scenes, qualitatively analyze the transformation of shaping points between different planes, highlight the differences in the changes of different shaping points, determine upright behavior, and quantify the behavior parameters of the time, duration, and frequency of the behavior.

2. A method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S1 specifically analyzes and extracts the intrinsic structural information of the experimental open field itself; encodes the markers and creates virtual structural features so that they can be superimposed on the real scene.

3. The method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S2 specifically includes adjusting the camera indicators, correcting distortion, and collecting mouse videos based on scene structuring; separating the foreground and background based on the background difference method of the mixed Gaussian model, and preparing a mouse data set using a segmentation algorithm; analyzing the mouse's movement characteristics, replacing body parts with semantic shaping points, and constructing a semantic posture model of the mouse.

4. The method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S3 is specifically as follows: compare and analyze the network structure and performance of the rodent tracking model, select the deep neural network model Deeplabcut, and use the subgraph dataset to train and learn the network model, use the trained model to predict the position of the mouse semantic shaping point, and extract the dynamic information of the mouse semantic shaping point.

5. The method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S4 is specifically as follows: based on the principle of single-view projective geometry of computer vision, the semantic shaping points of the experimental mouse in the image plane of the video sequence are mapped to the real scene, and the shaping points in the 2D image plane are mapped to the 3D space through inverse projective transformation to achieve regression positioning of the shaping points and reproduce the motion information of the shaping points in the 3D space.

6. A method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 5, characterized in that: The reverse projection transformation includes: first, in camera calibration, the intrinsic and extrinsic parameters of the camera are determined, so that the positioning points in the real scene correspond to the positioning points in the image one by one, and the projection matrix P of the camera is determined; the projection matrix P is a 3×4 matrix with 11 degrees of freedom, and the formula is as follows: sx=PX=K[R,t]X=[P1,P2,P3,P4]X Where s is a scalar, X represents the actual position of the i-th anchor point in space, x represents the position of the i-th scene anchor point in the pixel coordinate system, and P i represents the i-th column of the projection matrix; Secondly, according to the projection matrix P, under the pinhole camera model, the coordinates of the image point on the image plane are only related to the choice of the image coordinate system and have nothing to do with the world coordinate system. Therefore, an o-xy plane is established with the bottom surface π of the open field used in the experiment as the world coordinate system. The Z-axis coordinate of the modeling point on this plane is zero, and the following formula is deduced: (xy 1) T represents the homogeneous coordinates of the shaping point in the image plane, (XY 0 1) T represents the homogeneous coordinates of the shaping point on the open ground in the real scene, h ij represents the i-th row and j-th column of the H matrix, then the above formula is abbreviated as x = HX, H is a 3×3 non-singular homogeneous matrix with 8 degrees of freedom; once the homography matrix of the image plane and the world plane is determined, the image point passes through H -1 The reverse mapping is performed to a point on the world plane to achieve the regression positioning of the statue point in the real scene.

7. The method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S5 specifically includes: using the DBSCAN algorithm based on density clustering to perform cluster analysis on the motion information of the shaping points in the space, counting the closeness between the points, dividing them into different categories, analyzing the clustering results, locking the different sequences corresponding to different categories, screening the sequences, and finally extracting the key motion subgraph sequence.

8. The method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S6 includes: according to the movement characteristics of the mouse and the geometric characteristics of the scene, including utilizing the geometric constraint relationship between the shaping points and the points, lines and surfaces in the space, the vanishing lines in the scene structure space, the constraint relationship between the vanishing points and the intersection ratio geometric constraint relationship between the shaping points, clarifying the relationship between the shaping points and the geometric constraints of the scene in different spaces, making the physical meaning of the changes in the shaping points more obvious, and measuring the differences in the shaping points.

9. The method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 1, characterized in that: Step S7 specifically includes: establishing a strong geometric homography constraint between the 2D pixel plane and the mouse crawling plane using a homography matrix in the scene geometry; making the shaping points pass through the mapping between the planes, emphasizing the dynamic physical quantity change parameters, and highlighting their differences; The establishment of the combined recognition condition includes calculating the shape vector composed of the shaping points in the 2D and 3D spaces, and analyzing the differences in the size and direction changes between the shape vectors in different planes under the scene geometry constraints in the same frame as the judgment condition; For different discrimination conditions, limit thresholds are set, and double investigation and analysis are carried out to determine whether the trend changes in the discrimination conditions are met as a pre-determination of the existence of behavior. Finally, after analysis of the combined special discrimination conditions, it is determined to be an upright behavior sequence, and the duration of the sequence and the time node at which it is located are recorded.

10. A method for analyzing mouse characteristic behavior based on scene geometric constraints and deep learning according to claim 9, characterized in that: The method of calculating the shape vector composed of the shaping points in the 2D and 3D spaces and analyzing the difference in the modulus of the shape vector composed of the shaping points in different spaces after the shaping points are mapped under the scene geometric constraints in the same frame includes: calculating the shape vector composed of the two shaping points of the nose tip and the tail root in the 2D space as The vector composed of the two regression positioning shaping points of the nose tip and tail root in 3D space is Calculate the modulus of the vectors in different spaces of the same frame, the ratio is: The reference vector includes the vectors of two shaping points in the frame before the first frame of the key subsequence.

Images