Ultra-miniature fluorescence microscope visual monitoring follow-up system based on mouse behavior recognition

Abstract

The invention relates to the technical field of biological recognition, in particular to an ultra-miniature fluorescence microscope visual monitoring follow-up system based on mouse behavior recognition, which comprises a large-range experimental framework observation camera, a manipulator, an ultra-miniature fluorescence microscope, a wire harness and a processing system, and is characterized in that the observation camera is used for collecting motion data of a mouse; the processing system is characterized by further comprising a picture analysis module used for extracting the mouse contour from the motion data and extracting a continuous historical track set based on the mouse contour. According to the invention, all motion tracks of the manipulators are simulated to ensure that paths selected by different manipulators are always in a safe range, so that the collision risk is avoided; by setting dynamic window limitation, it is ensured that the planning result is feasible within the movement capacity range of the mechanical arm, and non-execution paths cannot appear; meanwhile, multi-dimensional evaluation is conducted on different movement tracks, and the mechanical arm can move in a more natural mode; and finally, cooperative regulation and control of the multiple manipulators on wire harness unwinding are achieved.

Description

Technical Field

[0001] The invention relates to the technical field of biological identification, and in particular to an ultra-micro fluorescent microscope visual monitoring follow-up system based on mouse behavior identification. Background Art

[0002] In the field of neuroscience and behavioral research, real-time monitoring and analysis of brain activity and behavioral patterns of animals, especially mice, in a natural state has always been a challenge. Since 2011, GHOSHK, BURNSLD and others have done pioneering work on miniature fluorescence microscopes. In recent years, many researchers have made their own designs based on the aforementioned research results. MiniScopeV3 provides detailed open source design files. The latest research results of projects such as CHEndoscope and TINIscope in recent years have promoted the development of miniature single-photon fluorescence microscopes. With the development of ultra-miniature fluorescence microscopy technology, researchers are able to conduct real-time observations inside brain regions in freely moving small animals, which provides new possibilities for a deeper understanding of the connection between brain function and behavior. However, these technologies still have some limitations, especially in the large-scale observation of animal behavior patterns.

[0003] In brain science research, when using mice to conduct large-scale complex behavioral pattern research experiments, head-mounted ultra-miniature fluorescence microscopes are often used for synchronous observation of multiple brain regions. In the past decade, micro-fluorescence microscopy has continued to update and develop, and has made considerable progress in all aspects. On the basis of the original micro-fluorescence microscopy, the new micro-fluorescence microscope has made important improvements in integrating other cutting-edge technologies in neuroscience research (such as optogenetics), open source, volume imaging, wireless transmission, multi-modality and multi-brain region brain activity recording, and has gradually developed into one of the most indispensable mainstream research tools in neuroscience research. Although the various ultra-miniature fluorescence microscopes currently developed have significant advantages in size and weight, they are still limited by current technical conditions. Due to the need to transmit image signal rate, the length of the signal line is limited, and wired transmission will limit the activity of mice in a larger range, making it impossible to conduct large-scale behavioral pattern experiments. Another reason is that when animals are performing behavioral patterns, because the harness is connected to the head. It will become entangled as the head turns. The above reasons make it difficult for researchers to conduct subsequent large-scale animal behavioral pattern research, thereby limiting the depth and breadth of the research.

[0004] Therefore, an ultra-miniature fluorescence microscope visual monitoring tracking system based on mouse behavior recognition is proposed to solve the above-mentioned problems. Summary of the invention

[0005] Technical issues solved

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides an ultra-micro fluorescence microscope visual monitoring and following system based on mouse behavior recognition, which can effectively solve the problems that the image of the ultra-micro fluorescence microscope in the prior art is affected by the wired signal and cannot monitor a large range of behavior patterns, and the wire harness will interfere with the activities of mice, thereby limiting the study of large-scale behavior patterns.

[0007] Technical Solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] The present invention provides an ultra-micro fluorescent microscope visual monitoring tracking system based on mouse behavior recognition. The technical solution adopted by the present invention is as follows: comprising a large-scale experimental framework observation camera, a manipulator, an ultra-micro fluorescent microscope, a harness and a processing system, wherein the observation camera is used to collect the movement data of the mouse; the processing system also includes:

[0010] The image analysis module is used to extract the mouse outline from the motion data, and extract a continuous historical trajectory collection based on the mouse outline;

[0011] The data extraction module extracts the effective features of the mouse based on the mouse outline and historical trajectory collection;

[0012] The behavior prediction module builds a behavior prediction model based on the historical trajectory collection and effective features of the mouse to obtain the mouse's behavior data;

[0013] The data control module calculates the control parameters of the manipulator based on the predicted behavior data, and uses the control parameters to regulate the manipulator;

[0014] The multi-behavior linkage control module calculates the coordination parameters based on the control parameters, and the coordination parameters are used to control the coordinated regulation of multiple manipulators.

[0015] Wherein, the method of extracting the mouse outline is:

[0016] Perform frame synchronization on the motion data collected by the observation camera, calibrate the spatial position collected by the camera, and build a 3D model of the experimental space;

[0017] Extract each frame image from the motion data and calculate the background image of the current frame, B t (x, y) = (1-α)·B t-1 (x, y)+α·I t (x, y); where B t (x, y) represents the current frame background image corresponding to the pixel point with coordinate position (x, y) at time t; B t-1 (x, y) is the background image of the previous frame; I t(x, y) represents the pixel value at the coordinate position (x, y) at time t; α is the learning rate;

[0018] Then calculate the pixel difference value D t (x, y) = |I t (x, y)-B t (x, y)|; then calculate the foreground mask map Where τ is the difference threshold; the foreground mask image M of each observation camera is t (x, y) and the pixel coordinate position (x, y) are transferred to the global coordinate system to obtain the global perspective projection map (x′, y′) is the coordinate position in the global coordinate system;

[0019] Generate a global foreground mask map by pixel superposition of all global view projection maps Where N is the total number of observation cameras;

[0020] The depth information is used to determine the conflicting pixels in different global perspective projection images, and the final global mask image M is obtained. final (x′, y′); extract the depth values ​​D(x′, y′) of all pixels in the final global mask image, and find the maximum depth value D in the experimental space max and the minimum depth value D min , and calculate the depth range ΔD=D max -D min , the foreground The distribution of is marked as the occluded area; is the pixel threshold;

[0021] Segment the occluded area and generate the segmentation area map M corresponding to K mice K (x S ,y S ), the segmented region map M K (x S ,y S ) as the original image, and obtain the grayscale image of the mouse area, and extract the mouse contour CTR from it mice .

[0022] The method of segmenting the occluded area is as follows:

[0023] All the depth values ​​in the occluded area are constructed into a depth value set, and the depth values ​​D(x′) of all the pixels in it are extracted. S , y′ S );

[0024] Randomly initialize K cluster centers, namely: CC 1 , C.C. 2 , ..., CCK ; Then assign each pixel in the occluded area to the nearest cluster center to obtain the depth cluster center to which the pixel belongs. The assignment formula is:

[0025] In the formula, L(x′ S , y′ S ) is the coordinate position (x′ S , y′ S ) is assigned to the depth cluster center; k Represents the kth deep cluster center; then, according to the allocation result, recalculate each deep cluster center, and the calculation formula is Where P k represents the number of pixels contained in the kth deep cluster center; S k represents the set of all pixels of the kth deep cluster center; repeat the above steps until the deep cluster center υ k convergence.

[0026] The method of extracting a continuous historical trajectory collection is as follows:

[0027] Starting from the first frame of the motion data, based on the segmentation region map M K (x S ,y S ) Calculate the center of mass of all mice using the formula:

[0028]

[0029] In the formula, (x mice，i ,y mice，i ) is the coordinate position of the center of mass of the ith mouse; N K M is the segmentation region map K (x S ,y S ) The number of all pixels in ;

[0030] Initialize the historical trajectory collection MICE of each mouse and get:

[0031] MICE i (t) = {x mice，i (t), y mice，i (t)};MICE i (t) is the historical trajectory of the i-th mouse at time t; (x mice，i (t), y mice，i (t)) represents the coordinate position of the center of mass of the i-th mouse in the current frame; then calculate the coordinate position of the center of mass of the mouse in the next frame (x mice，i (t+1), y mice，i(t+1)); For each mouse, calculate the distance D between its center of mass coordinate position in the current frame and the previous frame i , find the nearest trajectory point And add it to the corresponding historical track collection MICE i In (t), after completing the matching of all mice, the historical trajectory collection of all mice is updated to obtain:

[0032] MICE i (t)=MICE i (t+1)∪{x mice，i ,y mice，i}, Among them, MICE i (t+1) is the historical trajectory set of the next frame of the i-th mouse.

[0033] Wherein, the method of extracting effective features of mice is:

[0034] Calculate the mouse area A of the mouse contour based on the closed boundary of the mouse contour mice and mouse shape characteristics Where D max is the major axis length of the mouse outline, D min is the minor axis length of the mouse outline;

[0035] Mouse-based segmentation region map M K (x S ,y S ) Fitting the main axis direction of the mouse In the formula, ζ x and y are the components of the mouse's principal axis direction on the x-axis and y-axis, respectively;

[0036] Based on the historical trajectory collection MICE of the mouse, the movement speed and direction of the mouse are extracted by calculating the displacement of the center of mass of two consecutive frames, including:

[0037] The displacement of the center of mass on the x-axis is Δx = x mice (t)-x mice (t-1):

[0038] The displacement of the center of mass on the y-axis is Δy = y mice (t)-y mice (t-1);

[0039] Among them, (x mice (t-1), y mice (t-1)) represents the coordinate position of the center of mass of the mouse in the previous frame; (x mice (t), y mice (t)) represents the coordinate position of the center of mass of the mouse in the current frame;

[0040] The mouse's movement speed V mice The calculation formula is:

[0041] Where, T f is the frame time interval; the mouse's movement direction DIR mice The calculation formula is:

[0042] The method of constructing the behavior prediction model is as follows:

[0043] Construct the mouse outline, valid features and historical trajectory collection into a fused feature vector

[0044] X={CTR mice , MICE, A mice , R mice ,DIR,V mice ,DIR mice};

[0045] Collect the mouse contours, effective features and historical trajectory collections within a fixed period of time in the past, and construct the corresponding historical fusion feature vectors; expand the historical fusion feature vectors by time steps to form time series data as the training input of the behavior prediction model, and label the sequence data according to the vector time steps; the labels are the behavior patterns of the mice; initialize the network parameters of the behavior prediction model, and define the loss function of the behavior prediction model Where S is the length of the time series data; θ d is the true value of the dth vector time step; is the predicted value of the dth vector time step; (Tag d ) is the behavior pattern of the mouse labeled at the dth vector time step; ω(Tag d ) is the loss weight function weighted by the mouse's behavior pattern;

[0046] Pass the data sequence data to the behavior prediction model and output the predicted value; calculate the value of the corresponding loss function, and backpropagate the error gradient from the output layer to the input layer; according to the error gradient, use the optimization algorithm to update the network parameters; repeat the training until the behavior prediction model converges or reaches the preset number of iterations, that is, the training of the behavior prediction model is completed.

[0047] The method of calculating the control parameters of the manipulator is as follows:

[0048] The control parameters of the robot are: moving direction DIR robot , Movement Speed ​​V robot and the rotation angle DEG robot; Collect the coordinate position of the robot (x robot ,y robot ), determine whether the robot needs to move, the judgment formula is:

[0049] Where D robot is the straight-line distance between the coordinate position of the manipulator and the coordinate position of the center of mass of the mouse. robot and the preset distance threshold D th In contrast, if the straight-line distance D robot Greater than the distance threshold D th , it means the robot needs to move; otherwise, it does not need to move; if the robot needs to move, calculate the control parameters, the calculation formula is:

[0050]

[0051] Where ΔV is the safe speed increment, is the maximum moving speed of the manipulator;

[0052] DEG robot =DIR robot -DEG′ robot ; In the formula, DEG′ robot is the current direction angle of the robot; at the same time, the rotation angle DEG robot and the preset rotation angle threshold DEG th For comparison, if the rotation angle DEG robot Greater than the rotation angle threshold DEG th , it means that the manipulator needs to rotate; otherwise, it does not need to rotate; the coordinate position of the manipulator is updated based on the control parameters, and the update formula is:

[0053] In the formula, is the coordinate position of the manipulator at time t; is the coordinate position of the robot at time t+1.

[0054] The method of calculating the coordination parameters is as follows:

[0055] Get the harness connection status of each mouse HS i , and calculate the unwinding score US i ; Based on the unwinding score US of each manipulator i Prioritize the values ​​by size;

[0056] Then calculate the distance D between any two manipulators i and z i，z and compare it with the preset safety distance threshold D thFor comparison; if D i，z ≥D th , then the distance between manipulators i and z is safe; if D i，z <D th , then the manipulators i and z are too close, which will trigger the collision avoidance control. The collision avoidance control method is:

[0057] If detected Increase the rotation angles of manipulators i and z to avoid each other;

[0058] If detected: The avoidance path of the manipulator is extracted by constructing a dynamic window.

[0059] The method of increasing the rotation angles of the manipulators i and z is as follows:

[0060] Calculate the relative position vectors of manipulator i and z And get the unit vector of the movement direction of robot i and the unit vector of the robot's z direction of motion

[0061] Calculate the angle between the manipulator i and z using the following formula: In the formula, AOI i，z is the unit vector of the motion direction of robot i and its relative position vector to the robot z The angle between i，z If AOI is greater than 0, the robot i will rotate clockwise. i，z Less than 0, robot i chooses to rotate counterclockwise;

[0062] By rotating the avoidance direction, the moving direction of the robot is updated. The update formula is: In the formula, is the new movement direction of robot i; based on the new movement direction, the coordinate position of robot i is updated to obtain the updated coordinate position The update formula is:

[0063]

[0064] In the formula, is the movement speed of robot i; T s is the time interval.

[0065] The method of extracting the avoidance path of the manipulator is as follows:

[0066] Set the dynamic speed range of the robot, including the maximum speed of the robot and minimum robot speed Among them, μ maxis the maximum acceleration of the manipulator, Δt is the time step; the speed adjustment range of the manipulator is obtained

[0067] Set the dynamic movement direction range of the robot, including the maximum movement direction of the robot and the direction of minimum movement in is the maximum rotational angular velocity of the manipulator; the movement direction adjustment range of the manipulator is obtained

[0068] Extract all motion trajectories based on the speed adjustment range and the motion direction adjustment range, extract the new coordinate positions, and then evaluate them, including:

[0069] Safety Assessment:

[0070]

[0071] like This means that the motion trajectories of manipulators i and z are safe;

[0072] Target proximity assessment:

[0073] Among them, (x tag ,y tag ) is the coordinate position of the target point;

[0074] Balance Assessment:

[0075] SMTH=|V κ -V robot |+|DIR κ -DIR robot |; SMTH is the balance score; V κ It is the k-th speed value in the speed adjustment range; DIR κ The k-th motion direction in the motion direction adjustment range;

[0076] Calculate the motion trajectory score S based on the above parameters TP , the calculation formula is:

[0077] Where η 1 is the weight coefficient of safety assessment, η 2 is the weight coefficient of target proximity evaluation, η 3 is the weight coefficient for balance assessment;

[0078] From all motion trajectories, select the motion trajectory score S TP The path with the largest value is used as the obstacle avoidance path, and the motion state of the robot is updated.

[0079] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0080] 1. In the present invention, the image analysis module can be applied to the situation where there are multiple mice in the experimental space and the depth distribution is uneven by extracting the mouse outline. It has the advantages of strong adaptability and the ability to automatically divide the depth data with complex distribution. In addition, in the previous background processing process, the dynamic foreground target such as the mouse can be directly extracted from the static background. Even if the brightness, shape and position of the mouse area change during the experiment, as long as the difference with the background is significant, the formula can still extract the mouse area, thereby providing an accurate and fast mouse area for subsequent motion trajectory tracking and behavior pattern analysis.

[0081] 2. In the present invention, by quantifying the effective features of mice, a comprehensive feature description can be provided to ensure high-quality input features for the subsequent construction of a behavior prediction model, laying the foundation for automated behavior recognition and experimental results analysis, and improving the accuracy and robustness of the behavior prediction model, including: the area of ​​the mouse, which can reflect the size of the mouse, and the change in area can be used to monitor the weight change or physical condition of the mouse; the shape feature reflects the posture characteristics of the mouse, such as whether it is in a stretched or compressed state, and a sudden change in the aspect ratio may indicate a special behavior of the mouse (such as rolling or jumping), thereby analyzing the behavior pattern of the mouse; the main axis direction reflects the orientation information of the mouse's body, which is an important feature for distinguishing behavior patterns, and can be used to optimize trajectory tracking to ensure the continuity and accuracy of the trajectory; the movement speed reflects the movement intensity of the mouse, which is an important indicator for behavior classification and is used to evaluate the activity level and athletic ability of the mouse; the movement direction reflects the direction of the mouse's movement trajectory, which is an important feature of trajectory analysis and is used to analyze the exploratory behavior and goal-oriented behavior of the mouse.

[0082] 3. In the present invention, by collecting the historical behavior pattern data of mice for analysis, the long-term dependency in the fused feature vector can be captured, which is very suitable for processing the dynamic characteristics in the mouse trajectory data, so that the behavior prediction model can predict the next trajectory or behavior category of the mouse in real time, provide support for real-time monitoring and behavior analysis, and flexibly set the judgment rules of behavior changes according to experimental requirements to meet the needs of different experimental scenarios; importantly, a loss function of the mouse behavior pattern is designed in a targeted manner, which increases the weight of the mouse behavior pattern change, guides the behavior abnormality model to pay more attention to the key areas of behavior pattern changes, and further optimizes the accuracy of the mouse behavior pattern prediction.

[0083] 4. In the present invention, by simulating all the motion trajectories of the manipulator, it is ensured that the paths selected by different manipulators are always within a safe range, thereby avoiding the risk of collision; and by setting dynamic window restrictions, it is ensured that the planning results are feasible within the motion capabilities of the manipulator, and there will be no unexecutable paths that exceed the kinematic capabilities; at the same time, multi-dimensional evaluation of different motion trajectories can allow the manipulator to move in a more natural way; ultimately, coordinated control of multiple manipulators is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of the system flow structure of the present invention;

[0085] Figure 2 This is a schematic diagram of the structure of the visual tracking system in the present invention.

[0086] Markings in the figure: 1. Large-scale experimental framework; 2. Observation camera; 3. Manipulator; 4. Ultra-micro fluorescence microscope; 5. Wire harness. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0088] Example

[0089] Reference Figure 1-2 This case proposes an ultra-micro fluorescent microscope visual monitoring and tracking system based on mouse behavior recognition, including a large-scale experimental frame 1, an observation camera 2, a manipulator 3, an ultra-micro fluorescent microscope 4, a harness 5 and a processing system; the large-scale experimental frame 1 is used to limit the experimental space for large-scale activities of mice, the observation camera 2 and the manipulator 3 are hoisted above the large-scale experimental frame 1, the ultra-micro fluorescent microscope 4 is worn on the head of the mouse, and is connected to the manipulator 3 through the harness 5; the observation camera 2 is used to detect the motion data of the experimental mouse below in real time (i.e., the mouse's behavior and activity trajectory image), and transmit the motion data to the processing system, and obtain the control parameters of the manipulator after processing; the manipulator 5 unwinds the harness 5 based on the control parameters.

[0090] Among them, the processing system includes an image analysis module, a data extraction module, a behavior prediction module, a data control module and a multi-behavior linkage control module; among them, the image analysis module is used to extract the mouse outline from the motion data, and extract a continuous historical trajectory collection based on the mouse outline; the data extraction module extracts the effective features of the mouse based on the mouse outline and the historical trajectory collection; the behavior prediction module constructs a behavior prediction model based on the historical trajectory collection and effective features of the mouse to obtain the behavior data of the mouse; the data control module calculates the control parameters of the manipulator based on the predicted behavior data, and uses the control parameters to regulate the manipulator; the multi-behavior linkage control module calculates the coordination parameters based on the control parameters, and the coordination parameters are used to control the coordinated regulation of multiple manipulators.

[0091] Specifically, in this case, when extracting the mouse contour, when there are multiple mice in the experimental space, individual occlusion may occur when the mice interact or converge, resulting in errors in the extraction of the mouse contour.

[0092] Therefore, in this case, the way to extract the mouse outline is:

[0093] The observation cameras are frame synchronized based on the time synchronizer to ensure that the motion data captured by the observation cameras at different positions are consistent in time; then the spatial positions of all observation cameras are calibrated, a 3D model of the experimental space is constructed, and the motion data captured by different observation cameras are mapped to a unified three-dimensional coordinate system;

[0094] The background of the motion data captured by each observation camera is subtracted to avoid errors caused by different viewing angles. The subtraction method is: extract each frame image from the motion data and calculate the background image B of the current frame t (x, y), the calculation formula is: t (x, y) = (1-α)·B t-1 (x, y)+α·I t (x, y); where B t (x, y) represents the background image of the current frame corresponding to the pixel point with coordinate position (x, y) at time t, which is used to extract the foreground (i.e., the mouse outline) of the subsequent image; B t-1 (x, y) is the background image of the previous frame; I t (x, y) represents the pixel value at the coordinate position (x, y) at time t; α is the learning rate, which represents the weight of the background image of the current frame. Its value range is between 0 and 1 and is usually determined based on experimental environment data; (1-α) represents the weight of the background image of the previous frame;

[0095] Then the formula for calculating the pixel difference value is: t (x, y) = |I t (x, y)-B t (x, y)|; where Dt (x, y) represents the pixel difference value at the coordinate position (x, y) at time t; calculate the foreground mask map Where, τ is the difference threshold, which is used to determine whether the current pixel belongs to the foreground; M t (x, y) is the binary result of the pixel at the coordinate position (x, y) at time t, which is used to indicate whether it is the foreground; the foreground mask map M of each observation camera t (x, y) and the coordinate position of the pixel point (x, y) are converted to the global coordinate system to obtain the global perspective projection map. The conversion formula is: In the formula, is the global perspective projection image, which represents the result of transforming the foreground mask image generated by each observation camera to the global coordinate system; (x′, y′) is the coordinate position in the global coordinate system, corresponding to the pixel plane of the experimental space; [K, R, T] is the internal and external parameter matrix of the observation camera; all global perspective projection images are superimposed by pixels to generate the global foreground mask image Where N is the total number of observation cameras; at the same time, if a pixel point conflicts in the global perspective projection map of multiple observation cameras, it is determined by depth information, M final (x′, y′) = argmin i∈{1，2，...，N} D t，i (x′, y′); where M final (x′, y′) is the final global mask map, D t，i (x′, y′) is the depth value, which represents the depth estimation value of the coordinate position (x′, y′) in the global coordinate system of each observation camera of the ith observation camera, reflecting the distance from the target to the observation camera;

[0096] When there are multiple mice in the experimental space, extract the depth values ​​D(x′, y′) of all pixels in the final global mask map and find the maximum depth value D in the experimental space. max and the minimum depth value D min , and calculate the depth range ΔD=D max -D min , the depth range ΔD is used to determine whether there are multiple mice in the experimental space; when When , all the distributions with large depth changes in the foreground are marked as occluded areas; is the pixel threshold;

[0097] All the depth values ​​in the occluded area are constructed into a depth value set, and the depth values ​​D(x′) of all the pixels in it are extracted. S , y′ S ); Assuming K targets (i.e., K to mouse 0), randomly initialize K cluster centers (the cluster centers correspond to the depth layer centers of the mice), which are: CC 1 , C.C. 2, ..., CC K ; Then assign each pixel in the occluded area to the nearest cluster center to obtain the depth cluster center to which the pixel belongs. The assignment formula is: In the formula, L(x′ S , y′ S ) is the coordinate position (x′ S , y′ S ) is assigned to the depth cluster center; k Represents the kth deep cluster center; then, according to the allocation result, recalculate each deep cluster center, and the calculation formula is Where P k represents the number of pixels contained in the kth deep cluster center; S k represents the set of all pixels of the kth deep cluster center; repeat the above steps until the deep cluster center υ k Convergence (i.e. the center no longer changes or the change is less than the set threshold); finally generate the segmentation region map M corresponding to K mice K (x S ,y S ), denoted as {M 1 (x S ,y S ), M 2 (x S ,y S ), .., M K (x S ,y S )}, each segmentation region map M K (x S ,y S ) are both binary masks, corresponding to the depth value area of ​​a mouse; finally, the segmentation area map M K (x S ,y S ) as the original image, and obtain the grayscale image of the mouse area, and then extract the mouse contour CTR mice ;

[0098] In summary, the above method for extracting mouse contours can be applied to situations where there are multiple mice in the experimental space and the depth distribution is uneven. It has the advantages of strong adaptability and the ability to automatically divide complex distributed depth data. In the previous background processing process, dynamic foreground targets such as mice can be directly extracted from the static background. Even if the brightness, shape, and position of the mouse area change during the experiment, as long as the difference with the background is significant, the formula can still extract the mouse area, providing accurate and fast mouse area for subsequent motion trajectory tracking and behavior pattern analysis.

[0099] Furthermore, the method of extracting a continuous historical trajectory collection based on the mouse outline is as follows:

[0100] Starting from the first frame image of the motion data, based on the segmentation region map M K (x S ,y S ) detects and calculates the centroid of all mice using the following formula:

[0101]

[0102] In the formula, (x mice，i ,y mice，i ) is the centroid coordinate position of the ith mouse, reflecting the segmentation region map M K (x S ,y S ) is the average position of all pixels in the image, i.e., the center point of the mouse shape; N K M is the segmentation region map K (x S ,y S ) The number of all pixels in ;

[0103] Initialize the historical trajectory collection MICE of each mouse, MICE i (t) = {x mice，i (t), y mice，i (t)};MICE i (t) is the historical trajectory of the i-th mouse at time t; (x mice，i (t), y mice，i (t)) represents the coordinate position of the center of mass of the ith mouse in the current frame (i.e., the initial frame); then calculate the coordinate position of the center of mass of the mouse in the next frame (x mice，i (t+1), y mice，i (t+1)); For each mouse, calculate the distance D between its center of mass coordinate position in the current frame and the previous frame i , find the nearest trajectory point And add it to the corresponding historical track collection MICE i In (t), after all mice are matched, the historical trajectory collection MICE of all mice is updated. i (t)=MICE i (t+1)∪{x mice，i ,y mice，i}, Among them, MICE i (t+1) is the collection of historical trajectories of the next frame of the i-th mouse. Through deep segmentation and matching, multiple targets (such as mice) can be accurately separated in occluded scenes and the continuity of target identity (i.e., the integrity of the mouse's activity trajectory) can be maintained, providing support for subsequent behavioral analysis.

[0104] Furthermore, the method of extracting effective features of mice is:

[0105] Calculate the mouse area A of the mouse contour based on the closed boundary of the mouse contour mice and mouse shape characteristics Where D max is the main axis length of the mouse outline, that is, the longest diameter of the mouse outline, D min is the minor axis length of the mouse outline, i.e., the shortest diameter of the mouse outline;

[0106] Mouse-based segmentation region map M K (x S ,y S ) fits the mouse's main axis direction DIR and obtains: In the formula, ζ x and y The components of the mouse's main axis direction on the x-axis and y-axis are obtained by constructing the covariance matrix of the point set based on the mouse's centroid coordinate position, and then performing eigenvalue decomposition on the covariance matrix to obtain the variance of the main axis and the eigenvector (ζ x and y );

[0107] Based on the historical trajectory collection MICE of the mouse, the movement speed and direction of the mouse are extracted by calculating the displacement of the center of mass of two consecutive frames, including:

[0108] The displacement of the center of mass on the x-axis is Δx = x mice (t)-x mice (t-1):

[0109] The displacement of the center of mass on the y-axis is Δy = y mice (t)-y mice (t-1);

[0110] Among them, (x mice (t-1), y mice (t-1)) represents the coordinate position of the center of mass of the mouse in the previous frame; (x mice (t), y mice (t)) represents the coordinate position of the center of mass of the mouse in the current frame;

[0111] The mouse's movement speed V mice The calculation formula is: Where, T f is the frame time interval; the mouse's movement direction DIR mice The calculation formula is:

[0112] In summary, by quantifying the effective features of mice, a comprehensive feature description can be provided to ensure the provision of high-quality input features for the subsequent construction of the behavior prediction model, laying the foundation for automated behavior recognition and experimental results analysis, and improving the accuracy and robustness of the behavior prediction model. Specifically, the mouse area A mcie , can reflect the size of the mouse, and the change in area can be used to monitor the weight change or physical condition of the mouse; the shape feature R mice It reflects the posture characteristics of the mouse, such as whether it is in a stretched or compressed state. Sudden changes in the aspect ratio may indicate special behaviors of the mouse (such as rolling or jumping), and then analyze the behavior pattern of the mouse; the main axis direction DIR reflects the orientation information of the mouse body and is an important feature for distinguishing behavior patterns. It can be used to optimize trajectory tracking and ensure the continuity and accuracy of the trajectory; the movement speed V mice It reflects the intensity of the mouse's movement and is an important indicator of behavioral classification. It is used to evaluate the activity level and exercise ability of mice. The direction of movement DIR mice It reflects the direction of the mouse's movement trajectory and is an important feature of trajectory analysis. It is used to analyze the mouse's exploratory behavior and goal-directed behavior.

[0113] More specifically, the behavior prediction model is constructed as follows:

[0114] The mouse outline, effective features and historical trajectory collection are constructed into a fused feature vector X, expressed as:

[0115] X={CTR mice , MICE, A mice , R mice ,DIR,V mice ,DIR mice};

[0116] Collect the mouse outline, effective features and historical trajectory collection of the mouse in the past fixed time, and construct the corresponding historical fusion feature vector; expand the historical fusion feature vector by time step to form time series data as the training input of the behavior prediction model, and label the sequence data according to the vector time step; the label is the behavior pattern of the mouse;

[0117] Define the basic structure of the behavior prediction model, which includes the input layer, hidden layer and output layer; the input layer is used to receive the fused feature vector X d As input, d represents the vector time step; the hidden layer consists of several recursive neurons; the calculation formula of each recursive neuron is: H d =f(W·X d +W′·H d-1 +b H );where H dis the hidden state vector of the current vector time step d; f(...) is the activation function, such as tanh or ReLU; W is the weight matrix from the input layer to the hidden layer; W′ is the recurrent weight matrix of the hidden layer; b H is the bias vector of the hidden layer;

[0118] The output layer calculates the behavior prediction model based on the output of the hidden layer to obtain the predicted value θ d =g(W″·H d +b θ );where θ d =g(...) is the activation function of the output layer, for example, a linear function or softmax; W″ is the weight matrix from the hidden layer to the output layer; b θ is the bias vector of the output layer;

[0119] Initialize the network parameters of the behavior prediction model; the network parameters include the weight matrix W from the input layer to the hidden layer, the cyclic weight matrix W′ of the hidden layer, and the bias vector b of the hidden layer H , the weight matrix W″ from the hidden layer to the output layer and the bias vector b of the output layer θ ;

[0120] Define the loss function LOSS of the behavior prediction model:

[0121] Where S is the length of the time series data; θ d is the true value of the d-th vector time step, that is, the collection of mouse profiles, effective features, and historical trajectories in the actual mouse behavior data; is the predicted value of the dth vector time step, that is, the mouse profile, effective features and historical trajectory collection in the mouse behavior data predicted by the behavior prediction model; (Tag d ) is the behavior pattern of the mouse labeled at the dth vector time step; ω(Tag d ) is the loss weight function weighted by the mouse's behavior pattern;

[0122] The data sequence data is passed to the input layer, and then passes through the hidden layer and the output layer in turn to output the predicted value; the corresponding loss function value is calculated, and the error gradient is back-propagated from the output layer to the input layer; according to the error gradient, the network parameters are updated using an optimization algorithm (such as gradient descent, Adam, etc.) to reduce the value of the loss function; the data sequence data is repeatedly trained until the behavior prediction model converges (that is, the value of the loss function no longer changes) or reaches the preset number of iterations, and the training of the behavior prediction model is completed.

[0123] In summary, by collecting historical behavior pattern data of mice for analysis, we can capture long-term dependencies in the fused feature vector, which is very suitable for processing the dynamic characteristics of mouse trajectory data, so that the behavior prediction model can predict the next trajectory or behavior category of the mouse in real time, providing support for real-time monitoring and behavior analysis, and can flexibly set the judgment rules of behavior changes according to experimental needs to meet the needs of different experimental scenarios; importantly, a loss function of the mouse behavior pattern is designed specifically, which increases the weight of the mouse behavior pattern changes, guides the behavior abnormality model to pay more attention to the key areas of behavior pattern changes, and further optimizes the accuracy of the mouse behavior pattern prediction.

[0124] It is worth noting that in this case, the behavior data predicted and output by the behavior prediction model include: predicted centroid coordinate position (x′ mice , y′ mice ), predict the direction of movement DIR′ mice and predicted motion speed V′ mice ; Collect the coordinate position of the robot (x robot ,y robot ); The data control module calculates the control parameters of the manipulator based on the above data, including the moving direction DIR of the manipulator robot , Movement Speed ​​V robot and the rotation angle DEG robot ; Then use the control parameters to regulate the manipulator. The specific regulation method is:

[0125] To determine whether the robot needs to move, the judgment formula is:

[0126] Where D robot is the straight-line distance between the coordinate position of the manipulator and the coordinate position of the center of mass of the mouse. robot and the preset distance threshold D th In contrast, if the straight-line distance D robot Greater than the distance threshold D th , it means that the robot needs to move; otherwise, it does not need to move; if the robot needs to move, calculate the moving direction, moving speed and rotation angle, the calculation formula is:

[0127]

[0128] Where ΔV is the safety speed increment, which is used to provide a certain degree of operational flexibility in the robot speed response and is determined based on the robot's challenge speed and experimental data; is the maximum movement speed allowed by the robot;

[0129] DEG robot =DIRrobot -DEG′ robot ; In the formula, DEG′ robot is the current direction angle of the robot; at the same time, the rotation angle DEG robot and the preset rotation angle threshold DEG th For comparison, if the rotation angle DEG robot Greater than the rotation angle threshold DEG th , it means the robot needs to rotate; otherwise, it does not need to rotate;

[0130] After the manipulator is adjusted according to the control parameters, the coordinate position of the manipulator is updated. The update formula is:

[0131]

[0132] In the formula, is the coordinate position of the manipulator at time t (i.e. the coordinate position before adjustment); is the coordinate position of the manipulator at time t+1 (i.e. the adjusted coordinate position); T s is the time interval;

[0133] In summary, this solution predicts the behavior pattern of the mouse, calculates the control parameters of the manipulator based on the prediction results, and creates control parameters under the complex behavior of the mouse, thereby achieving high-precision motion generation, ensuring that the manipulator can follow the mouse quickly and smoothly, and effectively untwist the harness to avoid the problem of harness entanglement when the mouse is moving; in addition, by setting the distance threshold and the rotation angle threshold, the intelligent control level of the manipulator can be improved. For example, if the mouse and the manipulator are close enough and the direction adjustment angle is very small, the system will stop automatically to avoid issuing unnecessary commands, avoiding useless movements and frequent rotations, extending the life of the manipulator and improving control stability.

[0134] It is worth mentioning that in this case, when there are multiple mice in the experimental space, each mouse is equipped with an independent ultra-micro fluorescent microscope on its head, and each ultra-micro fluorescent microscope is connected to an independent harness, each harness is controlled by an independent manipulator, and when multiple manipulators are synchronously regulated, interference may occur; therefore, the coordination parameters of multiple manipulators are calculated through the multi-behavior linkage control module, and the coordination parameters are used to control multiple manipulators to avoid mutual interference during the regulation of the manipulators; the coordination parameters are calculated as follows:

[0135] Get the harness connection status of each mouse HS i , including the harness length L HS，i and kink score index KS iAs the mouse moves (fleeing, circling, stretching, etc.), the length of the harness will change continuously; the kink scoring index KS i Determine whether the wiring harness is kinked and whether the wiring harness needs to be untwisted;

[0136] Harness length L HS，i The method of obtaining is: calculate the center of mass coordinate position (x mice，i ,y mice，i ) and the coordinate position (x HS，i ,y HS，i ), i.e. the harness length

[0137] Kink score index KS i The calculation method is: In the formula, Q is the total number of sampling points; It represents the change in the movement direction of the ith mouse between two adjacent times, reflecting whether the mouse has circled, turned, or gone around. The calculation formula is: in is the movement direction of the i-th mouse at time t; is the movement direction of the i-th mouse at time t; the kink scoring index KS i and the preset kink threshold KS th For comparison, if the kink scoring index KS i Greater than the kink threshold KS th , it can be determined that the wiring harness is kinked;

[0138] Based on the harness connection status of each mouse HS i Calculation of unwinding score US i , the calculation formula is:

[0139] In the formula, is the relative speed between the movement speed of the ith mouse and the theoretical maximum speed of the mouse; ε 1 is the weight coefficient of the kink scoring index, ε 2 is the weight coefficient of the harness length, ε 3 is the weight coefficient of the relative speed of the mouse; ε 1 , ε 2 and ε 3 All are determined by experimental data; finally, the unwinding score US of each bundle is i Sort the regulatory order of multiple manipulators and score US iThe mouse behavior and harness status are comprehensively considered to efficiently quantify the priority of untwisting requirements, provide a standardized reference for the allocation of untwisting tasks among multiple manipulators, avoid misjudgment caused by a single factor, and help improve the reliability and flexibility of untwisting control.

[0140] Then calculate the distance D between any two manipulators i and z i，z , in is the coordinate position of robot i, is the coordinate position of the robot z; and it is combined with the preset safety distance threshold D th Make a comparison;

[0141] If D i，z ≥D th , then the distance between manipulators i and z is safe and they can continue to perform operations at the same time;

[0142] If D i，z <D th , then the manipulators i and z are too close, triggering collision avoidance control. The collision avoidance control method is:

[0143] If detected: This means that the manipulators i and z are relatively close, but there is still enough space to adjust the direction. For example, the rotation angles of manipulators i and z are increased to avoid each other. Specifically:

[0144] Calculate the relative position vectors of manipulator i and z And get the unit vector of the movement direction of robot i and the unit vector of the robot's z direction of motion Calculate the angle between the manipulator i and z using the following formula: In the formula, AOI i，z is the unit vector of the motion direction of robot i and its relative position vector to the robot z The angle between i，z If it is greater than 0, it means that the position of robot z relative to robot i is on the right side of the current direction, and robot i chooses to rotate clockwise; if AOI i，z Less than 0, it means that the position of robot z relative to robot i is on the left side of the current direction, and robot i chooses to rotate counterclockwise;

[0145] By determining the rotation avoidance direction of manipulators i and z, the moving directions of manipulators i and z are updated; taking manipulator i as an example, the update formula is: In the formula, is the new movement direction of robot i; based on the new movement direction, the coordinate position of robot i is updated to obtain the updated coordinate position The update formula is:

[0146]

[0147] In the formula, is the movement speed of robot i; T s is the time interval;

[0148] The robot follows the updated The robot can operate according to the coordinate position of the target to achieve efficient and smooth avoidance behavior between robots.

[0149] If detected: This means that there is an impending collision between manipulators i and z, or there is an obstacle on the path to the target point; therefore, the avoidance method between manipulators i and z is:

[0150] Set the dynamic speed range of the robot, including the maximum speed of the robot and minimum robot speed Among them, μ max is the maximum acceleration of the manipulator, Δt is the time step; and then the speed adjustment range of the manipulator is obtained

[0151] Set the dynamic movement direction range of the robot, including the maximum movement direction of the robot and the direction of minimum movement in is the maximum rotational angular velocity of the manipulator; and then the movement direction adjustment range of the manipulator is obtained

[0152] Based on the speed adjustment range and the movement direction adjustment range, all possible movement speeds and movement directions are listed, and the corresponding movement trajectories are calculated, and the new coordinate positions are extracted and evaluated, including:

[0153] Safety assessment, that is, the minimum distance between manipulator i and manipulator z on the trajectory, the evaluation formula is:

[0154]

[0155] like This means that the motion trajectories of manipulators i and z are safe; otherwise, the motion trajectories are unsafe;

[0156] Target proximity assessment, that is, the distance D from the end point of the motion trajectory to the target point goal , the evaluation formula is:

[0157] Among them, (x tag ,y tag ) is the coordinate position of the target point; D goalThe smaller it is, the better the motion trajectory is;

[0158] Balance assessment, that is, whether the speed and direction of the motion trajectory change smoothly, the evaluation formula is:

[0159] SMTH=|V κ -V robot |+|DIR κ -DIR robot |; SMTH is the balance score, the smaller the value, the smoother the motion trajectory; V κ It is the k-th speed value in the speed adjustment range; DIR κ The k-th motion direction in the motion direction adjustment range;

[0160] Calculate the motion trajectory score S based on the above parameters TP , the calculation formula is:

[0161] Where η 1 is the weight coefficient of safety assessment, η 2 is the weight coefficient of target proximity evaluation, η 3 The weight coefficients for balance assessment are obtained through calibration of experimental data;

[0162] From all motion trajectories, select the motion trajectory score S TP The largest value is used as the obstacle avoidance path, and the motion state of the manipulator is updated, thereby realizing the coordinated control of obstacle avoidance among multiple manipulators.

[0163] In summary, this solution ensures that the paths selected by different manipulators are always within a safe range by simulating all the motion trajectories of the manipulator, thereby avoiding the risk of collision. It also ensures that the planning results are feasible within the range of the manipulator's motion capabilities by setting dynamic window limits, and that there will be no unexecutable paths that exceed the kinematic capabilities. At the same time, multi-dimensional evaluation of different motion trajectories allows the manipulator to move in a more natural way, ultimately achieving coordinated control of multiple manipulators.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An ultra-miniature fluorescence microscope visual monitoring tracking system based on mouse behavior recognition, comprising a large-scale experimental framework observation camera, a manipulator, an ultra-miniature fluorescence microscope, a wiring harness and a processing system, wherein the observation camera is used to collect the movement data of the mouse; characterized in that: The processing system further comprises: The image analysis module is used to extract the mouse outline from the motion data, and extract a continuous historical trajectory collection based on the mouse outline; The data extraction module extracts the effective features of the mouse based on the mouse outline and historical trajectory collection; The behavior prediction module builds a behavior prediction model based on the historical trajectory collection and effective features of the mouse to obtain the mouse's behavior data; The data control module calculates the control parameters of the manipulator based on the predicted behavior data, and uses the control parameters to regulate the manipulator; The multi-behavior linkage control module calculates the coordination parameters based on the control parameters, and the coordination parameters are used to control the coordinated regulation of multiple manipulators.

2. The ultra-micro fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 1, characterized in that: The method of extracting the mouse outline is: Perform frame synchronization on the motion data collected by the observation camera, calibrate the spatial position collected by the camera, and build a 3D model of the experimental space; Extract each frame image from the motion data and calculate the background image of the current frame, B t (x, y) = (1-α)·B t-1 (x, y)+α·I t (x, y); where B t (x, y) represents the current frame background image corresponding to the pixel point with coordinate position (x, y) at time t; B t-1 (x, y) is the background image of the previous frame; Y t (x, y) represents the pixel value at the coordinate position (x, y) at time t; α is the learning rate; Then calculate the pixel difference value D t (x, y) = |I t (x, y)-B t (x, y)|; then calculate the foreground mask map Where τ is the difference threshold; the foreground mask image M of each observation camera is t (x, y) and the pixel coordinate position (x, y) are transferred to the global coordinate system to obtain the global perspective projection map (x′, y′) is the coordinate position in the global coordinate system; Generate a global foreground mask map by pixel superposition of all global view projection maps Where N is the total number of observation cameras; The depth information is used to determine the conflicting pixels in different global perspective projection images, and the final global mask image M is obtained. final (x′, y′); extract the depth values ​​D(x′, y′) of all pixels in the final global mask map and find the maximum depth value D in the experimental space max and the minimum depth value D min , and calculate the depth range ΔD=D max -D min , the foreground The distribution of is marked as the occluded area; is the pixel threshold; Segment the occluded area and generate a segmentation map N corresponding to K mice K (x S ,y S ), the segmented region map M K (x S ,y S ) as the original image, and obtain the grayscale image of the mouse area, and extract the mouse contour CTR from it mice .

3. The ultra-micro fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 2, characterized in that: The method for segmenting the occluded area is as follows: All the depth values ​​in the occluded area are constructed into a depth value set, and the depth values ​​D(x′) of all the pixels in it are extracted. S , y′ S ); Randomly initialize K cluster centers, namely: CC1, CC2, ..., CC K ; Then assign each pixel in the occluded area to the nearest cluster center to obtain the depth cluster center to which the pixel belongs. The assignment formula is: In the formula, L(x′ S , y′ S ) is the coordinate position (x′ S , y′ S ) is assigned to the depth cluster center, v k Represents the kth deep cluster center; then, according to the allocation result, recalculate each deep cluster center, and the calculation formula is Where P k represents the number of pixels contained in the kth deep cluster center; S k represents the set of all pixels of the kth deep cluster center; repeat the above steps until the deep cluster center v k convergence.

4. The ultra-micro fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 2, characterized in that: The method of extracting a continuous collection of historical trajectories is: Starting from the first frame of the motion data, based on the segmentation region map M K (x S ,y S ) Calculate the center of mass of all mice using the formula: In the formula, (x mice，i ,y mice，i ) is the coordinate position of the center of mass of the ith mouse; N K M is the segmentation region map K (x S ,y S ) The number of all pixels in ; Initialize the historical trajectory collection MICE of each mouse and get: MICE i (t) = {x mice，i (t), y mice，i (t)}; MICE i (t) is the historical trajectory of the i-th mouse at time t; (x mice，i (t), y mice，i (t)) represents the coordinate position of the center of mass of the i-th mouse in the current frame; then calculate the coordinate position of the center of mass of the mouse in the next frame (x mice，i (t+1), y mice，i (t+1)); For each mouse, calculate the distance D between its center of mass coordinate position in the current frame and the previous frame i , find the nearest trajectory point And add it to the corresponding historical track collection MICE i In (t), after completing the matching of all mice, the historical trajectory collection of all mice is updated to obtain: Where MICE(t+1) is the collection of historical trajectories of the next frame of the i-th mouse.

5. The ultra-micro fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 4, characterized in that: The method of extracting effective features of mice is: Calculate the mouse area A of the mouse contour based on the closed boundary of the mouse contour mice and mouse shape characteristics Where D max is the major axis length of the mouse outline, D min is the minor axis length of the mouse outline; Mouse-based segmentation region map M K (x S ,y S ) Fitting the main axis direction of the mouse In the formula, ζ x and y are the components of the mouse's principal axis direction on the x-axis and y-axis, respectively; Based on the historical trajectory collection MICE of the mouse, the movement speed and direction of the mouse are extracted by calculating the displacement of the center of mass of two consecutive frames, including: The displacement of the center of mass on the x-axis is Δx = x mice (t)-x mice (t-1): The displacement of the center of mass on the y-axis is Δy = y mice (t)-y mice (t-1); Among them, (x mice (t-1), y mice (t-1)) represents the coordinate position of the center of mass of the mouse in the previous frame; (x mice (t), y mice (t)) represents the coordinate position of the center of mass of the mouse in the current frame; The mouse's movement speed V mice The calculation formula is: Where, T f is the frame time interval; the mouse's movement direction DIR mice The calculation formula is:

6. The ultra-miniature fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 5, characterized in that: The method of constructing the behavior prediction model is: Construct the mouse outline, valid features and historical trajectory collection into a fused feature vector X={CTR mice ,MICE,A mice ,R mice ,DIR,V mice ,DIR mice }; Collect the mouse contours, effective features and historical trajectory collections within a fixed period of time in the past, and construct the corresponding historical fusion feature vectors; expand the historical fusion feature vectors by time steps to form time series data as the training input of the behavior prediction model, and label the time series data according to the vector time steps; the labels are the behavior patterns of the mice; initialize the network parameters of the behavior prediction model, and define the loss function of the behavior prediction model Where s is the length of the time series data; θ d is the true value of the dth vector time step; is the predicted value of the dth vector time step; (Tag d ) is the behavior pattern of the mouse labeled at the dth vector time step; ω(Tag d ) is the loss weight function weighted by the mouse's behavior pattern; Pass the data sequence data to the behavior prediction model and output the predicted value; Calculate the value of the corresponding loss function and backpropagate the error gradient from the output layer to the input layer; use the optimization algorithm to update the network parameters based on the error gradient; repeat the training until the behavior prediction model converges or reaches the preset number of iterations, thus completing the training of the behavior prediction model.

7. The ultra-miniature fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 6, characterized in that: The method for calculating the control parameters of the manipulator is: The control parameters of the robot are: moving direction DIR robot , Movement Speed ​​V robot and the rotation angle DEG robot ; Collect the coordinate position of the robot (x robot ,y robot ), determine whether the robot needs to move, the judgment formula is: Where D robot is the straight-line distance between the coordinate position of the manipulator and the coordinate position of the center of mass of the mouse. robot and the preset distance threshold D th In contrast, if the straight-line distance D robot Greater than the distance threshold D th , it means the robot needs to move; otherwise, it does not need to move; if the robot needs to move, calculate the control parameters, the calculation formula is: Where ΔV is the safe speed increment, is the maximum moving speed of the manipulator; DEG robot =DIR robot -DEG′ robot ; In the formula, DEG′ robot is the current direction angle of the robot; at the same time, the rotation angle DEG robot and the preset rotation angle threshold DEG th For comparison, if the rotation angle DEG robot Greater than the rotation angle threshold DEG th , it means that the manipulator needs to rotate; otherwise, it does not need to rotate; the coordinate position of the manipulator is updated based on the control parameters, and the update formula is: In the formula, is the coordinate position of the manipulator at time t; is the coordinate position of the robot at time t+1.

8. The ultra-miniature fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 7, characterized in that: The method for calculating the coordination parameters is: Get the harness connection status of each mouse HS i , and calculate the unwinding score US i ; Based on the unwinding score US of each manipulator i Prioritize the values ​​by size; Then calculate the distance D between any two manipulators i and z i，z and compare it with the preset safety distance threshold D th For comparison; if D i，z ≥D th , then the distance between manipulators i and z is safe; if D i，z <D th , then the manipulators i and z trigger collision avoidance control, and the collision avoidance control method is: If detected Increase the rotation angles of manipulators i and z to avoid each other; If detected: The avoidance path of the manipulator is extracted by constructing a dynamic window.

9. The ultra-micro fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 8, characterized in that: The method of increasing the rotation angle of the manipulator i and z is: Calculate the relative position vectors of manipulator i and z And get the unit vector of the movement direction of robot i and the unit vector of the robot's z direction of motion Calculate the angle between the manipulator i and z using the following formula: In the formula, AOI i，z is the unit vector of the motion direction of robot i and its relative position vector to the robot z The angle between i，z If AOI is greater than 0, the robot i will rotate clockwise. i，z Less than 0, robot i chooses to rotate counterclockwise; By rotating the avoidance direction, the moving direction of the robot is updated. The update formula is: In the formula, is the new movement direction of robot i; based on the new movement direction, the coordinate position of robot i is updated to obtain the updated coordinate position The update formula is: In the formula, is the movement speed of robot i; T s is the time interval.

10. The ultra-miniature fluorescence microscope visual monitoring tracking system based on mouse behavior recognition as claimed in claim 8, characterized in that: The method of extracting the avoidance path of the manipulator is: Set the dynamic speed range of the robot, including the maximum speed of the robot and minimum robot speed Among them, μ max is the maximum acceleration of the manipulator, Δt is the time step; the speed adjustment range of the manipulator is obtained Set the dynamic movement direction range of the robot, including the maximum movement direction of the robot and the direction of minimum movement in is the maximum rotational angular velocity of the manipulator; the movement direction adjustment range of the manipulator is obtained Extract all motion trajectories based on the speed adjustment range and the motion direction adjustment range, extract the new coordinate positions, and then evaluate them, including: Safety Assessment: like This means that the motion trajectories of manipulators i and z are safe; Target proximity assessment: Among them, (x tag ,y tag ) is the coordinate position of the target point; Balance Assessment: SMTH=|V κ -V robot |+DIR κ -DIR robot |; SMTH is the balance score; V κ It is the k-th speed value in the speed adjustment range; DIR κ The k-th motion direction in the motion direction adjustment range; Calculate the motion trajectory score S based on the above parameters TP , the calculation formula is: In the formula, η1 is the weight coefficient of safety assessment, η2 is the weight coefficient of target proximity assessment, and η3 is the weight coefficient of balance assessment; From all motion trajectories, select the motion trajectory score S TP The path with the largest value is used as the obstacle avoidance path, and the motion state of the robot is updated.

Images