Method, device, equipment and medium for analyzing video motion of model organisms
Through deep learning and three-dimensional reconstruction technology, the analysis of mode biological video actions is automatically solved, and the problems of low efficiency and poor accuracy in the existing technology are achieved, and efficient action analysis and the applicability of multiple mode biological organisms are achieved.
Patent Information
- Application Number
- CN202210522410.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-13
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-05-13
AI Technical Summary
In the prior art, the behavioral information recording efficiency of model organisms is low and the accuracy is poor, which affects the efficiency of motor behavior analysis, and lacks automated processing methods.
Using a video action analysis method based on deep learning, the category tags of image frames are extracted through feature extractors, dynamically planned and segmented video segments, combined with three-dimensional reconstruction technology, the action types of patterned creatures are automatically identified to achieve action analysis without human intervention.
It improves the accuracy and efficiency of pattern biological video action analysis, can be applied to multiple pattern biologicals, has strong scalability, and realizes automated analysis without manual intervention.
Smart Images

Figure CN115115967B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of biological image processing technology, and in particular to a method, device, equipment and medium for analyzing video motion of a model organism. Background Art
[0002] Model organisms are often used in biological research. For example, mice are a common model organism in biological research. Biological researchers often verify their experimental hypotheses by observing and recording the behavioral information of model organisms such as mice. However, most of the records in related technologies are still done manually by researchers, so there are characteristics of low efficiency and poor accuracy, which in turn affects the efficiency of subsequent motor behavior analysis. Summary of the invention
[0003] The present application provides a video motion analysis method, device, electronic device and storage medium for model organisms, which can automatically complete the motion analysis of model organism videos without any human intervention, and can be applied to a variety of model organisms with strong scalability.
[0004] The first aspect of the present application provides a method for analyzing video motions of a model organism, comprising the following steps: obtaining an image to be analyzed of a target model organism; inputting the image to be analyzed into a pre-trained feature extractor, extracting the category label of each image frame in the image to be analyzed, and dividing the image to be analyzed into multiple video segments based on the category label of each image frame, wherein the feature extractor is trained based on image samples carrying image frame category labels; calculating the category similarity between all video segments, and aggregating the video segments whose category similarity is greater than a preset value to obtain at least one aggregation result, and identifying the at least one aggregation result to obtain the actual action type of the target model organism.
[0005] Furthermore, the image to be analyzed is input into a pre-trained feature extractor to extract the category label of each image frame in the image to be analyzed, including: extracting the category features of each image frame and reducing the dimension of the category features to the target dimension; performing category clustering on the category features after dimensionality reduction, and assigning the same category label to image frames of the same category.
[0006] Furthermore, segmenting the image to be analyzed into multiple video segments based on the category label of each image frame includes: performing coarse-grained segmentation on the image to be analyzed through a dynamic programming strategy to obtain the multiple video segments, wherein the dynamic programming strategy is that the number of image frames in each video segment is less than a preset number of frames, and the category labels of all image frames in the video segment are consistent.
[0007] Furthermore, the identifying of at least one aggregation result to obtain the actual action type of the target model organism includes: acquiring the movement speed information of the target model organism obtained by three-dimensional reconstruction; reducing the dimensions of all aggregation results to two dimensions, and using the movement speed information as the third dimension of the reduced aggregation result to generate three-dimensional coordinates for each aggregation result; and visualizing and clustering the action types of all category explanation results based on the three-dimensional coordinates of each aggregation result to obtain the actual action type of the target model organism.
[0008] The second aspect of the present application provides a video motion analysis device for a model organism, comprising: an acquisition module for acquiring an image to be analyzed of a target model organism; an extraction module for inputting the image to be analyzed into a pre-trained feature extractor, extracting the category label of each image frame in the image to be analyzed, and segmenting the image to be analyzed into multiple video segments based on the category label of each image frame, wherein the feature extractor is trained based on image samples carrying image frame category labels; an analysis module for calculating the category similarity between all video segments, and aggregating video segments with category similarity greater than a preset value to obtain at least one aggregation result, and identifying the at least one aggregation result to obtain the actual action type of the target model organism.
[0009] Furthermore, the extraction module is used to: extract category features of each image frame and reduce the dimension of the category features to a target dimension; perform category clustering on the category features after dimensionality reduction and assign the same category label to image frames of the same category.
[0010] Furthermore, the extraction module is used to: perform coarse-grained segmentation on the image to be analyzed through a dynamic programming strategy to obtain the multiple video segments, wherein the dynamic programming strategy is that the number of image frames in each video segment is less than a preset number of frames, and the category labels of all image frames in the video segment are consistent.
[0011] Furthermore, the analysis module is used to: obtain the movement speed information of the target model organism obtained by three-dimensional reconstruction; reduce the dimensions of all aggregation results to two dimensions, and use the movement speed information as the third dimension of the aggregation results after dimensionality reduction to generate three-dimensional coordinates for each aggregation result; visualize all category explanation results and cluster the action types based on the three-dimensional coordinates of each aggregation result to obtain the actual action type of the target model organism.
[0012] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video motion analysis method of the model organism as described in the above embodiment.
[0013] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the video motion analysis method of a model organism as described in the above embodiment.
[0014] Therefore, this application has at least the following beneficial effects:
[0015] Based on deep learning, the model organism video motion analysis technology can automatically perform motion analysis by using the collected video of the model organism moving freely in a transparent breeding box and the model organism three-dimensional model data obtained by three-dimensional reconstruction. The three-dimensional model of the model organism obtained by three-dimensional reconstruction technology and the behavioral information of the model organism movement are obtained by deep learning unsupervised methods. The motion analysis of the model organism video can be automatically completed without any human intervention, which effectively improves the accuracy and efficiency of related research. It can also be applied to a variety of model organisms and has strong scalability.
[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0018] Figure 1 A schematic diagram of a flow chart of a video motion analysis method for a model organism provided in an embodiment of the present application;
[0019] Figure 2 This is an example diagram of a video motion analysis device for a model organism provided according to an embodiment of the present application;
[0020] Figure 3 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0021] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0022] Model organisms take mice as an example. Mice are one of the most commonly used model organisms in biological research, and the analysis of mouse behavior is a very important topic in biological research. The related technology of three-dimensional reconstruction enables the embodiment of the present application to obtain a reconstructed three-dimensional mouse model, thereby digitizing the video sequence of the mouse. If the embodiment of the present application can utilize the behavioral information of the reconstructed three-dimensional mouse to develop related automatic motion analysis technology, it can provide great convenience for biological researchers. In recent years, with the rapid development of deep learning technology, it has been widely used in various fields. Unsupervised video motion classification technology, as an emerging technology, has also gradually emerged in various major fields. Therefore, the embodiment of the present application can utilize related technology and use it in mouse experiments.
[0023] The following will describe the video motion analysis method, device, electronic device and storage medium of the model organism of the present application with reference to the accompanying drawings. Specifically, Figure 1 A schematic flow chart of a method for analyzing video motion of a model organism provided in an embodiment of the present application.
[0024] like Figure 1 As shown, the video motion analysis method of the model organism comprises the following steps:
[0025] In step S101 , an image to be analyzed of a target model organism is obtained.
[0026] Among them, the target model organism can be specifically selected according to the actual analysis needs, such as mice, etc., and there is no specific limitation on this; the image to be analyzed can be a long video sequence.
[0027] In step S102, the image to be analyzed is input into a pre-trained feature extractor, the category label of each image frame in the image to be analyzed is extracted, and the image to be analyzed is segmented into multiple video segments based on the category label of each image frame, wherein the feature extractor is trained based on image samples carrying image frame category labels.
[0028] The structure of the feature extractor is mainly based on MaskRCNN. The embodiment of the present application modifies the key point extraction network of MaskRCNN and changes it to the regression of 3D-pose. The data used for training is the result obtained by using the optimization method in the embodiment of the present application.
[0029] In an embodiment of the present application, the image to be analyzed is input into a pre-trained feature extractor to extract the category label of each image frame in the image to be analyzed, including: extracting the category features of each image frame and reducing the dimension of the category features to the target dimension; performing category clustering on the reduced category features and assigning the same category label to image frames of the same category.
[0030] The target dimension may be set according to actual conditions, for example, it may be two-dimensional, etc., and no specific limitation is made to this.
[0031] It is understandable that the embodiment of the present application can construct an unsupervised convolutional neural network as a feature extractor for video sequences, cluster the obtained features, and obtain category labels for small video segments.
[0032] Specifically, the embodiment of the present application can delete the last regression layer (linear layer) of Mask-RCNN, use its backbone (Resnet-50) as a feature extractor, use the feature extractor to obtain the features of each frame (for example, 256 dimensions), and then use umap to reduce the features to the target dimension, for example, 2 dimensions, and cluster the features after dimensionality reduction to assign a category label to each frame.
[0033] In an embodiment of the present application, the image to be analyzed is segmented into multiple video segments based on the category label of each image frame, including: coarse-grained segmentation of the image to be analyzed is performed through a dynamic programming strategy to obtain multiple video segments, wherein the dynamic programming strategy is that the number of image frames in each video segment is less than a preset number of frames, and the category labels of all image frames in the video segment are consistent.
[0034] The preset number of frames can be specifically set according to actual segmentation requirements and is not specifically limited thereto.
[0035] It is understandable that the embodiments of the present application can use the dynamic programming method to complete the coarse-grained segmentation and clustering of long video sequences to obtain video segmentation results, that is, category labels of small video segments.
[0036] Specifically, the embodiment of the present application can perform coarse-grained division according to the following two principles to obtain many video segments, where the principles are as follows: first, each video segment cannot exceed t frames; second, all frame labels of each video segment are consistent.
[0037] In step S103, the category similarities between all video segments are calculated, and the video segments with category similarities greater than a preset value are aggregated to obtain at least one aggregation result, and the at least one aggregation result is identified to obtain the actual action type of the target model organism.
[0038] It can be understood that after obtaining the video segments, the embodiment of the present application can use the DTAK algorithm to measure the similarity of the video segments and aggregate similar video segments to obtain the aggregated segmentation results, and identify the aggregated segmentation results to obtain the actual action type, where the actual action type is such as running, standing, etc., so as to facilitate biological researchers to conduct action analysis of model organisms.
[0039] In an embodiment of the present application, identifying at least one aggregation result to obtain the actual action type of the target model organism includes: obtaining the movement speed information of the target model organism obtained by three-dimensional reconstruction; reducing the dimensions of all aggregation results to two dimensions, and using the movement speed information as the third dimension of the reduced aggregation results to generate three-dimensional coordinates for each aggregation result; visualizing all category explanation results and clustering the action types based on the three-dimensional coordinates of each aggregation result to obtain the actual action type of the target model organism.
[0040] It can be understood that the embodiments of the present application can reduce the dimension of the video segmentation results obtained by clustering and visualize them. Specifically: video segments of the same category are aggregated, and the model biological motion speed information obtained by three-dimensional reconstruction is combined to interpret the categories of the aggregated video segments, and reduce the dimension for visualization.
[0041] Specifically, since the dimensions of the video segments after aggregation are large, this time, the embodiment of the present application can use the umap dimensionality reduction coordinates of all frames in the video segment as its two dimensions, and then use its average speed as the third dimension, so that all video segments can be represented as a three-dimensional coordinate. All video segments are visualized and clustered, and each class represents a movement category of a model organism.
[0042] It should be noted that the three-dimensional reconstruction is completed before the video analysis. In the reconstruction step, the embodiment of the present application retrains a network Mask-RCNN, which inputs a multi-view color picture of the model organism and outputs point cloud information including the rotation axis angle of the joint point of the organism. The color picture can obtain key point information. The embodiment of the present application can use an optimization method based on the key point information and the point cloud information to perform three-dimensional reconstruction of the model organism. After having a large number of reconstruction results, the results can be used to train the network, which can accelerate the subsequent reconstruction.
[0043] According to the video motion analysis method of model organisms proposed in the embodiments of the present application, based on the model organism video motion analysis technology of deep learning, and using the collected video of the model organism moving freely in a transparent breeding box and the model organism three-dimensional model data obtained by three-dimensional reconstruction thereof, motion analysis can be automatically performed, thereby using the three-dimensional model of the model organism obtained by the three-dimensional reconstruction technology, and using the deep learning unsupervised method to obtain the behavioral information of the model organism's movement, the motion analysis of the model organism video can be automatically completed without any human intervention, effectively improving the accuracy and efficiency of related research, and can be applied to a variety of model organisms with strong scalability.
[0044] Next, the video motion analysis device of the model organism proposed in the embodiment of the present application is described with reference to the accompanying drawings.
[0045] Figure 2It is a block diagram of a video motion analysis device for a model organism according to an embodiment of the present application.
[0046] like Figure 2 As shown, the video motion analysis device 10 of the model organism includes: an acquisition module 100, an extraction module 200 and an analysis module 300.
[0047] Among them, the acquisition module 100 is used to obtain the image to be analyzed of the target model organism; the extraction module 200 is used to input the image to be analyzed into a pre-trained feature extractor, extract the category label of each image frame in the image to be analyzed, and divide the image to be analyzed into multiple video segments based on the category label of each image frame, wherein the feature extractor is trained based on image samples carrying image frame category labels; the analysis module 300 is used to calculate the category similarity between all video segments, and aggregate video segments whose category similarity is greater than a preset value to obtain at least one aggregation result, and identify at least one aggregation result to obtain the actual action type of the target model organism.
[0048] Furthermore, the extraction module 200 is used to: extract the category features of each image frame, and reduce the dimension of the category features to the target dimension; perform category clustering on the category features after the dimension reduction, and assign the same category label to the image frames of the same category.
[0049] Furthermore, the extraction module 200 is used to: perform coarse-grained segmentation on the image to be analyzed through a dynamic programming strategy to obtain multiple video segments, wherein the dynamic programming strategy is that the number of image frames in each video segment is less than a preset number of frames, and the category labels of all image frames in the video segment are consistent.
[0050] Furthermore, the analysis module 300 is used to: obtain the movement speed information of the target model organism obtained by three-dimensional reconstruction; reduce the dimensions of all aggregation results to two dimensions, and use the movement speed information as the third dimension of the aggregation results after dimensionality reduction to generate three-dimensional coordinates for each aggregation result; visualize all category explanation results and cluster the action types based on the three-dimensional coordinates of each aggregation result to obtain the actual action type of the target model organism.
[0051] It should be noted that the above explanation of the embodiment of the video motion analysis method of the model organism is also applicable to the video motion analysis device of the model organism in this embodiment, and will not be repeated here.
[0052] According to the video motion analysis device of model organisms proposed in the embodiment of the present application, based on the model organism video motion analysis technology of deep learning, and using the collected video of the model organism moving freely in a transparent breeding box and the model organism three-dimensional model data obtained by three-dimensional reconstruction thereof, motion analysis can be automatically performed, thereby using the three-dimensional model of the model organism obtained by the three-dimensional reconstruction technology and the behavioral information of the model organism's movement using a deep learning unsupervised method, the motion analysis of the model organism video can be automatically completed without any human intervention, effectively improving the accuracy and efficiency of related research, and can be applied to a variety of model organisms with strong scalability.
[0053] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0054] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .
[0055] When the processor 302 executes the program, the video motion analysis method of the model organism provided in the above embodiment is implemented.
[0056] Furthermore, the electronic device further comprises:
[0057] The communication interface 303 is used for communication between the memory 301 and the processor 302 .
[0058] The memory 301 is used to store computer programs that can be run on the processor 302 .
[0059] The memory 301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.
[0060] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0061] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.
[0062] The processor 302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.
[0063] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for analyzing the video motion of a model organism.
[0064] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0065] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0066] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0067] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.
[0068] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0069] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for analyzing video motion of a model organism, characterized in that: The following steps are involved: Obtain images of target model organisms to be analyzed; Input the image to be analyzed into a pre-trained feature extractor, extract the category label of each image frame in the image to be analyzed, and divide the image to be analyzed into multiple video segments based on the category label of each image frame, wherein the feature extractor is trained based on image samples carrying image frame category labels; input the image to be analyzed into a pre-trained feature extractor, extract the category label of each image frame in the image to be analyzed, including: extracting category features of each image frame, and reducing the dimension of the category features to the target dimension; performing category clustering on the category features after dimensionality reduction, and assigning the same category label to image frames of the same category; dividing the image to be analyzed into multiple video segments based on the category label of each image frame, including: performing coarse-grained segmentation on the image to be analyzed through a dynamic programming strategy to obtain the multiple video segments, wherein the dynamic programming strategy is that the number of image frames in each video segment is less than a preset number of frames, and the category labels of all image frames in the video segment are consistent; Calculate the category similarity between all video segments, aggregate the video segments whose category similarity is greater than a preset value, obtain at least one aggregation result, and identify the at least one aggregation result to obtain the actual action type of the target model organism; the identification of the at least one aggregation result to obtain the actual action type of the target model organism includes: obtaining the movement speed information of the target model organism obtained by three-dimensional reconstruction; reducing the dimensions of all aggregation results to two dimensions, and using the movement speed information as the third dimension of the reduced aggregation results to generate the three-dimensional coordinates of each aggregation result; based on the three-dimensional coordinates of each aggregation result, visualize all category explanation results and cluster the action types to obtain the actual action type of the target model organism.
2. A video motion analysis device for a model organism, characterized in that: include: An acquisition module, used to acquire the image to be analyzed of the target model organism; An extraction module is used to input the image to be analyzed into a pre-trained feature extractor, extract the category label of each image frame in the image to be analyzed, and divide the image to be analyzed into multiple video segments based on the category label of each image frame, wherein the feature extractor is trained based on image samples carrying image frame category labels; input the image to be analyzed into a pre-trained feature extractor, extract the category label of each image frame in the image to be analyzed, including: extracting the category features of each image frame, and reducing the dimension of the category features to the target dimension; performing category clustering on the category features after dimensionality reduction, and assigning the same category label to image frames of the same category; dividing the image to be analyzed into multiple video segments based on the category label of each image frame, including: performing coarse-grained segmentation on the image to be analyzed through a dynamic programming strategy to obtain the multiple video segments, wherein the dynamic programming strategy is that the number of image frames in each video segment is less than a preset number of frames, and the category labels of all image frames in the video segment are consistent; The analysis module is used to calculate the category similarity between all video segments, aggregate the video segments whose category similarity is greater than a preset value, obtain at least one aggregation result, and identify the at least one aggregation result to obtain the actual action type of the target model organism; the analysis module is used to: obtain the movement speed information of the target model organism obtained by three-dimensional reconstruction; reduce the dimensions of all aggregation results to two dimensions, and use the movement speed information as the third dimension of the reduced aggregation results to generate the three-dimensional coordinates of each aggregation result; visualize all category explanation results and cluster the action types based on the three-dimensional coordinates of each aggregation result to obtain the actual action type of the target model organism.
3. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video motion analysis method of a model organism as claimed in claim 1.
4. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the video motion analysis method of a model organism as claimed in claim 1.
Citation Information
Patent Citations
Behavior recognition device and method
CN111652133A
Video feature extraction method and device, computer equipment and storage medium
CN114329070A
Action recognition method and apparatus, device, and medium
WO2021184468A1
