Method and apparatus for evaluating viability of an organism

Abstract

The application discloses a kind of worm activity evaluation method and device, using SAM-Track method to carry out instance segmentation and tracking to worm in short video, accurate video containing worm only is obtained by removing background, then regression training is carried out to pre-trained VideoMAE model, the powerful space-time feature extraction capability of VideoMAE is used, automatically extract the feature representing worm activity in video, realize end-to-end praziquantel concentration prediction, finally, automatic evaluation of video worm activity is realized by sigmoid mapping.The blood fluke activity quantification evaluation method based on short video is realized, this method can be used to screen very large anti-schistosome compound set, and the principle is applicable to other parasites.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, and in particular relates to a method and device for assessing insect vitality. Background Technology

[0002] Schistosomiasis is a neglected disease of poverty caused by infection with Schistosoma species. For the past 40 years, schistosomiasis control in endemic areas has primarily relied on the use of a single drug, praziquantel. Due to the limitations of this single-agent chemotherapy approach in maintaining schistosomiasis control in the future, the drug discovery community is increasingly seeking alternative anti-schistosomiasis compounds. Anti-schistosomiasis compounds belong to the category of small-molecule inhibitors in drug discovery. Phenotypic screening of whole organisms in in vitro and / or animal models is the most commonly used method for finding lead compounds (i.e., screening in vitro viability compounds based on defined activity thresholds). The development of reliable high-throughput screening (HTS) tools promises to accelerate the identification of novel anti-schistosomiasis compounds. The paper "Paveley RA, Mansour NR, Hallyburton I, et al. Whole organism high-content screening by label-free, image-based Bayesian classification for parasitic diseases[J]. 2012." demonstrates high-throughput screening of schistosomes in a 384-well format. Using an automated high-content imaging platform that quantifies schistosome phenotype and motility, it rapidly collects large quantities of compounds for antischistosome drug classification. They evaluated compound efficacy through a combination of quantified larval motility and morphology using Bayesian analysis. In this assay, the high-throughput screening system (HCS) collected bright-field images of each well in a 384-well microplate in two different modes: time-lapse image acquisition using a 4x objective for motion analysis, and simultaneous acquisition of four adjacent images using a 10x objective for morphological measurements. Image analysis was performed using Pipeline Pilot 8.5 software (Accelrys Inc., San Diego, USA), including a series of sequential image operations such as thresholding, filtering, and boundary detection before worm segmentation. The morphological (e.g., area, texture, pixel intensity) and motion-related features of each schistosome were quantified and summarized into phenotypic (processed using a Bayesian model) and motion scores, respectively. The final score was obtained by averaging the scores of all parasites in each well. A test compound was considered a "hit" if the phenotypic and motion scores exceeded the defined thresholds for each metric.

[0003] In vitro assays can now measure the effects of compounds on development, growth, behavior, and locomotion. Automated phenotypic analysis of short video recordings provides a useful analytical method for high-throughput whole-organism phenotypic screening and avoids manual scoring. Past methods for assessing schistosome locomotion were manual and low-throughput; some methods use optical flow algorithms to automatically screen for parasite locomotion. These methods provide quantitative readings, unaffected by subjective scoring, but they do not necessarily provide an assessment of worm viability, as paralyzed worms may still be alive and recover after drug treatment. This invention aims to provide a solution to this problem.

[0004] Existing video-based whole-organism phenotypic screening systems suffer from complex image processing and insufficient image segmentation accuracy, leading to inaccurate feature extraction. Furthermore, determining the screening threshold requires quantifying multiple morphological and motion-related features separately. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for assessing insect vitality. Through accurate and efficient image segmentation and end-to-end vitality identification, it provides an accurate quantitative analysis method for high-throughput whole-organism phenotypic screening. This invention uses the SAM-Track method to segment and track insects in short videos, obtaining videos that accurately remove the background and contain only the insects. Then, leveraging the powerful spatiotemporal feature extraction capabilities of the pre-trained VideoMAE model, it automatically extracts features representing insect vitality from the video, and achieves automatic assessment of insect vitality through sigmoid mapping.

[0006] According to a first aspect of the embodiments of this application, a method for assessing insect vitality is provided, comprising: (1) Obtain videos of schistosomes treated with different concentrations of praziquantel and establish a short video dataset labeled with concentration; (2) Based on the short video dataset, the SAM-Track method is used to segment and track the insect instances to obtain the video instance segmentation results; (3) Based on the video instance segmentation results, select videos of individual schistosomes and construct a schistosome video dataset using the copy-paste method; (4) Based on the schistosomiasis video dataset, the pre-trained VideoMAE classification model is regressed using mean squared error loss, and the trained VideoMAE regression model is used to predict the praziquantel concentration corresponding to the schistosomiasis video dataset. (5) Based on the predicted praziquantel concentration, establish a mapping from praziquantel concentration to insect viability; (6) Obtain the video of the worm to be evaluated, segment and track the worm using the SAM-Track method, obtain the segmentation result of the worm to be evaluated, use the trained VideoMAE regression model to predict the concentration of praziquantel, and obtain the corresponding worm vitality through the established mapping.

[0007] Further, step (2) includes: Using "worms" as a cue word, we detect worms in all video frames of each short video using Grounding DINO to obtain worm bounding boxes; By using the bounding boxes of each detected insect as cues for segmenting arbitrary object models, insect instances in all video frames can be segmented. Use DeAOT to track the segmentation results of each insect instance.

[0008] Furthermore, the arbitrary object segmentation model includes an image encoder, a cue encoder, and a mask decoder. The image encoder is used to receive input video frames and calculate image embeddings. The cue encoder is used to receive the insect body bounding box as a cue and calculate cue embeddings. The mask decoder is used to predict a segmentation mask, i.e., the segmentation result of the current video frame, based on the image embeddings and cue embeddings.

[0009] Further, step (3) includes: Multiply the original video corresponding to the segmentation result of the video instance with the segmented video to obtain a short video containing only the insect body after removing the background; Several schistosomes were randomly selected from the set of individual schistosomes corresponding to each concentration, and the data augmentation was performed using the copy-paste method to obtain a schistosome video dataset.

[0010] Furthermore, in step (4), during the regression training of the pre-trained VideoMAE classification model, the schistosomiasis video dataset is used as the training input, the corresponding praziquantel concentration is used as the training output, and the loss function is the mean squared error loss.

[0011] Further, in step (4), the trained VideoMAE regression model is used to equate the insect viability after treatment with any concentration of drug for any time to the viability after treatment with a certain concentration of praziquantel for 24 hours, and the equivalent concentration is used as the predicted concentration.

[0012] Furthermore, in step (5), the sigmoid function is used to establish a mapping function between praziquantel concentration and insect viability.

[0013] According to a second aspect of the embodiments of this application, a device for assessing insect vitality is provided, comprising: The dataset creation module is used to acquire videos of schistosomes treated with different concentrations of praziquantel and to create a short video dataset labeled with concentration. The insect segmentation module is used to segment and track insect instances based on the short video dataset using the SAM-Track method to obtain video instance segmentation results; The data augmentation module is used to filter out individual schistosome videos based on the video instance segmentation results and construct a schistosome video dataset using the copy-paste method. The model training module is used to perform regression training on the pre-trained VideoMAE classification model based on the schistosomiasis video dataset using mean squared error loss. The trained VideoMAE regression model can be used to predict praziquantel concentration. The mapping module is used to establish a mapping from praziquantel concentration to insect viability based on the predicted praziquantel concentration. The insect viability assessment module is used to acquire videos of the insects to be assessed, segment and track the insects using the SAM-Track method to obtain the segmentation results of the insects to be assessed, predict the praziquantel concentration using the trained VideoMAE regression model, and obtain the corresponding insect viability through the established mapping.

[0014] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0015] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects: As can be seen from the above embodiments, this application leverages fundamental models in the field of computer vision to achieve efficient and accurate segmentation and tracking of worm instances; utilizes the copy-paste method to perform data augmentation on short videos containing only worms after removing the background, reducing the number of short video samples that need to be collected; leverages the powerful spatiotemporal feature representation capabilities of the pre-trained VideoMAE to achieve end-to-end praziquantel concentration recognition on a small dataset of schistosomiasis videos; and implements a quantification method for schistosomiasis viability based on short videos, which can be used to screen a very large set of anti-schistosomiasis compounds, and the principle is applicable to other parasites.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 This is a block diagram illustrating a method for assessing insect vitality according to an exemplary embodiment.

[0020] Figure 2 This is an example diagram of schistosome video worm instance segmentation according to an exemplary embodiment.

[0021] Figure 3 This is an example image of data augmentation of a short video (0.38 μM concentration) with background removed and containing only insects, according to an exemplary embodiment.

[0022] Figure 4 This is a Pearson correlation graph illustrating the predicted praziquantel concentration versus the applied concentration (0-1 μM) according to an exemplary embodiment.

[0023] Figure 5 This is a mapping of praziquantel concentration to insect viability, illustrated according to an exemplary embodiment.

[0024] Figure 6 This is a block diagram illustrating an insect vitality assessment device according to an exemplary embodiment.

[0025] Figure 7 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] like Figure 1 As shown, this application provides a method for assessing the viability of schistosomes. The following description uses schistosome viability assessment as an example. The method includes the following steps: (1) Obtain videos of schistosomes treated with different concentrations of praziquantel and establish a short video dataset labeled with concentration; In one embodiment, videos were collected 24 hours after schistosomes were treated with praziquantel at concentrations of 0.1, 0.20, 0.26, 0.28, 0.30, 0.32, 0.34, 0.36, 0.38, 0.40, 0.50, 0.60, 0.80, and 1.00 μM, respectively. The acquisition time was 10 seconds, and three video samples were collected for each concentration.

[0030] (2) Based on the short video dataset, the SAM-Track method is used to segment and track the insect instances to obtain the video instance segmentation results; Specifically, using "worms" as a cue word, the bounding boxes of each worm are obtained by detecting worms in the first frame of each short video using Grounding DINO. These detected bounding boxes are then used as cues for segmenting an arbitrary object model (SAM) to achieve worm instance segmentation in the current video frame. Figure 2 SAM consists of three parts: an image encoder, a cue encoder, and a mask decoder. The image encoder receives input video frames and calculates image embeddings. The cue encoder receives cues (i.e., the bounding boxes of each schistosome) and obtains cue embeddings. The lightweight mask decoder predicts a segmentation mask based on the image embeddings and cue embeddings to obtain the segmentation result for the current frame. For subsequent frames, schistosomes are segmented in the same way, and then DeAOT is used to track the segmentation results of each schistosome instance. This integrated approach ensures accurate and efficient instance segmentation and tracking of individual schistosomes automatically throughout the video.

[0031] (3) Based on the video instance segmentation results, select videos of individual schistosomes and construct a schistosome video dataset using the copy-paste method; Specifically: the original video in the segmented instance data is multiplied by the segmented video to obtain a short video containing only the insect body after removing the background. Figure 3 Based on the instance segmentation results, videos of each schistosome at each concentration (background removed) can be obtained. Referring to the copy-paste method, a blank 3D array of size 50*1824*1216 is created as the background for the enhanced samples. Then, for each concentration, 4 to 8 schistosomes are randomly selected from all the segmented single schistosome sets for that concentration, and they are pasted into the 3D array to obtain the enhanced samples for that concentration. In this embodiment, this process is repeated to randomly generate 30 enhanced samples for each concentration.

[0032] (4) Based on the schistosomiasis video dataset, the pre-trained VideoMAE classification model is regressed using mean squared error loss, and the trained VideoMAE regression model is used to predict the praziquantel concentration corresponding to the schistosomiasis video dataset. As a self-supervised video pre-training data-efficient learning method, VideoMAE does not utilize label information but learns spatiotemporal representation information from video data through masking and reconstruction tasks. This application uses a VideoMAE classification model pre-trained on Kinetics-400 as the initial model, and performs supervised training on the labeled overlapping insect segmentation dataset using concentration regression. The loss function is the mean squared error loss. The input to VideoMAE is the original video multiplied by the segmented video to obtain a short video containing only insects after background removal; the output is the applied praziquantel concentration. After the model is trained, the insect viability after treatment with any concentration of the drug for any time can be equated to the viability after treatment with a certain concentration of praziquantel for 24 hours.

[0033] The trained VideoMAE was used to predict the praziquantel concentration corresponding to the schistosomiasis video dataset.

[0034] (5) Based on the predicted praziquantel concentration, establish a mapping from praziquantel concentration to insect viability; Based on the concentration prediction results in step (4), a mapping function between praziquantel concentration and insect viability is established using the sigmoid function, ultimately realizing automatic assessment of insect viability based on short videos.

[0035] Specifically, based on experimental data, such as Figure 4 As shown, within the concentration range of 0-1 μM, the concentrations predicted by VideoMAE on the validation dataset exhibit a good correlation with the actual concentrations (correlation coefficient of 0.977); from Figure 4It can be seen that nonlinearity mainly occurs near the concentration of 0 and at concentrations greater than 0.6 μM, which is consistent with the observation of insect activity in videos at these concentrations by the human eye. The videos show that at concentrations greater than 0.6 μM, the insects are basically motionless after 24 hours; while at extremely low concentrations, the human eye can hardly distinguish the difference in insect activity between videos of different concentrations.

[0036] Based on this, such as Figure 5 As shown, when the concentration is greater than 0.6 μM, the insect viability is set to 0.05; when the concentration is 0, the insect viability is set to 1. The sigmoid function is used to map the praziquantel concentration to insect viability. Thus, using the praziquantel concentration at 24 hours as an intermediate value, the insect viability after treatment with any concentration of the drug for any time is equivalent to the viability after treatment with a certain concentration of praziquantel for 24 hours, providing an accurate quantitative analysis method for high-throughput whole-organism phenotypic screening.

[0037] (6) Obtain the video of the worm to be evaluated, use the SAM-Track method to segment and track the worm, obtain the segmentation result of the worm to be evaluated, use the trained VideoMAE regression model to predict the praziquantel concentration, and obtain the corresponding worm vitality through the established mapping. Specifically, the process involves acquiring videos of the worms to be evaluated, segmenting and tracking the worms using the SAM-Track method to obtain the segmentation results, predicting the corresponding praziquantel concentration using a trained VideoMAE classification model, and finally obtaining the worm viability through an established mapping between praziquantel concentration and worm viability. Corresponding to the aforementioned embodiments of the insect vitality assessment method, this application also provides embodiments of the insect vitality assessment device.

[0038] Figure 6 This is a block diagram illustrating an insect viability assessment device according to an exemplary embodiment. (Refer to...) Figure 6 The device may include: The dataset building module 21 is used to acquire videos of schistosomes treated with different concentrations of praziquantel and to build a short video dataset labeled with concentration. The insect segmentation module 22 is used to segment and track insect instances using the SAM-Track method based on the short video dataset to obtain video instance segmentation results; Data augmentation module 23 is used to filter out videos of a single schistosome based on the video instance segmentation results and construct a schistosome video dataset using the copy-paste method; Model training module 24 is used to perform regression training on the pre-trained VideoMAE classification model based on the schistosomiasis video dataset using mean squared error loss, and the trained VideoMAE regression model can be used to predict praziquantel concentration. The mapping module 25 is used to establish a mapping from praziquantel concentration to insect viability based on the predicted praziquantel concentration. The insect vitality assessment module 26 is used to acquire videos of the insects to be assessed, segment and track the insects using the SAM-Track method, obtain the segmentation results of the insects to be assessed, predict the praziquantel concentration using the trained VideoMAE regression model, and obtain the corresponding insect vitality through the established mapping.

[0039] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0040] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0041] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the insect viability assessment method as described above. Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including a worm vitality assessment device provided in an embodiment of the present invention. Except for... Figure 7 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0042] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the insect viability assessment method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0043] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0044] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for assessing insect viability, characterized in that, include: (1) Obtain videos of schistosomes treated with different concentrations of praziquantel and establish a short video dataset labeled with concentration; (2) Based on the short video dataset, the SAM-Track method is used to segment and track the insect instances to obtain the video instance segmentation results; (3) Based on the video instance segmentation results, select videos of individual schistosomes and construct a schistosome video dataset using the copy-paste method; (4) Based on the schistosomiasis video dataset, the pre-trained VideoMAE classification model is regressed using mean squared error loss, and the trained VideoMAE regression model is used to predict the praziquantel concentration corresponding to the schistosomiasis video dataset. (5) Based on the predicted praziquantel concentration, establish a mapping from praziquantel concentration to insect viability; (6) Obtain the video of the worm to be evaluated, segment and track the worm using the SAM-Track method, obtain the segmentation result of the worm to be evaluated, use the trained VideoMAE regression model to predict the concentration of praziquantel, and obtain the corresponding worm vitality through the established mapping.

2. The method according to claim 1, characterized in that, Step (2) includes: Using "worms" as a cue word, we detect worms in all video frames of each short video using Grounding DINO to obtain the worm bounding boxes; By using the bounding boxes of each detected insect as cues for segmenting arbitrary object models, insect instances in all video frames can be segmented. Use DeAOT to track the segmentation results of each insect instance.

3. The method according to claim 2, characterized in that, The arbitrary object segmentation model includes an image encoder, a cue encoder, and a mask decoder. The image encoder is used to receive input video frames and calculate image embeddings. The cue encoder is used to receive the insect body bounding box as a cue and calculate cue embeddings. The mask decoder is used to predict a segmentation mask, i.e., the segmentation result of the current video frame, based on the image embeddings and cue embeddings.

4. The method according to claim 1, characterized in that, Step (3) includes: Multiply the original video corresponding to the segmentation result of the video instance with the segmented video to obtain a short video containing only the insect body after removing the background; Several schistosomes were randomly selected from the set of individual schistosomes corresponding to each concentration, and data augmentation was performed using the copy-paste method to obtain a schistosome video dataset.

5. The method according to claim 1, characterized in that, In step (4), during the regression training of the pre-trained VideoMAE classification model, the schistosomiasis video dataset is used as the training input, the corresponding praziquantel concentration is used as the training output, and the loss function is the mean squared error loss.

6. The method according to claim 1, characterized in that, In step (4), the trained VideoMAE regression model is used to equate the insect viability after treatment with any concentration of drug for any time to the viability after treatment with a certain concentration of praziquantel for 24 hours, and the equivalent concentration is used as the predicted concentration.

7. The method according to claim 1, characterized in that, In step (5), the sigmoid function is used to establish a mapping function between praziquantel concentration and insect viability.

8. A device for assessing insect vitality, characterized in that, include: The dataset creation module is used to acquire videos of schistosomes treated with different concentrations of praziquantel and to create a short video dataset labeled with concentration. The insect segmentation module is used to segment and track insect instances based on the short video dataset using the SAM-Track method to obtain video instance segmentation results; The data augmentation module is used to filter out individual schistosome videos based on the video instance segmentation results and construct a schistosome video dataset using the copy-paste method. The model training module is used to perform regression training on the pre-trained VideoMAE classification model based on the schistosomiasis video dataset using mean squared error loss. The trained VideoMAE regression model can be used to predict praziquantel concentration. The mapping module is used to establish a mapping from praziquantel concentration to insect viability based on the predicted praziquantel concentration. The insect viability assessment module is used to acquire videos of the insects to be assessed, segment and track the insects using the SAM-Track method to obtain the segmentation results of the insects to be assessed, predict the praziquantel concentration using the trained VideoMAE regression model, and obtain the corresponding insect viability through the established mapping.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images