Nematode short-term posture behavior prediction method and system based on deep learning

Abstract

This invention belongs to the field of image processing and provides a method and system for predicting the short-term posture and behavior of nematodes based on deep learning. The method includes acquiring and preprocessing the original image of the nematode to obtain a preprocessed nematode image; extracting the nematode contour based on the preprocessed nematode image to obtain a nematode contour image; calculating the nematode shape vector based on the nematode contour image to obtain a standardized nematode shape vector sequence; and predicting the nematode posture using a pre-trained nematode posture and shape prediction model based on the standardized nematode shape vector sequence to obtain the short-term posture and behavior prediction result. This invention provides a shape description method for quantitatively describing and characterizing the body shape of nematodes, which simplifies the behavior and body posture of nematodes, ensures the uniqueness of the nematode body shape, and improves the accuracy of model prediction.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for predicting the short-term posture behavior of nematodes based on deep learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous development of computer vision and deep learning technologies, pose estimation and behavior prediction have become relatively simple. Both human and animal pose estimation rely on keypoint detection methods.

[0004] For human pose prediction, body shape estimation and behavior recognition are performed by detecting and locating the coordinates of skeletal keypoints. However, these methods rely on a large dataset containing labeled coordinates of skeletal keypoints, making the task of labeling keypoints more challenging and time-consuming. While many keypoint detection technologies exist, they impose strict requirements on image quality and resolution. Furthermore, directly feeding images containing keypoint location information into the model for prediction inevitably increases the model's prediction time, and the accuracy of behavior prediction is also affected by image quality.

[0005] For nematodes, their small size, simple nervous system, and ease of cultivation make them one of the most widely used model organisms in animal behavior analysis. Nematodes are typically cultured in laboratory petri dishes. Food and noise around the nematode can interfere with image quality. Secondly, there is currently no pre-labeled dataset available for our experiments. The coordinates of key skeletal points in nematodes are not as clearly defined as those in humans. Therefore, labeling key skeletal points in nematodes is a tedious and massive task, and the placement of each key point can be influenced by human subjectivity. This significantly impacts the accuracy of experimental results and the reliability of the dataset.

[0006] Although nematodes have a simple body structure, they exhibit a complex movement pattern. Generally, nematodes move in a near-sine curve pattern, exhibiting periodic repetitive motion. To capture the movement trends, body shape changes, and these unique periodic variations of nematodes, previous human posture and behavior prediction models cannot be directly applied to the prediction of nematode behavior (shape vectors). Because of the periodicity and repetitiveness of nematode movement, this aspect significantly contributes to the results of nematode behavior prediction.

[0007] Previous shape descriptions typically employed a mathematical model to characterize the sinusoidal motion of nematodes, matching shape changes through parameter variations. However, nematode motion is often complex and diverse (still exhibiting periodicity and repetition), making it difficult for such models to mathematically describe all nematode shapes. Furthermore, proposing an ideal model to describe and accurately fit the shape curves of nematodes is extremely challenging. Previous pose and shape predictions relied on existing labeled datasets. Currently, there is no open dataset available for nematodes, further complicating the labeling task, and experimental results are also affected by image quality and manual annotation. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a deep learning-based method and system for predicting the short-term posture and behavior of nematodes. This invention adds a Time2Vec module to the existing prediction model to better capture the periodic and non-periodic movements of nematodes. Furthermore, the hierarchical structure of Layer Normalization is modified to accelerate model training. Finally, TS-former is provided to predict future nematode posture and behavior, establishing a network model with long-term dependencies to improve the accuracy of nematode shape prediction.

[0009] According to some embodiments, the first aspect of the present invention provides a method for predicting the short-term posture behavior of nematodes based on deep learning, employing the following technical solution:

[0010] The original image of the nematode is acquired and preprocessed to obtain the preprocessed image of the nematode.

[0011] Based on the preprocessed nematode image, the nematode contour is extracted to obtain the nematode contour image.

[0012] The nematode shape vector is calculated from the nematode outline image to obtain a standardized nematode shape vector sequence;

[0013] Based on the standardized nematode shape vector sequence, the nematode posture is predicted using a pre-trained nematode posture shape prediction model, and the short-term posture behavior prediction results of the nematode are obtained.

[0014] The nematode posture and shape prediction model includes a time feature encoding module, an encoder, and a decoder.

[0015] The time feature encoding module performs time encoding on the standardized nematode shape vector sequence, embedding the time features into the standardized nematode shape vector sequence to obtain an intermediate sequence; the intermediate sequence features are further encoded at position before being input into the encoder to obtain a nematode shape vector sequence containing time and position.

[0016] Further, the step of acquiring the original image of the nematode and performing preprocessing to obtain the preprocessed nematode image includes:

[0017] Based on the original nematode image, median filtering was used to remove image noise.

[0018] Image binarization is performed on the nematode image after noise removal;

[0019] For the binarized nematode image, the morphological closure operator is used to remove spots and disconnected regions outside the nematode body to obtain the nematode image after removing spots.

[0020] For the nematode image after spot removal, the component labeling order algorithm is used to retain the largest connected region in the nematode image, resulting in a preprocessed nematode image.

[0021] Further, the step of extracting nematode contours from the preprocessed nematode image to obtain a nematode contour image includes:

[0022] Based on the preprocessed nematode image, the contour points are resampled uniformly at fixed intervals using a linear interpolation method.

[0023] Obtain the coordinates of the points on both sides of the nematode outline arranged in sequence, and store them sequentially in the array index;

[0024] The center coordinates of the two sides of the nematode outline are extracted from the coordinate points on both sides of the outline, and the skeleton curve of the nematode is obtained by using the cubic spline interpolation algorithm.

[0025] Based on the skeletal curves of the nematode, the outline image of the nematode is obtained.

[0026] Further, the step of calculating the nematode shape vector based on the nematode contour image to obtain a standardized nematode shape vector sequence includes:

[0027] Based on the nematode outline image, multiple coordinate points are set at equal intervals on the nematode skeleton curve, with the head coordinates marked as p;

[0028] Let the next two coordinate points be a and q respectively. Calculate the angle θ1 formed between points a and p and q, and determine the supplementary angle β1 of θ1.

[0029] Keep the head coordinate p fixed, move a and q by one coordinate point, then calculate the angle θ2 formed by points a and p and q, and determine the supplementary angle β2 of θ2;

[0030] Similarly, iteratively calculate the angle formed by points a and p, q, and calculate the supplementary angle of the angle;

[0031] The calculation of the coordinate points of the nematode skeleton curve is completed, resulting in an angle sequence containing multiple supplementary angles β.

[0032] Normalize all included angles in the angle sequence containing multiple supplementary angles β to obtain a standardized nematode shape vector sequence.

[0033] Furthermore, the time feature encoding module uses the time2vec algorithm to perform time encoding on the nematode shape vector and time vector features, embedding the time features into the nematode shape vector to obtain intermediate sequence features.

[0034] Furthermore, the encoder consists of two identical multi-head self-sparse attention layers stacked together; each multi-head self-sparse attention layer is followed by a distillation layer and a fully connected layer, which concatenates the feature maps of the distilled sequence as the input to the decoder.

[0035] Furthermore, the decoder is composed of four stacked decoder layers; each decoder layer consists of three sub-layer connection structures:

[0036] The connection structure between the first and second sub-layers includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection layer;

[0037] The third sublayer consists of a feedforward fully connected sublayer, a normalization layer, and a residual connected layer.

[0038] According to some embodiments, a second aspect of the present invention provides a deep learning-based system for predicting the short-term posture behavior of nematodes, employing the following technical solution:

[0039] A deep learning-based system for predicting the short-term posture and behavior of nematodes includes:

[0040] The image acquisition module is configured to acquire raw images of nematodes and perform preprocessing to obtain preprocessed images of nematodes.

[0041] The contour extraction module is configured to extract nematode contours based on the preprocessed nematode image to obtain a nematode contour image.

[0042] The image processing module is configured to calculate the nematode shape vector based on the nematode outline image, and obtain a standardized nematode shape vector sequence;

[0043] The nematode pose prediction module is configured to predict nematode pose based on a standardized nematode shape vector sequence and a pre-trained nematode pose shape prediction model to obtain short-term pose behavior prediction results for nematodes.

[0044] The nematode posture and shape prediction model includes a time feature encoding module, an encoder, and a decoder.

[0045] The time feature encoding module performs time encoding on the standardized nematode shape vector sequence, embedding the time features into the standardized nematode shape vector sequence to obtain intermediate sequence features; the intermediate sequence features are further encoded at position before being input into the encoder to obtain a nematode shape vector sequence containing time and position.

[0046] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0047] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in the deep learning-based method for predicting short-term pose behavior of nematodes as described in the first aspect above.

[0048] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0049] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the deep learning-based method for predicting short-term pose behavior of nematodes as described in the first aspect above.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0051] This invention adds a Time2Vec module to the existing prediction model to better capture the periodic and non-periodic movements of nematodes. Additionally, the hierarchical structure of Layer Normalization is modified to accelerate model training. Finally, TS-former is provided to predict the future posture and behavior of nematodes, establishing a network model with long-term dependencies to improve the accuracy of nematode shape prediction.

[0052] This invention does not simply use images or images with keypoint markers to predict the shape of nematodes. Instead, it obtains parameterized vectors of the nematode in each frame of a video using a shape representation method. This accurately depicts the shape of the nematode, and then uses the sequence data composed of these shape vectors to predict subsequent shape changes. Using a center curve to represent the shape of the nematode simplifies the representation and makes it easier to parameterize the nematode in one-dimensional space. Compared to previous methods using mathematical models for shape description, this method is simpler and more convenient. Furthermore, the method ensures the uniqueness of the shape, which remains unchanged regardless of image rotation or translation. It also fully preserves the sequence features of the nematode's shape and posture, and completely reflects and retains the periodicity and repetitiveness of the nematode. Attached Figure Description

[0053] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0054] Figure 1 This is a flowchart of the deep learning-based short-term posture behavior prediction method for nematodes described in this embodiment of the invention;

[0055] Figure 2 This is a structural diagram of the nematode posture and shape prediction model described in an embodiment of the present invention;

[0056] Figure 3 This is a modified Layer Normalization structure diagram as described in an embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0060] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0061] Example 1

[0062] like Figure 1As shown, this embodiment provides a method for predicting the short-term posture behavior of nematodes based on deep learning. This embodiment uses the application of this method to a server as an example for illustration. It can be understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through the interaction between the terminal and the server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0063] The original image of the nematode is acquired and preprocessed to obtain the preprocessed image of the nematode.

[0064] Based on the preprocessed nematode image, the nematode contour is extracted to obtain the nematode contour image.

[0065] The nematode shape vector is calculated from the nematode outline image to obtain a standardized nematode shape vector sequence;

[0066] Based on the standardized nematode shape vector sequence, the nematode posture is predicted using a pre-trained nematode posture shape prediction model, and the short-term posture behavior prediction results of the nematode are obtained.

[0067] The nematode posture and shape prediction model includes a time feature encoding module, an encoder, and a decoder.

[0068] The time feature encoding module performs time encoding on the standardized nematode shape vector sequence, embedding the time features into the standardized nematode shape vector sequence to obtain an intermediate sequence; the intermediate sequence features are further encoded at position before being input into the encoder to obtain a nematode shape vector sequence containing time and position.

[0069] The process of acquiring the original image of the nematode and preprocessing it to obtain the preprocessed image of the nematode includes:

[0070] Based on the original nematode image, median filtering was used to remove image noise.

[0071] Image binarization is performed on the nematode image after noise removal;

[0072] For the binarized nematode image, the morphological closure operator is used to remove spots and disconnected regions outside the nematode body to obtain the nematode image after removing spots.

[0073] For the nematode image after spot removal, the component labeling order algorithm is used to retain the largest connected region in the nematode image, resulting in a preprocessed nematode image.

[0074] The step of extracting nematode contours from the preprocessed nematode image to obtain a nematode contour image includes:

[0075] Based on the preprocessed nematode image, the contour points are resampled uniformly at fixed intervals using a linear interpolation method.

[0076] Obtain the coordinates of the points on both sides of the nematode outline arranged in sequence, and store them sequentially in the array index;

[0077] The center coordinates of the two sides of the nematode outline are extracted from the coordinate points on both sides of the outline, and the skeleton curve of the nematode is obtained by using the cubic spline interpolation algorithm.

[0078] Based on the skeletal curves of the nematode, the outline image of the nematode is obtained.

[0079] The step of calculating the nematode shape vector from the nematode outline image to obtain a standardized nematode shape vector sequence includes:

[0080] Based on the nematode outline image, multiple coordinate points are set at equal intervals on the nematode skeleton curve, with the head coordinates marked as p;

[0081] Let the next two coordinate points be a and q respectively. Calculate the angle θ1 formed between points a and p and q, and determine the supplementary angle β1 of θ1.

[0082] Keep the head coordinate p fixed, move a and q by one coordinate point, then calculate the angle θ2 formed by points a and p and q, and determine the supplementary angle β2 of θ2;

[0083] Similarly, iteratively calculate the angle formed by points a and p, q, and calculate the supplementary angle of the angle;

[0084] The calculation of the coordinate points of the nematode skeleton curve is completed, resulting in an angle sequence containing multiple supplementary angles β.

[0085] Normalize all included angles in the angle sequence containing multiple supplementary angles β to obtain a standardized nematode shape vector sequence.

[0086] The time feature encoding module uses the time2vec algorithm to encode the nematode shape vector and time vector features, embedding the time features into the nematode shape vector to obtain intermediate sequence features.

[0087] The encoder consists of two identical multi-head self-sparse attention layers stacked together; each multi-head self-sparse attention layer is followed by a distillation layer and a fully connected layer, which concatenates the feature maps of the distilled sequence as the input to the decoder.

[0088] The decoder consists of four stacked decoder layers; each decoder layer comprises three sub-layer interconnect structures:

[0089] The connection structure between the first and second sub-layers includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection layer;

[0090] The third sublayer consists of a feedforward fully connected sublayer, a normalization layer, and a residual connected layer.

[0091] like Figure 1 As shown, in a specific embodiment, the method includes:

[0092] 1. Acquisition of raw images of nematodes: Movement videos of 8 different strains of nematodes were randomly selected from the nematode behavioral phenotypic database. Different strains of nematodes have different movement patterns and varying degrees of differences in movement posture. For each nematode, 50 videos were randomly selected, and each frame of the nematode was extracted, named in chronological order, and saved in *jpg format.

[0093] 2. Preprocessing the original nematode image: First, median filtering is used to remove image noise, with the sliding window size set to 3x3. Then, image binarization is performed. A morphological closure operator (dilation followed by erosion) is used to remove small spots on the nematode's body. Finally, a component-labeling sequential algorithm is used to remove unwanted isolated objects. The final image retains only the largest connected region, which is the nematode itself.

[0094] The nematode is obtained by using a sequential algorithm to remove unwanted isolated objects and identifying the region with the largest connected component in the image.

[0095] Scan each pixel of the image in the x and y directions.

[0096] Pixels with the same pixel value are grouped together and connected components are marked (as shown in the diagram above).

[0097] Subsequently, the connectivity components of all pixels in the image are obtained.

[0098] To ensure that there is only one object in the binary image, the largest component, namely the nematode, needs to be selected from all components.

[0099] Then, the nematode's outline is extracted, and linear interpolation is used to resample the outline points uniformly at fixed intervals. The coordinates of the two sides of the nematode outline are obtained in sequence and stored sequentially in an array index. Finally, the center coordinates of the two sides of the outline are obtained from the center point on the line connecting the center points of the two sides of the outline. Cubic spline interpolation is then applied to the center coordinates of the two sides of the outline to obtain the nematode's skeleton curve, which represents the nematode's shape. This simplifies the representation of the nematode's shape.

[0100] 3. Shape description:

[0101] Then, 38 coordinate points are placed at equal intervals on the skeleton, with the head coordinates denoted as p. The next two coordinate points are then denoted as a and q. The angle θ formed by points a and p, q is calculated, and the supplementary angle β of θ is calculated and stored in the nematode's shape vector. During each calculation, a and q are moved one coordinate point to the right for updating, and the angle calculation is iteratively performed.

[0102] In other words, after each calculation, points a and q move backward by one coordinate point. In the second calculation, a is at the third coordinate point, and q is at the fourth coordinate point. p represents the head coordinates, which remain stationary during subsequent angle calculations. Each time, points a and q move backward by one coordinate point.

[0103] θ=arccos((l p 2 +l q 2 -l a 2 ) / 2*l p *l q )

[0104] β=π-θ

[0105] Among them, l p ,l q ,l a These are the corresponding side lengths of the triangle formed by points p, q, and a, where arccos() is the inverse function of cos(). The calculated angle β (β is the supplementary angle to θ) is then used. The remaining angles β are calculated sequentially using this method. To prevent the nematode shape from being affected by factors such as image rotation, the angles need to be standardized (each angle is subtracted from the average of all angles). Finally, the shape of each nematode can be represented by a unique shape vector containing 36 angles β.

[0106] In other words, the final shape vector representation of the nematode contains 36 standardized angles β. It's important to understand that 36 here doesn't necessarily mean the nematode's shape vector must have that many angles; other numbers are also acceptable, but more isn't always better. It's a suitable value chosen based on the nematode's body length. The advantage of using these angles to describe the nematode's body posture is that these angles are only related to the nematode's body posture and do not change due to image rotation.

[0107] The shape vector of the nematode is calculated to describe its shape and body pose in the current video frame. This vector uniquely represents the nematode's shape and is unaffected by factors such as image rotation. The shape vector is 1x36 in size. A shape vector for each nematode is obtained every 6 frames according to the video sequence. These vectors are then placed sequentially into an array according to their chronological order. Thus, each sequence contains 40 shape vectors representing nematode poses.

[0108] Follow the steps above to construct the sequence dataset required for the experiment.

[0109] 4. Data partitioning and processing:

[0110] Eighty percent of the dataset was used as training data and saved as a CSV file. To improve training speed, the data was first standardized using a Min-Max Scaler before being fed into the model to ensure that the data were within the range [0,1], thus facilitating faster model convergence. The Adam optimization algorithm was used with default parameters. Finally, the best prediction model was constructed. MSE was used as the evaluation metric for the model's prediction results. The remaining 20% ​​was then used for testing. After obtaining the model's prediction results, the true and predicted shape vectors were denormalized, and then the restored data was smoothed. This yielded the final vectors used to describe the shape of the nematodes. These vectors can be used to compare the differences in body posture between different strains of nematodes at the next time step.

[0111] 5. Prediction model of posture and shape

[0112] This patent provides a TS-former model for predicting the behavior and posture of nematodes. The model structure is as follows: Figure 2 As shown.

[0113] TS-former's main architecture and concepts:

[0114] (1) Time encoding module: The processed model input sequence was obtained because the movement posture of the nematode has a certain periodicity and repetition. In addition to time encoding, the time2vec algorithm was used to process the input sequence, embedding the time into the sequence data to obtain the intermediate sequence.

[0115] Time2Vec is a time representation with three identified properties. For a given scalar concept of time τ, we represent the time2vec of τ as t2v(τ). The Time2Vec of τ (represented as t2v(τ)) is a vector of size k+1, which can be defined as:

[0116]

[0117] Where t2v(τ)[i] is the i-th element of t2v(τ), and k represents the dimension of time2vec, that is, the size of the vector is k+1; These are the learning parameters, and F is the activation function (the activation function used is the cosine function). Adding time2vec has the advantage of better capturing the periodic and non-periodic features of nematode movement posture sequences, thus improving the accuracy of model predictions.

[0118] Position encoding is performed on the intermediate sequence to obtain a nematode shape vector sequence containing time and position.

[0119] Compared to the original sequence, time2vec can reveal whether the changes in each sequence are periodic.

[0120] Compared to the original sequence, the positional information of each sequence is obtained through positional encoding.

[0121] (2) In the encoder stage, X = (x1, x2, x3…xn) is considered as the input sequence data, where x1, x2…xn represent the body shape vector of the nematode in each frame of the sequence. Then, these sequences are mapped into a continuous representation z = (z1, z2, z3,…zn), which is used as the input to the decoder. The unique representation vector of the nematode shape in each frame can be regarded as each word in the sentence. The future body shape of the nematode is predicted by the previous shape, and this body shape has a long-term dependency relationship. At the same time, due to the periodicity of the nematode's movement, we embed time vector features into the shape vector of each frame in the original sequence to capture the periodicity and repetition of the nematode. For the specific time2vec, please refer to formula (1). After encoding by time vector and position, it is fed into the multi-head sparse self-attention layer in the model.

[0122] We feed the sequence data into the encoder, which extracts the key features and positional encoding information of the entire sequence.

[0123] In other words, extracting all the useful key information and features can better describe the characteristics of the entire sequence and its dependencies.

[0124] Traditional attention modules take (Query, Key, Value) as input and then perform a scaled dot product. Multi-head attention modules use concat to concatenate the single-head attention values.

[0125] Multihead(Q,K,V)=concat(head1,head2…head n )

[0126] head j =attention(q j ,k j ,v j )

[0127] in, They mapped the simple q, k, v values ​​to different representation spaces using nematode commutation, and then computed attention. After h iterations, they concatenated the attention values. Finally, they obtained the output of an encoder.

[0128] (3) In addition, the encoder contains two transformer layers. Each transformer layer has a multi-head self-sparse attention layer and a fully connected network, which is a two-layer feedforward network with ReLU activation. Given a sequence h1, h2, ... hn, the computation of the FFN sublayer at any position hi is defined as:

[0129] FFN(h i ) = Relu(h i W 1 +b 1 W 2 +b 2

[0130] Besides the two sub-layers mentioned above, residual connections and layer normalization are also key components of the transformer. For any vector v, the layer normalization is calculated as follows:

[0131]

[0132] Where v and σ are the mean and standard deviation of the elements in v, and γ and the deviation vector β are parameters.

[0133] The transformer hierarchy of the ts-former model used in this embodiment is as follows: Figure 3As shown, specifically, the residual is first calculated to obtain a vector 'a', then vector 'a' is used for layer normalization and passed to the fully connected (FC), and the output of the FC is added to 'a' to obtain the residual. This structure can reduce the number of hyperparameters and shorten the model training time.

[0134] (4) Self-attention has the potential for sparsity in its probability distribution. This allows for a selective technique for all p(kj|qi), effectively ignoring those dot products that contribute little to attention. The sparsity of each query can be calculated using the following formula.

[0135]

[0136] The first q in the formula above i It is the log-sum-exp for all keys. The second term is their arithmetic mean.

[0137] Then, based on the evaluation function above, we can obtain the formula for self-probabilistic attention:

[0138]

[0139] Q and q have sparse matrices of the same size, but the difference is that Q only contains the sparse estimate. This makes the complexity of the self-probabilistic attention mechanism only O(lnl) when performing the dot product operation on each query-key. q ).

[0140] In the sequence input stage, we added a temporal encoding module to capture the periodic and non-periodic behaviors of nematode body posture. During feature mapping in the encoding module, there is redundancy in the V-values. To address this, a distillation operation is used to assign higher weights to the main features of the sequence and generate feature maps in the next layer. The main process of the distillation operation from layer i to i+1 is as follows:

[0141]

[0142] It includes the key operations of the model and attention module. ELU is used as the activation function.

[0143] The feature sequence obtained by the decoder is then fed into the encoder. A masking operation is applied in the encoder module to the self-attention module to prevent the module from overlooking the position of the next sequence at each location. This avoids autoregression.

[0144] The data is then fed into the model for training to obtain predictions of the nematode's body pose. The loss function used is MSE.

[0145] like Figure 2 The process of the TS-former model is illustrated below:

[0146] Each sequence contains 40 shape vectors representing the shape of a nematode, and the sequence length is 1x1440. After preprocessing, the time2vec() model is used to embed temporal information into the shape vectors of each frame of the nematode to capture periodic changes. This yields the intermediate sequence data.

[0147] Before being fed into the encoder module, the shape vector of each frame is positionally encoded to preserve the order information of the shape vectors in each frame of the sequence. At this point, the sequence has been processed, resulting in a nematode shape vector sequence containing time and position information.

[0148] In the encoder, the input sequence (x1,…,xn) is mapped to a continuous representation sequence z = (z1,…,zn).

[0149] For the encoded z, the decoder generates one symbol at a time until a complete output sequence (y1,…,ym) is generated. For each decoding step, the model is autoregressive (i.e., the previously generated symbol is used as an additional input when generating the next symbol).

[0150] The final predicted nematode shape vector is obtained.

[0151] Composition of the TS-former model:

[0152] The time2vec() function will not be described here; only the encoder and decoder will be described:

[0153] Encoder components:

[0154] The decoder module consists of two identical multi-head self-sparse attention mechanisms stacked together. Each attention layer is followed by a distillation layer, which concatenates the feature maps of the distilled sequences as input to the decoder. This reduces computational cost.

[0155] Decoder composition: The decoder in this model consists of four stacked decoder layers.

[0156] Each decoding layer consists of a three-layer interconnected structure:

[0157] The first and second sub-layer connection structures consist of a multi-head self-attention sub-layer, a normalization layer, and a residual connection.

[0158] The third sublayer connection structure includes a feedforward fully connected sublayer, a normalization layer, and a residual connection.

[0159] Example 2

[0160] This embodiment provides a deep learning-based system for predicting the short-term posture behavior of nematodes, including:

[0161] The image acquisition module is configured to acquire raw images of nematodes and perform preprocessing to obtain preprocessed images of nematodes.

[0162] The contour extraction module is configured to extract nematode contours based on the preprocessed nematode image to obtain a nematode contour image.

[0163] The image processing module is configured to calculate the nematode shape vector based on the nematode outline image, and obtain a standardized nematode shape vector sequence;

[0164] The nematode pose prediction module is configured to predict nematode pose based on a standardized nematode shape vector sequence and a pre-trained nematode pose shape prediction model to obtain short-term pose behavior prediction results for nematodes.

[0165] The nematode posture and shape prediction model includes a time feature encoding module, an encoder, and a decoder.

[0166] The time feature encoding module performs time encoding on the standardized nematode shape vector sequence, embedding the time features into the standardized nematode shape vector sequence to obtain intermediate sequence features; the intermediate sequence features are further encoded at position before being input into the encoder to obtain a nematode shape vector sequence containing time and position.

[0167] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0168] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0169] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0170] Example 3

[0171] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the deep learning-based short-term posture behavior prediction method for nematodes as described in Embodiment 1 above.

[0172] Example 4

[0173] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the deep learning-based short-term posture behavior prediction method for nematodes as described in Embodiment 1 above.

[0174] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0175] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0179] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A deep learning-based method for predicting short-term posture behavior in nematodes, characterized in that, include: The original image of the nematode is acquired and preprocessed to obtain the preprocessed image of the nematode. Based on the preprocessed nematode image, the nematode contour is extracted to obtain the nematode contour image. The nematode shape vector is calculated from the nematode outline image to obtain a standardized nematode shape vector sequence; Based on the standardized nematode shape vector sequence, the nematode posture is predicted using a pre-trained nematode posture shape prediction model, and the short-term posture behavior prediction results of the nematode are obtained. The nematode posture and shape prediction model includes a time feature encoding module, an encoder, and a decoder. The time feature encoding module performs time encoding on the standardized nematode shape vector sequence, embedding the time features into the standardized nematode shape vector sequence to obtain intermediate sequence features; the intermediate sequence features are further encoded at position before being input into the encoder to obtain a nematode shape vector sequence containing time and position; the time feature encoding module uses the time2vec algorithm to perform time encoding on the nematode shape vector and time vector features, embedding the time features into the nematode shape vector to obtain intermediate sequence features; The encoder consists of two identical multi-head self-sparse attention layers stacked together; each multi-head self-sparse attention layer is followed by a distillation layer and a fully connected layer, which concatenates the feature maps of the distilled sequence as the input to the decoder. The decoder is composed of four stacked decoder layers; each decoder layer consists of three sub-layer interconnection structures: The connection structure between the first and second sub-layers includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection layer; The third sublayer consists of a feedforward fully connected sublayer, a normalization layer, and a residual connected layer.

2. The method for predicting short-term posture behavior of nematodes based on deep learning as described in claim 1, characterized in that, The process of acquiring the original image of the nematode and preprocessing it to obtain the preprocessed image of the nematode includes: Based on the original nematode image, median filtering was used to remove image noise. Image binarization is performed on the nematode image after noise removal; For the binarized nematode image, the morphological closure operator is used to remove spots and disconnected regions outside the nematode body to obtain the nematode image after removing spots. For the nematode image after spot removal, the component labeling order algorithm is used to retain the largest connected region in the nematode image, resulting in a preprocessed nematode image.

3. The method for predicting short-term posture behavior of nematodes based on deep learning as described in claim 1, characterized in that, The step of extracting nematode contours from the preprocessed nematode image to obtain a nematode contour image includes: Based on the preprocessed nematode image, the contour points are resampled uniformly at fixed intervals using a linear interpolation method. Obtain the coordinates of the points on both sides of the nematode outline arranged in sequence, and store them sequentially in the array index; The center coordinates of the two sides of the nematode outline are extracted from the coordinate points on both sides of the outline, and the skeleton curve of the nematode is obtained by using the cubic spline interpolation algorithm. Based on the skeletal curves of the nematode, the outline image of the nematode is obtained.

4. The method for predicting short-term posture behavior of nematodes based on deep learning as described in claim 1, characterized in that, The step of calculating the nematode shape vector from the nematode outline image to obtain a standardized nematode shape vector sequence includes: Based on the nematode outline image, multiple coordinate points are set at equal intervals on the nematode skeleton curve, and the head coordinates are marked. p ; The next two coordinate points are denoted as follows: a , q ,calculate a and p , q The angle formed by the two points θ 1. Confirm θ supplementary angle of 1 β 1; Head coordinates p Stay still. a , q Move the position of one coordinate point and then calculate. a and p , q The angle formed by the two points θ 2. Confirm θ supplementary angle of 2 β 2; And so on, iterative calculation a and p , q Find the angle formed by two points and calculate the supplementary angle of that angle; The calculation of the coordinates of the nematode skeleton curve is completed, resulting in a curve containing multiple supplementary angles. β Angle sequence; For those containing multiple supplementary angles β All included angles in the angle sequence are normalized to obtain a standardized nematode shape vector sequence.

5. A deep learning-based short-term posture behavior prediction system for nematodes, characterized in that, include: The image acquisition module is configured to acquire raw images of nematodes and perform preprocessing to obtain preprocessed images of nematodes. The contour extraction module is configured to extract nematode contours based on the preprocessed nematode image to obtain a nematode contour image. The image processing module is configured to calculate the nematode shape vector based on the nematode outline image, and obtain a standardized nematode shape vector sequence; The nematode pose prediction module is configured to predict nematode pose based on a standardized nematode shape vector sequence and a pre-trained nematode pose shape prediction model to obtain short-term pose behavior prediction results for nematodes. The nematode posture and shape prediction model includes a time feature encoding module, an encoder, and a decoder. The time feature encoding module performs time encoding on the standardized nematode shape vector sequence, embedding the time features into the standardized nematode shape vector sequence to obtain intermediate sequence features; the intermediate sequence features are further encoded at position before being input into the encoder to obtain a nematode shape vector sequence containing time and position; the time feature encoding module uses the time2vec algorithm to perform time encoding on the nematode shape vector and time vector features, embedding the time features into the nematode shape vector to obtain intermediate sequence features; The encoder consists of two identical multi-head self-sparse attention layers stacked together; each multi-head self-sparse attention layer is followed by a distillation layer and a fully connected layer, which concatenates the feature maps of the distilled sequence as the input to the decoder. The decoder is composed of four stacked decoder layers; each decoder layer consists of three sub-layer interconnection structures: The connection structure between the first and second sub-layers includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection layer; The third sublayer consists of a feedforward fully connected sublayer, a normalization layer, and a residual connected layer.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the deep learning-based method for predicting short-term posture behavior of nematodes as described in any one of claims 1-4.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the deep learning-based short-term posture behavior prediction method for nematodes as described in any one of claims 1-4.

Images