Cell detection method, electronic device, and storage medium

Abstract

The application provides a cell detection method, an electronic device and a storage medium. The method comprises the following steps: acquiring a plurality of cell images; determining a bounding box corresponding to each cell in each cell image; determining position information of the bounding box corresponding to each cell in the plurality of cell images; determining a motion direction of each cell according to the position information; determining brightness information of each cell based on the plurality of cell images; and determining a motion detection result of each cell based on the brightness information and the motion direction. The application can improve the effect of cell detection.

Description

Technical Field

[0001] This invention relates to the field of biology, and more particularly to a cell detection method, electronic device, and storage medium. Background Technology

[0002] In biological research, it is necessary to track multiple cells in microscopic videos. Accurately locating cells and observing their orientation can provide important information for cell research.

[0003] In related technologies, most deep learning-based approaches first locate cells in the microscope video using object detection algorithms, and then track the cells and observe their orientation using multi-object tracking algorithms. However, these methods cannot detect whether the cell orientation is correct, resulting in poor cell detection results. Summary of the Invention

[0004] In view of the above, it is necessary to propose a cell detection method, electronic device and storage medium that can solve the problem of poor cell detection results.

[0005] In a first aspect, embodiments of this application provide a cell detection method, the cell detection method comprising: acquiring multiple frames of cell images; determining a bounding box corresponding to each cell in each frame of cell images; determining the position information of the bounding box corresponding to each cell in the multiple frames of cell images; determining the motion direction of each cell based on the position information; determining the brightness information of each cell based on the multiple frames of cell images; and determining the motion detection result of each cell based on the brightness information and the motion direction.

[0006] In some embodiments, determining the bounding box corresponding to each cell includes: determining the channel position corresponding to each cell image frame; and determining the segmentation region corresponding to each cell image frame based on the channel position, such that a preset segmentation model determines the bounding box corresponding to each cell in each cell image frame based on the segmentation region.

[0007] In some embodiments, the method further includes: determining a cell sample and the cell size corresponding to the cell sample; and determining the bounding box information based on the cell size, wherein the bounding box information includes one or more of width information, height information, and aspect ratio information.

[0008] In some embodiments, determining the bounding box corresponding to each cell in each frame of cell image includes: calling a preset segmentation model to process each frame of cell image to obtain a cell mask corresponding to each cell; determining the bounding rectangle corresponding to the cell mask, and using the bounding rectangle as the bounding box corresponding to each cell.

[0009] In some embodiments, determining the position information of the bounding box corresponding to each cell in the multi-frame cell images includes: selecting a first cell image according to a preset time order; determining a first bounding box corresponding to each cell in the first cell image, a tracking marker corresponding to each first bounding box, and a first feature of each cell; predicting the feature corresponding to each cell in a second cell image based on the first feature to obtain a predicted feature, wherein the second cell image is adjacent to and follows the first cell image; determining a second feature corresponding to each cell in the second cell image; determining the tracking marker corresponding to each cell in the second cell image based on the predicted feature and the second feature; and determining the position information of the bounding box corresponding to each cell in the multi-frame cell images based on the first feature and the second feature having the same tracking marker.

[0010] In some embodiments, determining the movement direction of each cell based on the location information includes: determining the movement trajectory of each cell based on the location information; determining the channel information corresponding to the movement trajectory; and using the channel information as the movement direction of each cell.

[0011] In some embodiments, determining the brightness information of each cell based on the multiple cell images includes: determining the brightness value of the brightness region corresponding to each cell image; obtaining a brightness value sequence according to a preset time order; determining the frame number information corresponding to the cell image in which the preset cell appears; and selecting a target brightness value from the brightness value sequence as the brightness information corresponding to the preset cell based on the frame number information.

[0012] In some embodiments, determining the motion detection result of each cell based on the brightness information and the motion direction includes: determining the target motion direction corresponding to each cell based on the brightness information and a preset brightness threshold; if the motion direction is the same as the target motion direction, then determining the motion detection result of each cell as correct; if the motion direction is different from the target motion direction, then determining the motion detection result of each cell as incorrect.

[0013] Secondly, embodiments of this application provide a cell detection device, the cell detection device comprising: an image acquisition module for acquiring multiple frames of cell images; a bounding box determination module for determining the bounding box corresponding to each cell in each frame of cell images; a position determination module for determining the position information of the bounding box corresponding to each cell in the multiple frames of cell images; a direction determination module for determining the motion direction of each cell based on the position information; a brightness determination module for determining the brightness information of each cell based on the multiple frames of cell images; and a motion detection module for determining the motion detection result of each cell based on the brightness information and the motion direction.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the cell detection method described in any one of the above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a controller, implements the cell detection method described in any one of the above claims.

[0016] The cell detection method provided in this application includes: acquiring multiple frames of cell images; determining the bounding box corresponding to each cell in each frame of cell images; determining the position information of the bounding box corresponding to each cell in the multiple frames of cell images; determining the motion direction of each cell based on the position information; determining the brightness information of each cell based on the multiple frames of cell images; and determining the motion detection result of each cell based on the brightness information and the motion direction. This method, by accurately detecting cell brightness and combining it with motion direction recognition, can determine whether a cell has entered the correct channel, providing crucial information for cell behavior analysis and improving the effectiveness of cell detection. Attached Figure Description To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application scenario diagram of the cell detection method provided in the embodiments of this application.

[0018] Figure 2 This is a schematic flowchart of the cell detection method provided in the embodiments of this application.

[0019] Figure 3This is a schematic diagram of a cell image provided in an embodiment of this application.

[0020] Figure 4 This is a schematic diagram of the region of interest provided in the embodiments of this application.

[0021] Figure 5 This is a schematic diagram of the segmented region provided in the embodiments of this application.

[0022] Figure 6 This is a schematic diagram of the process for determining boundary information provided in the embodiments of this application.

[0023] Figure 7 This is a flowchart illustrating the bounding box determination method provided in an embodiment of this application.

[0024] Figure 8 This is a flowchart illustrating the location information determination method provided in the embodiments of this application.

[0025] Figure 9 This is a flowchart illustrating the brightness information determination method provided in the embodiments of this application.

[0026] Figure 10 This is a flowchart illustrating the motion detection result determination method provided in the embodiments of this application.

[0027] Figure 11 This is a schematic diagram of the cell detection device provided in the embodiments of this application.

[0028] Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, "at least one" means one or more. "More than one" means two or more. For example, at least one of a, b, or c can represent seven cases: a, b, c, a and b, a and c, b and c, and a, b, and c.

[0032] The following is an explanation of the relevant terms used in this application: YOLOv5 (You Only Look Once version 5) detection algorithm: It is a single-stage object detection algorithm that uses a convolutional neural network architecture. It is efficient and accurate, and is especially suitable for real-time object detection tasks.

[0033] Kalman filter tracking algorithm: It is a recursive data processing algorithm based on linear algebra and probability theory. It estimates the system state in the optimal way by continuously iterating through two steps: prediction and update.

[0034] In biological research, it is necessary to track multiple cells in microscopic videos. Accurately locating cells and observing their orientation, counting cells, and detecting whether the direction of cell movement is correct can provide important information for cell research.

[0035] In related technologies, most deep learning-based approaches first locate cells in a microscope video using object detection algorithms, then track the cells and observe their movement using multi-object tracking algorithms. Since each cell is assigned a unique index during tracking, cell counting can be performed based on these indices. For example, the object detection algorithm is the YOLOv5 algorithm, and the object tracking algorithm is the Kalman filter tracking algorithm. The YOLOv5 detection model is trained using a large number of cell images with cell location annotations, enabling the trained YOLOv5 detection model to correctly locate cells in the microscope video and generate bounding boxes at the corresponding cell locations, thus achieving object detection. The Kalman filter tracking algorithm includes the following steps: First, in the first frame of the microscopic video, the initial state of the cell (e.g., cell position, movement speed, etc.) is estimated to obtain an initial state covariance matrix. Then, using a pre-defined system dynamic model, based on the estimated initial state and the system's control input, the cell state at the next moment (i.e., the second frame of the cell image) is predicted, and the state covariance matrix is ​​updated to obtain the predicted state covariance matrix. Next, the actual observed state of the cell in the second frame of the cell image is obtained from the microscopic video, and the state covariance matrix is ​​updated to obtain the measured state covariance matrix. Finally, using the measured state covariance matrix and the predicted state covariance matrix, the Kalman gain is determined, and the optimal estimate of the cell state is obtained using the Kalman gain. As the microscopic video progresses, the prediction and update steps are continuously repeated to achieve real-time cell tracking.

[0036] However, the above methods require a large number of cell images with cell location information as training data, resulting in high training costs. Furthermore, the Kalman filter tracking algorithm is suitable for simple linear moving targets, but it is prone to losing targets or causing target tracking confusion in complex scenes, resulting in poor cell tracking performance. In addition, the above methods cannot detect whether the cell's motion direction is correct, resulting in poor cell detection performance.

[0037] In view of the above problems, embodiments of this application provide a cell detection method, electronic device and storage medium, which can solve the problems of high training cost and poor tracking performance in complex scenarios during cell tracking, and can detect whether the movement direction of the cell is correct.

[0038] Figure 1 This is an application scenario diagram of the cell detection method provided in the embodiments of this application. For example... Figure 1As shown, the application scenario includes an electronic device 10 and an image acquisition device 20. The electronic device 10 and the image acquisition device 20 are connected, and the connection method can include wired communication and wireless communication. For example, wired communication between the electronic device 10 and the image acquisition device 20 can be achieved through a preset data cable, or wireless communication can be achieved through Bluetooth, Wi-Fi, ZigBee, cellular networks, etc. The electronic device 10 can be an electronic product with data processing capabilities, such as a personal computer, tablet computer, or smartphone. The image acquisition device 20 is used to acquire video of cell movement. The image acquisition device 20 can include a microscope, camera, etc. In this embodiment, a microscope is used as an example for illustration. In some embodiments, the image acquisition device 20 includes a light source component (not shown in the figure), which provides a light source and can include a laser, halogen lamp, fluorescent lamp, and xenon lamp, etc.

[0039] In some embodiments, a biochip is pre-configured, and the biochip has multiple cell channels. A cell channel is a specific region or path designated for a cell when observing and recording cell movement under a microscope. In some embodiments, the configuration of the cell channels needs to be determined based on the specific experimental purpose and cell type. Different cell types and experimental purposes may require different cell channel configurations.

[0040] In some embodiments, a cell sample containing multiple cells is prepared in advance. The cell sample is dropped into the reaction area of ​​the biochip, and the biochip with the cell sample is placed on the stage of the image acquisition device 20. A designated area of ​​the biochip is illuminated using a light source assembly, emitting a light source of a specific wavelength. This causes the cells passing through the designated area to emit light upon excitation by the light source. The image acquisition device 20 then captures a video of the cell motion within the cell sample (hereinafter referred to as "cell video" for ease of description). The video includes the complete process of the cells emitting light upon excitation by the light source and the cells entering the cell channel. The designated area can be set according to actual needs and is not limited herein.

[0041] In some embodiments, the image acquisition device 20 sends the acquired cell video to the electronic device 10, which performs relevant processing on the cell video to obtain cell counting results and motion detection results within the cell video. Based on the motion detection results, it can be determined whether the motion is correct or an error has occurred.

[0042] In the application scenarios provided in the embodiments of this application, a light source component emits a light source of a specific wavelength to the cells, causing the cells to emit light after being excited by the light source. The cell video is acquired by an image acquisition device, and the cells in the cell video are processed by an electronic device to realize the counting of cells in the cell video and the determination of the corresponding motion detection results of the cells.

[0043] Figure 2 This is a schematic flowchart of the cell detection method provided in the embodiments of this application. The cell detection method is applied to electronic devices (e.g., Figure 1 Electronic devices 10). Figure 2 As shown, a cell detection method may include the following steps. Depending on different needs, the order of the steps in this flowchart may be changed, and some may be omitted.

[0044] S11, acquire multiple frames of cell images.

[0045] In at least one embodiment of this application, multiple cell images can be acquired based on cell video. Cell video can include the complete process of multiple cells in a cell sample emitting light when excited by a light source and the cells entering cell channels. By extracting video frames from the cell video, multiple cell images can be obtained, and these multiple cell images are ordered according to a preset time order, which can be from front to back.

[0046] In some embodiments, the cell image may include pre-defined channel information and information about multiple cells within the cell sample. For example, Figure 3 This is a schematic diagram of cell images provided in the embodiments of this application, such as... Figure 3 As shown, the cell image includes cell channels and multiple cells. The cell channels are generally Y-shaped, located at the bottom of the cell image. Initially a single channel (referred to as the sample channel for ease of description), it splits into two channels—a left channel and a right channel—after reaching a designated area. After passing through the sample channel and reaching the designated area, some cells in the cell sample enter the right channel, and some enter the left channel. Since the designated area is relatively narrow, only one cell passes through at a time. The number of channels can be set according to actual needs; for example, it can be two, three, or four channels, etc., and is not limited here. This embodiment uses two channels as an example for illustration.

[0047] In some embodiments, the light source component illuminates the cells in a designated area of ​​the cell channel, emitting a light source of a specific wavelength, causing the cells passing through the designated area to emit light upon excitation. The designated area can be the boundary between the sample channel and the left and right channels. When the cells reach the designated area after passing through the sample channel, their brightness value reaches its maximum. The cells are then divided according to their brightness value, with some entering the left channel and some entering the right channel. Cells with a pre-set brightness value greater than or equal to a preset brightness threshold enter the right channel, while cells with a brightness value less than the preset brightness threshold enter the left channel. An image acquisition device captures the entire process of the cells emitting light upon excitation and entering the cell channel, obtaining a cell video. Based on this cell video, the brightness information and movement direction of the cells are detected, enabling the detection of whether the cell movement direction is correct.

[0048] S12, determine the bounding box corresponding to each cell in each frame of cell image.

[0049] In at least one embodiment of this application, a preset segmentation model is used to segment multiple cells within each frame of a cell image, determining the bounding box corresponding to each cell in each frame of the cell image. The input data of the segmentation model is the cell image, and the output data is the bounding box corresponding to each cell in the cell image. In some embodiments, the preset segmentation model processes the cell image to generate corresponding mask information for each cell, and draws the bounding rectangle of each cell as the bounding box using the mask information. The segmentation model can be a Segment Anything Model (SAM), a model for image segmentation that can identify all objects in an image or video, even those objects that the model has never trained on. Thus, this segmentation model can be applied to new image domains (e.g., biological cell domains) without additional training. By using this segmentation model, embodiments of this application enable accurate detection of cells in cell videos even with limited training data, thereby saving significant training data acquisition costs.

[0050] S13, determine the position information of the bounding box corresponding to each cell in the multi-frame cell image.

[0051] In at least one embodiment of this application, a target tracking algorithm is used to track the bounding box of each detected cell, determining the position information of the bounding box of each cell in each frame of cell images, thereby obtaining the position information of the bounding box of each cell in multiple frames of cell images. The input data of the target tracking algorithm is multiple frames of cell images with labeled cell bounding boxes, and the output data is the position information of the bounding box of each cell in each frame of cell images. The target tracking algorithm can be a StrongSORT target tracking algorithm, which integrates deep learning technology and real-time tracking methods to efficiently track the motion trajectory of cells in a video sequence.

[0052] In some embodiments, considering that cells move relatively quickly, cells at the top of the cell image will rapidly move out of the shooting area of ​​the image acquisition device. Based on this, when using the StrongSORT target tracking algorithm to track the bounding box corresponding to each cell in multiple frames of cell images, tracking can start from the bottom of the cell image, which can avoid the problem of chaotic tracking effect caused by cells at the top of the cell image rapidly moving out of the shooting area.

[0053] S14, Based on the position information, determine the direction of movement of each cell.

[0054] In at least one embodiment of this application, the direction of motion can be determined based on channel information. Continuing with the above embodiments, the biochip includes a left channel and a right channel; thus, the direction of motion can include the left channel direction and the right channel direction. In some embodiments, the direction of motion of each cell can be determined by determining the position information of each cell in multiple frames of cell images.

[0055] S15, Based on the multi-frame cell images, determine the brightness information of each cell.

[0056] In at least one embodiment of this application, a light source component illuminates cells in a designated area of ​​the cell channel (e.g., the boundary between the sample channel and the left and right channels), emitting a light source of a specific wavelength, causing cells passing through the designated area to emit light upon excitation. Based on this, the position of the cell at its highest brightness before entering the channel is fixed. In the cell video, a flickering bright spot can be observed in the designated area just before the cell enters the left or right channel. When a bright spot of a cell is detected in the current frame, the cell will officially enter the left or right channel in the following frames. Thus, the cell's movement direction is closely related to the bright spot; by determining the brightness information of each cell, it is possible to detect whether the cell's movement direction is correct.

[0057] In some embodiments, when determining the brightness information of each cell based on the multiple cell images, the brightness region corresponding to each cell image (i.e., the aforementioned designated region) can be determined, and the brightness value of the bright spot region in each cell image can be detected to obtain a brightness value sequence. Based on the frame number information corresponding to the cell image where a new cell (also referred to as a "preset cell" in this application) appears, a target brightness value can be selected from the brightness value sequence as the brightness information of the new cell.

[0058] S16, Based on the brightness information and the direction of motion, determine the motion detection result of each cell.

[0059] In at least one embodiment of this application, cells with a brightness value greater than or equal to a preset brightness threshold are pre-defined to enter the right channel, and cells with a brightness value less than the preset brightness threshold are pre-defined to enter the left channel. By detecting the brightness information and the movement direction of the cells, it is possible to detect whether the movement direction of the cells is correct, and obtain the movement detection result for each cell. The movement detection result includes a result indicating correct movement and a result indicating incorrect movement.

[0060] For example, when the cell's brightness value is greater than or equal to a preset brightness threshold, if the cell's movement direction is in the right channel direction, then the cell's movement direction is determined to be correct; if the cell's movement direction is in the left channel direction, then the cell's movement direction is determined to be incorrect. When the cell's brightness value is less than the preset brightness threshold, if the cell's movement direction is in the right channel direction, then the cell's movement direction is determined to be incorrect; if the cell's movement direction is in the left channel direction, then the cell's movement direction is determined to be correct.

[0061] In the cell detection method provided in this application embodiment, a pre-set segmentation model is used to determine the bounding box corresponding to each cell, and a target tracking algorithm is used to determine the position information of the bounding box in multiple frames of cell images. Based on the position information, the movement direction of the cell is determined. The SAM model and StrongSORT target tracking algorithm are combined to locate the cell and observe its direction. Video cell tracking can be achieved without additional training costs and can improve tracking performance in complex scenes. In addition, the above method can determine whether the cell has entered the correct channel by accurately detecting the cell brightness and combining it with the movement direction recognition, providing key information for the analysis of cell behavior and improving the effect of cell detection.

[0062] In at least one embodiment of this application, since cell images contain noise and other objects outside the channels in addition to channel information and cell information, to avoid the problem of reduced accuracy in cell tracking due to the identification of other objects when performing cell detection based on the segmentation model, after obtaining multiple frames of cell images based on the cell video, it is also necessary to remove the noise region outside the channels in each frame of cell images. In some embodiments, determining the bounding box corresponding to each cell includes: determining the channel position corresponding to each frame of cell images; determining the segmentation region corresponding to each frame of cell images based on the channel position, so that a preset segmentation model determines the bounding box corresponding to each cell in each frame of cell images based on the segmentation region. The segmentation region can be the region where the cell channel is located.

[0063] For example, Figure 4 This is a schematic diagram of the region of interest provided in an embodiment of this application. Figure 5 This is a schematic diagram of the segmented region provided in an embodiment of this application. For example... Figure 4 As shown, after magnifying the original cell image, a preset region of interest is selected from the magnified cell image. The region of interest contains not only channel information and cell information, but also noise outside the channels. Figure 5 As shown, the region of interest is truncated, and the region containing only channel information and cell information is used as the segmentation region, and the noise region within the region of interest is removed.

[0064] In at least one embodiment of this application, before determining the bounding box corresponding to each cell in each frame of cell image based on a preset segmentation model, the bounding box information can be determined, and the bounding box corresponding to the cell can be determined based on the bounding box information. In some embodiments, combined with Figure 6 This application provides a schematic diagram illustrating the process of determining boundary information in its embodiments. Figure 6 As shown, cell samples and their corresponding cell sizes are determined; based on the cell sizes, the bounding box information is determined, including one or more of width information, height information, and aspect ratio information, so that a preset segmentation model determines the bounding box corresponding to each cell in each frame of cell image based on the bounding box information.

[0065] In some embodiments, a cell sample contains multiple cells, each with a corresponding cell size, and cells of the same cell type have similar sizes. Therefore, determining the cell size corresponding to a cell sample may include: determining the cell type corresponding to the cell sample; and determining the cell size corresponding to the cell sample based on a pre-defined correspondence between cell types and cell sizes. In some embodiments, if the cell sample contains multiple cell types, i.e., the cell sample corresponds to multiple cell sizes, data processing can be performed on the determined multiple cell sizes to obtain the cell size corresponding to the cell sample. The data processing methods may include averaging, taking the maximum or minimum value (e.g., taking the maximum or minimum value), taking the median, etc., and are not limited thereto.

[0066] In some embodiments, the border information may include one or more of the following: border width information, height information, and aspect ratio information, wherein the aspect ratio information may be the ratio of the border width to the border height. This application embodiment uses aspect ratio information as an example for illustration; for instance, the aspect ratio information may be any value between 2 / 3 and 3 / 2.

[0067] In the cell detection method provided in this application embodiment, the bounding box information is determined based on the cell size corresponding to the cell sample, and the bounding box is determined according to the bounding box information. This can avoid the problem of two cells that are close to each other being identified as one cell, which would cause interference to subsequent cell tracking. This can improve the accuracy of bounding box determination, thereby improving the accuracy of cell detection.

[0068] In at least one embodiment of this application, a preset segmentation model is used to segment cells within each frame of cell image and generate corresponding mask information for each cell, and draw the bounding rectangle of each cell as a bounding box using the mask information. Figure 7 This is a flowchart illustrating the bounding box determination method provided in an embodiment of this application. The bounding box determination method is applied to electronic devices. Figure 7 As shown, it includes the following steps: S21, call the preset segmentation model to process each frame of cell image to obtain the cell mask corresponding to each cell.

[0069] In at least one embodiment of this application, the segmentation model can be a SAM model. The input data of the SAM model is each frame of cell images, and the output data is the bounding box corresponding to each cell in each frame of cell images. In some embodiments, the SAM model can process each frame of cell images to obtain a cell mask corresponding to each cell. The cell mask can be a binary image, wherein a portion of the cell bounding box can be represented by a value (e.g., 1), while the portion inside the cell bounding box can be represented by another value (e.g., 0). In this way, the cell bounding boxes can be accurately distinguished.

[0070] S22, determine the bounding rectangle corresponding to the cell mask, and use the bounding rectangle as the bounding box corresponding to each cell.

[0071] In at least one embodiment of this application, the circumscribed rectangle can represent the smallest rectangular region that can completely enclose the cell. The position and size of the circumscribed rectangle can be obtained by calculating the minimum and maximum coordinates of the cell mask. For example, all pixels in the cell mask are traversed to find the leftmost, rightmost, topmost, and bottommost boundary pixel positions. Based on the positions of these boundary pixels, the four vertices of the circumscribed rectangle are determined, and the circumscribed rectangle is drawn or marked according to the determined positions of the four vertices, serving as the bounding box for each cell.

[0072] In the cell detection method provided in this application embodiment, the bounding box corresponding to the cell in each frame of cell image is determined by the SAM model. This enables the model to correctly detect cells in the video even without a large amount of data for training. Furthermore, this application determines the cell mask based on the bounding box information, and then determines the bounding rectangle corresponding to the cell mask as the bounding box corresponding to each cell. This avoids two cells that are close to each other being identified as one cell, which would interfere with subsequent cell tracking and improve the accuracy of cell detection.

[0073] In at least one embodiment of this application, each detected cell is tracked by a target tracking algorithm to determine the position information of each cell in each frame of cell image, thereby obtaining the position information of the bounding box of each cell in multiple frames of cell images. Figure 8 This is a flowchart illustrating the location information determination method provided in an embodiment of this application. The location information determination method is applied to electronic devices. Figure 8 As shown, it includes the following steps: S31, Select the first cell image according to the preset time sequence.

[0074] In at least one embodiment of this application, multiple cell images can be obtained by extracting video frames from a cell video. These multiple cell images are ordered according to a preset time order, which can be from front to back. The cell image that appears first in the time sequence is selected from the multiple cell images and designated as the first cell image; for example, the first cell image is used as the first cell image.

[0075] S32, determine the first bounding box corresponding to each cell in the first cell image, the tracking marker corresponding to each first bounding box, and the first feature of each cell.

[0076] In at least one embodiment of this application, when tracking the bounding box corresponding to each cell in multiple frames of cell images using the StrongSORT target tracking algorithm, a tracking marker can be set for the bounding box corresponding to each cell. The tracking marker is used to uniquely identify the cell. The tracking marker can be a numeric marker, an alphabetic marker, or other forms of marker. This application embodiment uses a numeric tracking marker as an example for illustration. For example, a cell image frame includes three cells, namely cell A, cell B, and cell C. Tracking marker 1 is set for cell A, tracking marker 2 is set for cell B, and tracking marker 3 is set for cell C. Based on the number of tracking markers, the number of cells can be determined, thereby achieving cell counting. In addition, to avoid the problem of chaotic tracking marker settings, for example, if a cell is set with multiple tracking markers, this application embodiment sets that when the bounding box at the bottom of the cell image does not have a corresponding tracking marker, a new tracking marker is generated.

[0077] In some embodiments, feature information corresponding to each tracking marker is determined. This feature information may include the Y-axis coordinates of the bounding box, relevant features of the cells within the bounding box, cell brightness values, image acquisition time, and direction of motion. The relevant features of the cells within the bounding box can represent the cell's morphological characteristics, such as cell shape, cell size, nucleocytoplasmic ratio, and cell membrane structure. Since cells in the cell video move from bottom to top, their position in the Y-axis direction will rise in the next frame. This embodiment of the application improves the accuracy of target tracking by introducing the Y-axis coordinates of the bounding box.

[0078] In some embodiments, the StrongSORT target tracking algorithm is used to set a tracking marker for the first bounding box corresponding to each cell in the first cell image, and the feature information of each cell in the first cell image (hereinafter referred to as "first feature" for ease of description) is added to the corresponding tracking marker.

[0079] S33, based on the first feature, predict the feature corresponding to each cell in the second cell image to obtain the predicted feature, wherein the second cell image is adjacent to the first cell image and follows the first cell image. In at least one embodiment of this application, the StrongSORT target tracking algorithm can predict the possible location (i.e., the coordinates of the bounding box in the Y-axis direction) of the cell in the first cell image based on information such as the cell's position (e.g., the coordinates of the bounding box in the Y-axis direction), movement speed, and movement direction, using prediction algorithms such as Kalman filtering, to obtain predicted features. The predicted features may include the predicted coordinates of the bounding box in the Y-axis direction, relevant features of the cell within the bounding box, the cell's brightness value, the image acquisition time, and the movement direction.

[0080] S34, determine the second feature corresponding to each cell in the second cell image.

[0081] In at least one embodiment of this application, a second bounding box corresponding to each cell in the second cell image is determined based on a preset segmentation model, and the feature information of each cell in the second cell image is determined using the StrongSORT target tracking algorithm (hereinafter referred to as "second feature" for ease of description).

[0082] S35, based on the predicted features and the second features, determine the tracking marker corresponding to each cell in the second cell image.

[0083] In at least one embodiment of this application, by comparing the predicted feature with a second feature, a cell corresponding to the second feature that is identical to the predicted feature is selected, and a corresponding tracking marker is added to the bounding box of the cell, and the feature information corresponding to the tracking marker is updated. For example, cell A corresponds to tracking marker 1 and to first feature T1 in the first cell image. The features of cell A in the second cell image are predicted using the first feature T1 to obtain predicted feature T2. A cell corresponding to the second feature that is identical to predicted feature T2 is selected from the second cell image, and tracking marker 1 is set for the cell.

[0084] S36, based on the first feature and the second feature with the same tracking marker, determine the position information of the bounding box corresponding to each cell in the multi-frame cell image.

[0085] In at least one embodiment of this application, cells with the same tracking marker are the same cells. By determining the position information of cells with the same tracking marker in multiple cell images, the position information of the bounding box corresponding to each cell in the multiple cell images can be obtained.

[0086] In the cell detection method provided in this application embodiment, by integrating deep learning technology and real-time tracking methods, the motion trajectory of cells in video sequences can be tracked efficiently. This enables a more comprehensive capture of cell motion characteristics when facing challenges of complex cell dynamics and scene changes, thereby improving the accuracy of tracking.

[0087] In at least one embodiment of this application, the direction of movement can be determined based on channel information. For example, the direction of movement may include a left channel direction and a right channel direction. In some embodiments, determining the direction of movement of each cell based on the position information includes: determining the movement trajectory of each cell based on the position information; determining the channel information corresponding to the movement trajectory, and using the channel information as the direction of movement of each cell.

[0088] In some embodiments, the position information of the cell in each frame of cell image is determined to obtain a sequence of position information of the cell in multiple frames of cell images. Based on the position information sequence, the motion trajectory of the cell can be determined. Based on the motion trajectory, the channel information that the cell entered can be determined, and the channel information may include a left channel and a right channel. Based on the channel information that the cell entered, the direction of cell movement can be determined.

[0089] In at least one embodiment of this application, a light source component irradiates cells in a designated area of ​​the cell channel, emitting a light source of a specific wavelength, causing cells passing through the designated area to emit light upon excitation. Based on this, the position where the cell reaches its maximum brightness before entering the left or right channel is fixed. In the cell video, a bright spot that flickers in and out can be observed in a fixed area just before the cell enters the left or right channel. Figure 9 This is a flowchart illustrating the brightness information determination method provided in an embodiment of this application. The brightness information determination method is applied to electronic devices. Figure 9 As shown, it includes the following steps: S41, determine the brightness value of the brightness region corresponding to each frame of cell image.

[0090] In at least one embodiment of this application, the brightness region of each cell image frame is a fixed region. After a cell enters the brightness region, the cell is illuminated and generates brightness, thus determining the brightness value of the corresponding brightness region for each cell image frame. The brightness region can be preset by the operator, or it can be determined by identifying areas with significant brightness variations among multiple cell images.

[0091] S42, obtain the brightness value sequence according to the preset time order.

[0092] In at least one embodiment of this application, the preset time order can be to sort the brightness values ​​according to the time sequence from front to back, resulting in a brightness value sequence. Each brightness value in the brightness value sequence corresponds to the frame number information of the cell image.

[0093] S43, determine the frame number information corresponding to the cell image of the preset cell.

[0094] In at least one embodiment of this application, a preset cell can represent a new cell appearing in the bottommost region of a cell image, and determine the frame number information where the new cell is located.

[0095] S44, based on the frame number information, select the target brightness value from the brightness value sequence as the brightness information corresponding to the preset cell.

[0096] In at least one embodiment of this application, a brightness value corresponding to the frame number information is determined from the brightness value sequence, and a target brightness value, i.e., the brightness information corresponding to a preset cell, is selected from a preset number of brightness values ​​preceding the determined brightness value. The preset number can be determined based on information such as the cell's movement rate and the video capture frame rate; for example, the preset number can be 2, 3, etc.

[0097] For example, when a new cell is detected at the bottom of the cell image in frame b, the maximum value from the numbers a to b-2 in the brightness list is selected as the brightness value of that cell, where a is initially 1. After determining the brightness value of the cell, a is changed to b-1, and this process is repeated until the brightness information of each cell is determined.

[0098] In the cell detection method provided in this application embodiment, a brightness value sequence is obtained by collecting the brightness value of the corresponding brightness area of ​​each frame of cell image. For newly appearing cells, the brightness information corresponding to the cell can be determined from the brightness value sequence based on the frame number information corresponding to the newly appearing cell, which can improve the accuracy of brightness information determination.

[0099] Figure 10 This is a flowchart illustrating the motion detection result determination method provided in an embodiment of this application. The motion detection result determination method is applied to electronic devices. Figure 10 As shown, it includes the following steps: S51, based on the brightness information and the preset brightness threshold, determine the target motion direction corresponding to each cell.

[0100] In at least one embodiment of this application, cells with a brightness greater than or equal to a preset brightness threshold are pre-set to enter the right channel, and cells with a brightness less than the preset brightness threshold are pre-set to enter the left channel.

[0101] S52, if the direction of motion is the same as the direction of motion of the target, then the motion detection result of each cell is determined to be correct.

[0102] S53, if the direction of motion is not the same as the target direction of motion, then the motion detection result of each cell is determined to be a motion error.

[0103] In the cell detection method provided in this application embodiment, by detecting the brightness information of the cell and the direction of cell movement, it is possible to detect whether the direction of cell movement is correct.

[0104] Please see Figure 11 , Figure 11This is a schematic diagram of the structure of the cell detection device provided in an embodiment of this application. In some embodiments, the cell detection device 20 may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the cell detection device 20 may be stored in the memory of the electronic device 10 and executed by at least one processor to perform (see details). Figure 2 (Description) The function of pollution treatment.

[0105] In this embodiment, the cell detection device 20 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an image acquisition module 201, a model invocation module 202, a position determination module 203, an orientation determination module 204, a brightness determination module 205, and a motion detection module 206.

[0106] The term "module" as used in this application refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0107] The image acquisition module 201 can be used to acquire multiple frames of cell images.

[0108] The bounding box determination module 202 can be used to determine the bounding box corresponding to each cell in each frame of cell image.

[0109] The position determination module 203 can be used to determine the position information of the bounding box corresponding to each cell in the multi-frame cell image.

[0110] The direction determination module 204 can be used to determine the movement direction of each cell based on the position information.

[0111] The brightness determination module 205 can be used to determine the brightness information of each cell based on the multi-frame cell images.

[0112] The motion detection module 206 can be used to determine the motion detection result of each cell based on the brightness information and the direction of motion.

[0113] It is understood that the cell detection device 20 and the cell detection method of the above embodiments belong to the same inventive concept. The specific implementation of each module of the cell detection device 20 corresponds to each step of the cell detection method in the above embodiments, and will not be repeated here.

[0114] The module division described above is a logical functional division, and other division methods may be used in actual implementation. Furthermore, the functional modules in the various embodiments of this application can be integrated into the same processing unit, or each module can exist physically separately, or two or more modules can be integrated into the same unit. The integrated modules described above can be implemented in hardware or in a combination of hardware and software functional modules.

[0115] Figure 12 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 12 As shown, the electronic device 10 includes a memory 11, at least one processor 12, and at least one communication bus 13. The processor 12 is used to implement a cell detection method when executing a computer program stored in the memory 11. The at least one communication bus 13 is configured to enable communication between the memory 11 and the at least one processor 12. In some embodiments, the electronic device 10 may also connect to a client device, which includes, but is not limited to, any electronic product that can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smartphone, digital camera, etc.

[0116] It should be noted that electronic device 10 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0117] In some embodiments, the electronic device 10 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0118] In some embodiments, the memory 11 stores a computer program that, when executed by at least one processor 12, implements all or part of the steps in the cell detection method, as described above. The memory 11 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0119] In some embodiments, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of the electronic device 10, etc.

[0120] In some embodiments, at least one processor 12 is the control unit of the electronic device 10, connecting various components of the electronic device 10 via various interfaces and lines. It executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions and process data. For example, when at least one processor 12 executes a computer program stored in the memory, it implements all or part of the steps of the cell detection method in this embodiment; or it implements all or part of the functions of the cell detection device. At least one processor 12 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0121] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause an electronic device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods of the various embodiments of this application.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0123] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0125] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that it can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in the specification may also be implemented by a single element or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A cell detection method, characterized in that, The cell detection method includes: Acquire multiple frames of cell images; Determine the bounding box corresponding to each cell in each frame of cell image; Determine the position information of the bounding box corresponding to each cell in the multi-frame cell image; Based on the location information, the direction of movement of each cell is determined; Based on the multi-frame cell images, the brightness information of each cell is determined; Based on the brightness information and the direction of motion, the motion detection result of each cell is determined.

2. The cell detection method as described in claim 1, characterized in that, Determining the bounding box corresponding to each cell includes: Determine the channel position corresponding to each frame of cell image; Based on the channel position, the segmentation region corresponding to each frame of cell image is determined, so that the preset segmentation model determines the bounding box corresponding to each cell in each frame of cell image according to the segmentation region.

3. The cell detection method as described in claim 1 or 2, characterized in that, The method further includes: Determine the cell sample and the corresponding cell size; Based on the cell size, the bounding box information is determined, and the bounding box information includes one or more of the following: width information, height information, and aspect ratio information.

4. The cell detection method as described in claim 1, characterized in that, Determining the bounding box corresponding to each cell in each frame of cell image includes: The preset segmentation model is called to process each frame of cell image to obtain the cell mask corresponding to each cell; Determine the bounding rectangle corresponding to the cell mask, and use the bounding rectangle as the bounding box corresponding to each cell.

5. The cell detection method as described in claim 1, characterized in that, Determining the position information of the bounding box corresponding to each cell in the multi-frame cell image includes: Select the first cell image according to the preset time sequence; Determine the first bounding box corresponding to each cell in the first cell image, the tracking marker corresponding to each first bounding box, and the first feature of each cell; Based on the first feature, the feature corresponding to each cell in the second cell image is predicted to obtain the predicted feature. The second cell image is adjacent to the first cell image and is after the first cell image. Determine the second feature corresponding to each cell in the second cell image; Based on the predicted features and the second features, a tracking marker corresponding to each cell in the second cell image is determined; Based on a first feature and a second feature that have the same tracking marker, the position information of the bounding box corresponding to each cell in the multi-frame cell image is determined.

6. The cell detection method as described in claim 1, characterized in that, Determining the movement direction of each cell based on the location information includes: Based on the location information, the movement trajectory of each cell is determined; The channel information corresponding to the motion trajectory is determined, and the channel information is used as the motion direction of each cell.

7. The cell detection method as described in claim 1, characterized in that, The step of determining the brightness information of each cell based on the multi-frame cell images includes: Determine the brightness value of the corresponding brightness region in each frame of cell image; A sequence of brightness values ​​is obtained based on a preset time order; Determine the frame number information corresponding to the cell image containing the preset cell; Based on the frame number information, a target brightness value is selected from the brightness value sequence as the brightness information corresponding to the preset cell.

8. The cell detection method as described in claim 1, characterized in that, The determination of the motion detection result for each cell based on the brightness information and the direction of motion includes: Based on the brightness information and the preset brightness threshold, the target motion direction corresponding to each cell is determined; If the direction of motion is the same as the direction of motion of the target, then the motion detection result of each cell is determined to be correct. If the direction of motion is not the same as the target direction of motion, then the motion detection result of each cell is determined to be a motion error.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the cell detection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by the controller, implements the cell detection method as described in any one of claims 1 to 8.

Images