A method and system for tracking nematode motion based on a multi-camera array

Abstract

This application provides a method and system for nematode motion tracking based on a multi-camera array, which can be applied to the field of combining biological behavioral research techniques with computer vision. The method includes: sending a acquisition command to the multi-camera array, so that the array simultaneously acquires images of a culture dish inoculated with nematodes from the locations of each camera in the array; in response to the image sequence acquired simultaneously by the multi-camera array, calling a pre-stored stitching template matching the locations of each camera, stitching the image sequence to obtain a panoramic image, where coordinate transformation parameters describe the geometric mapping relationship between two images in any two-image stitching, and the stitching seam position indicates the stitching boundary; and based on the panoramic images corresponding to multiple image sequences acquired at predetermined time intervals, performing motion tracking on the nematodes to generate their motion trajectories.

Description

Technical Field

[0001] This application relates to the field of combining biological behavioral research techniques with computer vision, specifically to a method and system for tracking the motion of nematodes based on a multi-camera array. Background Technology

[0002] In behavioral studies of Caenorhabditis elegans, achieving high-throughput and high-precision motion tracking is a key technological requirement. However, existing technologies struggle to balance macroscopic vision with microscopic resolution and often fail to meet the requirements for synchronous and undisturbed observation of nematode colonies. Summary of the Invention

[0003] In view of the above problems, this application provides a method and system for tracking the motion of nematodes based on a multi-camera array.

[0004] According to a first aspect of this application, a method for tracking the motion of nematodes based on a multi-camera array is provided, comprising: sending a collection command to the multi-camera array so that the multi-camera array, upon receiving the collection command, simultaneously collects images of a culture dish inoculated with nematodes from the locations of each camera in the multi-camera array; in response to the image sequence acquired by the multi-camera array at the same time, calling a pre-stored stitching template matching the locations of each camera, stitching the image sequence to obtain a panoramic image, wherein the stitching template includes multiple sets of intermediate stitching parameters, the intermediate stitching parameters including coordinate transformation parameters and stitching seam positions, the coordinate transformation parameters being used to describe the geometric mapping relationship between two images in any two-image stitching, and the stitching seam positions being used to indicate the stitching boundary; and tracking the motion of nematodes based on the panoramic images corresponding to the multiple image sequences acquired at predetermined time intervals, thereby generating the motion trajectory of the nematodes.

[0005] According to an embodiment of this application, each set of intermediate stitching parameters corresponds to each image stitching in a predetermined stitching path of the multi-camera array. The multiple sets of intermediate stitching parameters are obtained through the following steps: controlling the multi-camera array to acquire calibration images of a culture dish with a calibration plate placed on it at the same time to obtain a calibration image sequence. The culture dish with the calibration plate placed on it is the same size as the culture dish inoculated with nematodes, and the surface of the calibration plate has a standard pattern for feature detection. For each image stitching: extracting descriptive features of key points of the first image and the second image participating in the image stitching respectively; matching the key points of the first image and the second image based on the similarity between the descriptive features to obtain matching point pairs; calculating the coordinate transformation parameters that map the second image to the first image based on the matching point pairs, and performing the transformation to obtain the transformed image; calculating the stitching seam with the minimum visual difference using an energy minimization algorithm based on the overlapping area between the transformed image and the first image to obtain the stitching seam position.

[0006] According to an embodiment of this application, the coordinate transformation parameters include a homography matrix; based on matching point pairs, the coordinate transformation parameters for mapping the second image to the first image are calculated, and the transformation is performed to obtain the transformed image, including: using a random sampling consensus algorithm to fit an initial homography matrix from the matching point pairs to map the second image to the coordinate system of the first image; using the initial homography matrix as an initial value, optimizing the initial homography matrix by minimizing the reprojection error of the matching point pairs to obtain a homography matrix; based on the homography matrix, mapping the second image to the coordinate system of the first image to obtain a transformed image geometrically aligned with the first image.

[0007] According to an embodiment of this application, a pre-stored stitching template matching the location of each camera is invoked to stitch an image sequence to obtain a panoramic image. This includes: sequentially invoking intermediate stitching parameters corresponding to the image stitching according to a predetermined stitching path, and performing multiple stitching steps; wherein the current stitching step in the multiple stitching steps includes: determining a first current image and a second current image from the image sequence or generated intermediate images based on the image source identifier targeted by the current stitching step; transforming the second current image based on coordinate transformation parameters corresponding to the current stitching step to obtain a transformed second current image; correcting the brightness of the transformed second image based on pixel values ​​in the overlapping area between the transformed second current image and the first current image, so that the brightness of the corrected second image is the same as that of the first current image; stitching the corrected second image and the first current image based on the stitching seam position corresponding to the current stitching step to obtain a currently generated intermediate image; and determining the intermediate image obtained in the last stitching step as the panoramic image.

[0008] According to an embodiment of this application, motion tracking of nematodes is performed based on panoramic images corresponding to multiple image sequences acquired at predetermined time intervals to generate the nematode's motion trajectory. This includes: based on the estimated motion region of the nematode in the culture dish, sequentially performing local enhancement on the estimated motion region in the panoramic image to obtain a target image sequence; inputting the target image sequence into a nematode behavior analysis tool to output the nematode's motion trajectory.

[0009] According to an embodiment of this application, a multi-camera array includes a master camera and multiple slave cameras; sending an acquisition command to the multi-camera array includes: sending an acquisition command to the master camera so that the master camera responds to the acquisition command and generates an exposure trigger signal; wherein the exposure trigger signal is used to synchronously drive the multiple slave cameras so that the master camera and the multiple slave cameras acquire images of the culture dish within the same exposure cycle, thereby obtaining an image sequence at the same moment.

[0010] According to an embodiment of this application, each camera in the multi-camera array is an industrial camera of the same model and with the same parameters; synchronous driving is achieved through the following steps: configuring the trigger port of the master camera to output mode, configuring the trigger ports of each slave camera to input mode, and connecting the trigger port of the master camera to the trigger port of each slave camera through physical signal lines.

[0011] Another aspect of this application provides a nematode motion tracking system based on a multi-camera array, comprising: a client for sending acquisition commands to the multi-camera array; a multi-camera array synchronous imaging device, communicatively connected to the client, for acquiring images of a culture dish inoculated with nematodes from the locations of each camera in the multi-camera array simultaneously in response to the acquisition command; and an image processing device, communicatively connected to the multi-camera array synchronous imaging device, for calling a pre-stored stitching template matching the locations of each camera to stitch the image sequence acquired by the multi-camera array simultaneously, thereby obtaining a panoramic image. The stitching template includes multiple sets of intermediate stitching parameters, including coordinate transformation parameters and stitching seam positions. The coordinate transformation parameters describe the geometric mapping relationship between two images in any two-image stitching, and the stitching seam positions indicate the stitching boundaries. Based on the panoramic images corresponding to the multiple image sequences acquired at predetermined time intervals, the system tracks the motion of the nematodes and generates the nematode's motion trajectory.

[0012] According to an embodiment of this application, a multi-camera array synchronous imaging device includes: a stage for placing a culture dish; and a multi-camera array disposed directly above the stage, wherein the optical axis of the lens of each camera in the multi-camera array is perpendicular to the plane where the culture dish is located, and there is an overlapping area between the viewing angles of adjacent cameras.

[0013] According to an embodiment of this application, the multi-camera array synchronous imaging device further includes: multiple light sources, uniformly arranged around the lens optical axes of the multiple cameras, and located between the stage and the multi-camera array.

[0014] According to embodiments of this application, a method for tracking nematode movement based on a multi-camera array is provided. By using a multi-camera array, each camera can spatially cover a portion of the culture dish, overcoming the low resolution problem caused by adjusting the focus of a single camera to expand the field of view. Furthermore, a data acquisition command is sent to the multi-camera array, synchronously triggering the entire group of cameras to acquire images at the same time, effectively avoiding motion artifacts caused by temporal deviations. A global field of view can be obtained without moving the imaging platform, enabling continuous and high-resolution observation of nematode movement behavior in the culture dish, and avoiding interference to the nematodes caused by mechanical movement. In addition, since the surface texture of the agar culture dish where the nematodes are located is sparse, a panoramic image is obtained by stitching together the image sequence acquired by the camera array by calling a pre-stored stitching template matched to the location of each camera. This overcomes the problem of low success rate in directly matching and stitching experimental images, ensuring the stability, accuracy, and efficiency of panoramic image generation. This is beneficial for analyzing the fine behavioral parameters of multiple nematodes, improving the throughput and accuracy of nematode experimental research. Attached Figure Description

[0015] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart of a nematode motion tracking method based on a multi-camera array according to an embodiment of this application is shown;

[0017] Figure 2(a) shows a partial image simultaneously acquired by four cameras according to an embodiment of this application;

[0018] Figure 2(b) shows a panoramic image generated by stitching together local images synchronously acquired by four cameras according to an embodiment of this application;

[0019] Figure 3 The diagram shows the movement trajectories of six wild-type nematodes over 30 minutes according to an embodiment of this application.

[0020] Figure 4 A diagram showing the motion state of a single nematode over time according to an embodiment of this application is provided.

[0021] Figure 5 A schematic diagram of a nematode motion tracking system based on a multi-camera array according to an embodiment of this application is shown.

[0022] [Attached image labels]

[0023] 1-Main camera, 2-First slave camera, 3-Second slave camera, 4-Third slave camera, 5-First LED lighting source, 6-Second LED lighting source, 7-Third LED lighting source, 8-Fourth LED lighting source, 9-Brightness controller, 10-3D support, 11-Platform, 12-Z-axis manual lifting platform, 13-Base, 14-Image processing device, 15-Data communication cable. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, application, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0029] For existing technical solutions, in single-camera imaging schemes, due to physical constraints at the hardware level, there is an irreconcilable contradiction between the field of view and image resolution, which cannot simultaneously meet the needs of large-scale observation of groups and high-resolution detail capture of individuals; in dynamic platform tracking schemes, their serial working mode results in low throughput, and the mechanical movement itself will interfere with the natural behavior of nematodes, making it impossible to achieve true synchronous and undisturbed observation.

[0030] To address the above problems, embodiments of this application provide a method for tracking the motion of nematodes based on a multi-camera array, comprising: sending a acquisition command to the multi-camera array so that the multi-camera array, upon receiving the acquisition command, simultaneously acquires images of a culture dish inoculated with nematodes from the locations of each camera in the multi-camera array; in response to the image sequence acquired by the multi-camera array at the same time, calling a pre-stored stitching template matching the locations of each camera, and stitching the image sequence to obtain a panoramic image, wherein the stitching template includes multiple sets of intermediate stitching parameters, including coordinate transformation parameters and stitching seam positions, wherein the coordinate transformation parameters are used to describe the geometric mapping relationship between two images in any two-image stitching, and the stitching seam positions are used to indicate the stitching boundary; and tracking the motion of nematodes based on the panoramic images corresponding to the multiple image sequences acquired at predetermined time intervals, thereby generating the motion trajectory of the nematodes.

[0031] The following will be through Figures 1-4 This application provides a detailed description of a nematode motion tracking method based on a multi-camera array, according to an embodiment of the present application.

[0032] Figure 1 A flowchart of a nematode motion tracking method based on a multi-camera array according to an embodiment of this application is shown.

[0033] like Figure 1 As shown, the nematode motion tracking method based on a multi-camera array in this embodiment includes operations S110 to S130.

[0034] In operation S110, an acquisition command is sent to the multi-camera array so that the multi-camera array can simultaneously acquire images of the culture dish inoculated with nematodes from the locations of each camera in the multi-camera array after receiving the acquisition command.

[0035] A multi-camera array can consist of multiple high-resolution industrial cameras of the same model and parameters, fixed to a three-dimensional support in an M-row × N-column matrix. For example, four Hikvision MV-CS200-10UM cameras of the same model and with identical parameters can be fixed to a three-dimensional support in a 2-row × 2-column matrix. The optical axes of the camera lenses are perpendicular to the imaging plane, and through precise adjustment, there is a partial overlap between the fields of view of adjacent cameras.

[0036] When the multi-camera array receives the acquisition command, all cameras can acquire images of the culture dish inoculated with nematodes on the stage from their respective fixed spatial positions at the same physical moment, thereby obtaining a set of original image sequences that are strictly synchronized in time and have overlapping areas in space.

[0037] In operation S120, in response to the image sequence acquired by the multi-camera array at the same time, a pre-stored stitching template matching the location of each camera is called to stitch the image sequence to obtain a panoramic image.

[0038] The stitching template includes multiple sets of intermediate stitching parameters, including coordinate transformation parameters and stitching seam positions. The coordinate transformation parameters are used to describe the geometric mapping relationship between two images in the stitching of any two images, and the stitching seam positions are used to indicate the stitching boundary.

[0039] For example, an image sequence could be four images of a culture dish inoculated with nematodes, taken at the same time from different locations by four cameras in synchronous trigger mode.

[0040] Once the multi-camera array completes image acquisition and generates a synchronized image sequence at the same exposure time, the system immediately reads and loads a pre-stored stitching template matching the locations of each camera. This template contains multiple sets of intermediate stitching parameters, each corresponding to a single image stitching process. These parameters include coordinate transformation parameters that map the second image to the coordinate system of the first image, and stitching seam position information used to determine the optimal stitching boundary in the overlapping area. Based on this template, the system performs corresponding coordinate transformations and stitching operations on the image sequence to generate a seamless panoramic image.

[0041] In operation S130, based on the panoramic images corresponding to multiple image sequences acquired at predetermined time intervals, the nematode's motion is tracked, and the nematode's motion trajectory is generated.

[0042] Based on the continuous acquisition of panoramic images corresponding to each image sequence at fixed time intervals, the panoramic images are input into professional nematode behavior analysis software, such as the open-source Tierpsy Tracker, which automatically identifies the outline and skeleton of all nematodes in each frame and associates the same nematode between consecutive frames to form a complete motion trajectory.

[0043] By using a multi-camera array, each camera can spatially cover a portion of the culture dish, overcoming the low resolution problem caused by single-camera focusing to expand the field of view. Furthermore, by sending acquisition commands to the multi-camera array, the entire group of cameras is synchronously triggered to acquire images at the same time, effectively avoiding motion artifacts caused by temporal deviations. A global field of view can be obtained without moving the imaging platform, enabling continuous and high-resolution observation of nematode movement behavior in the culture dish, and avoiding interference to the nematodes caused by mechanical movement. In addition, since the surface texture of the agar culture dish where the nematodes are located is sparse, a panoramic image is obtained by stitching together the image sequences acquired by the camera array using pre-stored stitching templates matched to the locations of each camera. This overcomes the low success rate of directly performing feature matching and stitching on experimental images, ensuring the stability, accuracy, and efficiency of panoramic image generation. This is beneficial for analyzing the fine behavioral parameters of multiple nematodes, improving the throughput and accuracy of nematode experimental research.

[0044] According to the embodiments of the application, for example, Figure 1 Operation S120, as shown, involves multiple sets of intermediate stitching parameters, each corresponding to a specific image stitching step in a predetermined stitching path of the multi-camera array. These intermediate stitching parameters are obtained through the following steps: controlling the multi-camera array to simultaneously acquire calibration images of a culture dish containing a calibration plate, resulting in a calibration image sequence. The culture dish containing the calibration plate is identical in size to the culture dish inoculated with nematodes, and the surface of the calibration plate has a standard pattern for feature detection. For each image stitching step: extracting descriptive features of key points from both the first and second images involved in the stitching; matching key points of both the first and second images based on the similarity between the descriptive features, obtaining matching point pairs; calculating coordinate transformation parameters to map the second image to the first image based on the matching point pairs, and performing the transformation to obtain the transformed image; and calculating the stitching seam with the minimum visual difference using an energy minimization algorithm based on the overlapping area between the transformed image and the first image, thus obtaining the stitching seam position.

[0045] For example, a calibration plate with high contrast and a standard pattern can be placed on a blank culture dish of the same size as the nematode culture dish, and both can be placed together in the center of the imaging area of ​​the stage to ensure that the plane on which the plate is located is completely consistent with the imaging plane of the nematode sample in subsequent experiments, thereby ensuring the accuracy of the calibration parameters. A multi-camera array is controlled to simultaneously acquire calibration images of the culture dish with the calibration plate placed on it, resulting in a calibration image sequence.

[0046] When stitching together the calibrated image sequence, for each stitch, the Scale-Invariant Feature Transform (SIFT) algorithm is run on the first and second images with overlapping views, respectively, to obtain a feature set for each image. The feature set includes keypoints and descriptive features for each keypoint. Each keypoint contains location, scale, and orientation information. The descriptive feature can be a 128-dimensional feature vector used to represent the local visual content around the corresponding keypoint.

[0047] The similarity of descriptive features between the feature sets of the first and second images can be calculated using Euclidean distance; the smaller the distance, the more similar the two features are. Based on the similarity of descriptive features, the nearest neighbor to second nearest neighbor ratio method can be used to match key points between the two images, resulting in multiple initial sets of matching point pairs M_initial = {m_{1,2}}, where m_{1,2} represents the list of matching point pairs between images 1 and 2.

[0048] During the matching process, for each descriptive feature in the first image feature set, the first nearest neighbor descriptive feature with the smallest Euclidean distance and the second nearest neighbor descriptor with the smallest Euclidean distance are found in the second image feature set. The ratio of these two distances is calculated. If the ratio is lower than a set threshold, the first nearest neighbor is accepted as a reliable match; otherwise, it is not accepted.

[0049] Based on the matching point pairs, coordinate transformation parameters that map the second image to the coordinate system of the first image can be calculated. These parameters describe the transformation method of the second image, including pixel translation, rotation, scaling, and even projection. A geometric transformation is then performed on the second image according to these transformation parameters to generate a transformed image in the same coordinate system as the first image. This ensures that pixels representing the same real physical point in both images are located at exactly the same coordinate positions in the final composite image.

[0050] In the region where the transformed image overlaps with the first image, an energy function is defined that incorporates color and gradient differences. This function comprehensively measures the degree of unnaturalness that would result if the two images were cut and stitched together along a seam. It mainly consists of two parts: first, the color difference energy, which reflects the direct difference in brightness and color between corresponding pixels on both sides of the seam; and second, the gradient difference energy, which characterizes the continuity of texture and edge direction changes on both sides of the seam. The lower the value of the energy function, the less noticeable the visual artifacts of stitching along that point.

[0051] Pixels in the overlapping region are treated as nodes in the graph, and the adjacency relationships between pixels are considered as edges. Energy values ​​are assigned to the corresponding edges as weights. By applying an energy minimization algorithm, a cutting path traversing the overlapping region can be efficiently found in the graph, minimizing the sum of the weights of the edges traversed by this path, i.e., the total energy. This path is the sought-after optimal stitching seam, and its position is recorded as the stitching seam position, i.e., a binary mask that defines which input image each pixel should ultimately originate from on the panoramic canvas.

[0052] According to embodiments of this application, each set of intermediate stitching parameters corresponds to each image stitching in a predetermined stitching path of the multi-camera array. For example, when the multi-camera array contains four cameras, the predetermined stitching path can be the stitching order of the four images, including naming the four images by position: top left, top right, bottom left, bottom right, first stitching horizontally (top left + top right, bottom left + bottom right), and then stitching vertically (top and bottom parts).

[0053] By acquiring calibration images with standard patterns, the system can calculate the necessary coordinate transformation parameters and optimal stitching seam parameters with high precision in a single operation before the experiment, and then solidify them as a stitching template. This ensures that image stitching can be achieved efficiently and accurately in formal experiments without the need for time-consuming online feature calculations, improving the overall practicality, efficiency, and anti-interference capability of the system, and laying a reliable data foundation for high-throughput, high-precision continuous motion tracking.

[0054] According to an embodiment of this application, the coordinate transformation parameters include a homography matrix; based on matching point pairs, the coordinate transformation parameters for mapping the second image to the first image are calculated, and the transformation is performed to obtain the transformed image, including: using a random sampling consensus algorithm to fit an initial homography matrix from the matching point pairs to map the second image to the coordinate system of the first image; using the initial homography matrix as an initial value, optimizing the initial homography matrix by minimizing the reprojection error of the matching point pairs to obtain a homography matrix; based on the homography matrix, mapping the second image to the coordinate system of the first image to obtain a transformed image geometrically aligned with the first image.

[0055] A homography matrix can be a 3x3 matrix that describes the mapping transformation relationship between matching point pairs on two different viewpoints, used to map a point on one plane to a corresponding point on another plane.

[0056] For each initial matching point pair m_{1,2} in the initial matching point pair set, the Random Sample Consensus (RANSAC) algorithm is run to fit a homography matrix model. Outliers that do not conform to the model are then removed, resulting in a cleaned-up set of high-confidence matching point pairs. Based on this cleaned-up set of high-confidence matching point pairs, an initial homography matrix is ​​fitted to map the second image to the coordinate system of the first image.

[0057] Bundle adjustment optimization is a global optimization method that, through holistic adjustment, achieves an optimal state of consistent imaging geometry in a multi-camera system. Specifically, this method inputs all purified high-confidence matching point pairs and all initial homography matrices into an optimization model. Within this model, all initial homography matrices are iteratively optimized simultaneously to obtain the final homography matrix. The optimization objective is to minimize the sum of squares of the differences (i.e., reprojection errors) between the theoretical position of each 3D feature point reprojected onto the image plane of each camera according to the optimized camera parameters and its actual detected matching point position in the corresponding image. Based on the homography matrix, the second image is mapped to the coordinate system of the first image, resulting in a transformed image geometrically aligned with the first image.

[0058] This process not only ensures the accuracy of geometric mapping between adjacent images, but also guarantees the consistency and reliability of the entire stitching template in multiple calls, thus laying a precise geometric foundation for generating high-quality, seamless panoramic image sequences and directly improving the availability and accuracy of subsequent motion tracking.

[0059] Figure 2(a) shows a partial image simultaneously acquired by four cameras according to an embodiment of the present application; Figure 2(b) shows a panoramic image generated by stitching together the partial images simultaneously acquired by four cameras according to an embodiment of the present application.

[0060] According to embodiments of this application, for example, Figure 1The operation S120 shown calls a pre-stored stitching template matching the location of each camera to stitch the image sequence to obtain a panoramic image. This includes: according to a predetermined stitching path, sequentially calling intermediate stitching parameters corresponding to the image stitching and executing multiple stitching steps. The current stitching step in these multiple stitching steps includes: determining a first current image and a second current image from the image sequence or generated intermediate images based on the image source identifier corresponding to the current stitching step; transforming the second current image based on the coordinate transformation parameters corresponding to the current stitching step to obtain a transformed second current image; correcting the brightness of the transformed second image based on the pixel values ​​in the overlapping area between the transformed second current image and the first current image, so that the brightness of the corrected second image is the same as that of the first current image; stitching the corrected second image and the first current image based on the stitching seam position corresponding to the current stitching step to obtain the currently generated intermediate image; and determining the intermediate image obtained in the last stitching step as the panoramic image.

[0061] The system sequentially calls the intermediate stitching parameters corresponding to each image stitching operation according to the stitching order, and performs the stitching operation. As shown in Figure 2(a), the local images acquired synchronously by four cameras are stitched together using the stitching method described above in this application to generate a high-resolution panoramic image as shown in Figure 2(b).

[0062] In each stitching operation, the system determines the first and second current images to be stitched from the original image sequence or the generated intermediate result images, based on the image source identifier corresponding to that step.

[0063] Using the coordinate transformation parameters corresponding to this stitching step, a geometric transformation is performed on the second current image to align it spatially with the first current image, resulting in the transformed second current image.

[0064] Based on the pixel values ​​of the transformed second current image and the first current image within the overlapping region, the gain / bias coefficients of the second current image relative to the first current image are calculated by minimizing the difference in the overlapping region. These can be linear transformation parameters (gain and bias coefficients) to obtain the exposure compensation coefficients, which are the set of gain and bias coefficients for the three channels (Red, Green, Blue, or RGB for short). The brightness of the transformed second image is then corrected according to these exposure compensation coefficients to ensure that the corrected second image has the same brightness as the first current image.

[0065] Based on the pre-stored stitching seam position information corresponding to this step, the brightness-corrected second current image and the first current image are merged along the stitching seam to generate the intermediate image for this step. This process is repeated until all image sequences are merged, and the intermediate image generated by the last stitching operation is output as the final panoramic image.

[0066] Based on the above stitching process, the speed and success rate of stitching can be improved, and the stability of the output panoramic image sequence in terms of geometric consistency and visual quality can be guaranteed, providing an important data foundation for subsequent long-term, high-precision motion tracking analysis.

[0067] According to embodiments of this application, for example, Figure 1 The operation S230 shown involves tracking the movement of nematodes based on panoramic images corresponding to multiple image sequences acquired at predetermined time intervals, generating the movement trajectory of the nematodes. This includes: based on the estimated movement area of ​​the nematodes in the culture dish, sequentially performing local enhancement on the estimated movement area in the panoramic image to obtain a target image sequence; inputting the target image sequence into a nematode behavior analysis tool to output the movement trajectory of the nematodes.

[0068] One or more predicted movement regions can be determined based on the typical activity range of nematodes in a petri dish or a user-defined area of ​​interest. For the predicted movement regions in the panoramic image, a background modeling method based on temporal median or Gaussian mixture model is used to estimate and subtract the static background, highlighting the moving nematodes.

[0069] In a schematic way, a background model is constructed using a background modeling method, and a panoramic image of the region containing valid data is compared with the model. Static background parts that are fixed and unchanging in the image are faded or removed, while the moving nematode is completely and clearly preserved and highlighted because its pixels are significantly different from the background model, and the movement trajectory of the nematode is generated.

[0070] Background modeling methods can be categorized into two types: First, a median-based method. This method takes the median value of each pixel within a short timeframe (e.g., the first N frames) and uses it as the estimated background value for that pixel, effectively ignoring occasional outliers. Second, a Gaussian mixture model method. This method creates a statistical model for each pixel in the image, composed of multiple Gaussian distributions, to describe the various colors or brightness levels that the pixel might exhibit and their variations. Each Gaussian distribution represents a possible state, such as a stable background, a transient shadow, or a foreground object. The system dynamically updates the parameters of each distribution based on the captured image input and identifies the distribution with high weights and low variance as the background.

[0071] A contrast-limited adaptive histogram equalization (CLAHE) image enhancement algorithm is applied to the panoramic image after removing the static background. The panoramic image is divided into blocks, and the grayscale histogram of each block is calculated and limited to prevent excessive magnification of subtle textures other than those of the nematode. Histogram equalization is then performed on each block to stretch local contrast. Finally, a weighted averaging method is used to ensure natural transitions between blocks, resulting in a target image sequence with clear local details, making the nematode outline more prominent.

[0072] Input the target image sequence into the nematode behavior analysis tool, and the tool will output the nematode's movement trajectory. The output can be in two ways: one is as an image sequence, saved as a series of high-resolution bitmap files; the other is as a synthesized video, encoded and compressed (e.g., using MJPG or H.264 encoding) into a continuous video file for easy observation and playback.

[0073] Figure 3 The diagram shows the movement trajectories of six wild-type nematodes over 30 minutes according to an embodiment of this application. Figure 4 A diagram showing the motion state of a single nematode over time according to an embodiment of this application is provided.

[0074] The following is a further explanation using a specific example.

[0075] Adult nematodes (N2 strain) containing numerous eggs were selected and collected by washing them from the culture plate with M9 buffer. After centrifugation, the supernatant was removed, and lysis buffer was added and allowed to stand for 4 minutes to dissolve the adult body wall and release the internal eggs. After most of the nematodes had lysed, they were centrifuged again and the supernatant was removed. After washing twice with M9 buffer, the precipitate was resuspended and added dropwise to NGM culture plates inoculated with OP50 bacteria, and incubated at 20°C.

[0076] After 3 days of culture, the nematodes were transferred to NGM plates not inoculated with OP50, where they were allowed to crawl a distance of more than five body lengths to remove food attached to their bodies, and then transferred to large NGM plates. Using the nematode movement tracking method described above in this application, six wild-type nematodes were tracked within 0 minutes, yielding the following results: Figure 3 The movement trajectory is shown. Furthermore, based on the aforementioned nematode behavior analysis tools, the movement state of a single nematode can be analyzed over time, such as... Figure 4 As shown, this system was used for continuous image acquisition and motion tracking during the 30-minute period of free crawling of the nematode.

[0077] This processing strategy effectively highlights the morphology and outline of nematodes within the region, while significantly reducing the amount of data processing required for irrelevant background areas.

[0078] According to an embodiment of this application, a multi-camera array includes a master camera and multiple slave cameras; sending an acquisition command to the multi-camera array includes: sending an acquisition command to the master camera so that the master camera responds to the acquisition command and generates an exposure trigger signal; wherein the exposure trigger signal is used to synchronously drive the multiple slave cameras so that the master camera and the multiple slave cameras acquire images of the culture dish within the same exposure cycle, thereby obtaining an image sequence at the same moment.

[0079] The multi-camera array includes a master camera and multiple slave cameras. When the master camera receives a data acquisition command, it responds to the command and outputs an exposure trigger signal. The slave cameras receive the exposure trigger signal from the master camera. At the same time, each camera acquires images of the culture dish inoculated with nematodes at its location during the same exposure cycle, thus obtaining an image sequence.

[0080] This method eliminates time deviations caused by camera time-division operation or software delays, resulting in image sequences with strict physical time consistency, effectively avoiding motion blur, displacement artifacts, or object shape distortion caused by different shooting times.

[0081] According to an embodiment of this application, each camera in the multi-camera array is an industrial camera of the same model and with the same parameters; synchronous driving is achieved through the following steps: configuring the trigger port of the master camera to output mode, configuring the trigger ports of each slave camera to input mode, and connecting the trigger port of the master camera to the trigger port of each slave camera through physical signal lines.

[0082] In the multi-camera array, each camera is an industrial camera of the same model and with identical parameters. One camera is selected as the master camera, and its configurable I / O interface, which can be set to output an exposure trigger signal at the start of exposure, is configured via client software. The remaining cameras in the array are slave cameras, all configured in external trigger mode, and their hardware trigger input interfaces are connected to the master camera's output interface via physical signal lines.

[0083] This design transforms synchronization commands from software commands that rely on operating system scheduling into signals that are directly generated and responded to by hardware circuits, achieving synchronization accuracy at the sub-millisecond or even microsecond level. This effectively eliminates random delays and jitter caused by network transmission and software processing, ensuring strict uniformity of exposure time and providing accurate raw data for subsequent image stitching and motion analysis.

[0084] Figure 5 A schematic diagram of a nematode motion tracking system based on a multi-camera array according to an embodiment of this application is shown.

[0085] like Figure 5As shown, the nematode motion tracking system based on a multi-camera array may include a client 14, a multi-camera array synchronous imaging device, and an image processing device.

[0086] Client 14 is used to send acquisition commands to the multi-camera array.

[0087] The multi-camera array synchronous imaging device communicates with the client 14 and responds to the acquisition commands issued by the client 14, simultaneously acquiring images of the culture dish inoculated with nematodes from the locations of each camera in the multi-camera array. The multi-camera array consists of a main camera 1, a first slave camera 2, a second slave camera 3, and a third slave camera 4, all of which are of the same model as the main camera 1.

[0088] Image processing device 14 is communicatively connected to multi-camera array synchronous imaging device. In response to the image sequence acquired by the multi-camera array at the same time, it calls a pre-stored stitching template that matches the location of each camera to stitch the image sequence to obtain a panoramic image. Based on the panoramic images corresponding to the multiple image sequences acquired at predetermined time intervals, it tracks the movement of the nematode and generates the movement trajectory of the nematode.

[0089] This system is not a simple combination of multiple independent modules, but rather an organic whole that integrates key technologies such as multi-camera synchronous imaging, template-based rapid stitching, and continuous motion tracking into a cohesive system through communication connections and functional coupling. This forms an end-to-end automated solution from image acquisition to trajectory output. It not only ensures strict temporal and spatial alignment of the data stream but also simplifies user operation, improves system stability and result reliability, and ultimately achieves the technical goal of high-throughput, high-precision, and non-disruptive continuous observation and quantitative analysis of nematode colonies.

[0090] According to embodiments of this application, a multi-camera array synchronous imaging device may include: a stage 11 and a multi-camera array.

[0091] The stage 11 is used to place the culture dish. A multi-camera array is positioned directly above the stage 11. The optical axes of the lenses of each camera in the multi-camera array are perpendicular to the plane where the culture dish is located, and there is an overlapping area between the viewing angles of adjacent cameras.

[0092] The multi-camera array synchronous imaging device includes a stage 11 for supporting and fixing experimental samples, such as culture dishes inoculated with nematodes. Directly above the stage 11, a multi-camera array is fixedly mounted. The cameras in this array are regularly arranged, and the optical axes of their lenses are adjusted to be perpendicular to the plane of the culture dish on the stage, ensuring that each camera images the sample from a top-down angle. Furthermore, the installation positions of adjacent cameras are precisely designed to have partially overlapping areas in their imaging ranges. This overlapping area is a prerequisite for subsequently stitching multiple local images into a complete panoramic image, providing necessary information for feature matching and geometric alignment between images.

[0093] A multi-camera array was fixedly positioned directly above the stage, with the optical axes of each camera perpendicular to the plane of the culture dish. This ensured that all cameras imaged the sample from the same orthogonal perspective, thus preventing perspective distortion and scale differences caused by tilted viewing angles at the source. Simultaneously, precise settings maintained a stable overlap between the fields of view of adjacent cameras, ensuring the feasibility and accuracy of the stitching process.

[0094] According to an embodiment of this application, the multi-camera array synchronous imaging device further includes: multiple light sources, uniformly arranged around the lens optical axes of the multiple cameras, and located between the stage 11 and the multi-camera array.

[0095] like Figure 5 As shown, the first LED illumination source 5, the second LED illumination source 6, the third LED illumination source 7, and the fourth LED illumination source 8 are all identical in model and function, and are symmetrically arranged around the imaging platform. These light sources are uniformly arranged around the optical axes of the lenses of multiple cameras in a surrounding manner, and their installation positions are within the space between the stage and the multi-camera array. This geometric arrangement allows light to be uniformly illuminated from above the culture dish at an approximately vertical or inclined angle to the entire imaging area, thus providing the nematode sample with a bright, all-around, shadow-free illumination environment.

[0096] Uniform illumination effectively reduces color differences and brightness abrupt changes in image stitching caused by uneven lighting, laying a reliable optical foundation for the subsequent generation of high-quality panoramic images. Meanwhile, the layout between the camera and the stage avoids direct interference from the light source to the camera lens and makes lighting conditions easy to control and adjust.

[0097] In another example, the multi-camera array synchronous imaging device also includes a brightness controller 9. The brightness controller 9 is connected to the first LED illumination source 5, the second LED illumination source 6, the third LED illumination source 7, and the fourth LED illumination source 8 (connection cables are in...). Figure 5 (Not shown in the image), used to control the sharpness of the image.

[0098] In yet another example, the multi-camera array synchronous imaging device also includes: a stereo support 10, a Z-axis manual lifting platform 12, a base 13, and a data communication cable 15.

[0099] The stereo support 10 serves as the frame support structure for the entire multi-camera array synchronous imaging device, providing an installation reference for each component of the device. The top of the stereo support 10 supports the main camera 1, the first slave camera 2, the second slave camera 3, and the third slave camera 4. The sides are equipped with the first LED illumination source 5, the second LED illumination source 6, the third LED illumination source 7, and the fourth LED illumination source 8. The bottom is rigidly connected to the base 13, while also providing vertical movement guidance for the Z-axis manual lifting platform 12.

[0100] The Z-axis manual lifting platform 12 can provide stable support and precise height adjustment for the platform 11, and the height can be manually adjusted along the vertical direction of the three-dimensional support 10.

[0101] One end of the data communication cable 15 is connected to the main camera 1, the first slave camera 2, the second slave camera 3, and the third slave camera 4 for transmitting image acquisition data. The other end is connected to the image processing device 14 to realize the transmission and interaction of image data. In addition, the data communication cable 15 can transmit control signals from the brightness controller 9 to realize the brightness adjustment control of the first LED lighting source 5, the second LED lighting source 6, the third LED lighting source 7, and the fourth LED lighting source 8.

[0102] The three-dimensional bracket provides a precise three-dimensional installation benchmark and Z-axis guidance. The Z-axis manual lifting platform enables adjustable stage height. The base provides stable support and integrates a brightness controller. The data communication cable enables high-speed data transmission between multiple cameras and image processing devices, as well as lighting control signal transmission. This improves imaging accuracy and stability, ease of operation and scene adaptability, and ensures the reliability and synchronization of data transmission.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0104] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0105] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for tracking the motion of nematodes based on a multi-camera array, characterized in that, The method includes: Send an acquisition command to a multi-camera array so that the multi-camera array can simultaneously acquire images of the culture dish inoculated with nematodes from the locations of each camera in the multi-camera array after receiving the acquisition command. In response to the image sequence acquired by the multi-camera array at the same time, a pre-stored stitching template matching the location of each camera is invoked to stitch the image sequence to obtain a panoramic image. The stitching template includes multiple sets of intermediate stitching parameters, including coordinate transformation parameters and stitching seam positions. The coordinate transformation parameters are used to describe the geometric mapping relationship between the two images in any two-image stitching, and the stitching seam positions are used to indicate the stitching boundary. Based on the panoramic images corresponding to the multiple image sequences acquired at predetermined time intervals, the nematode is motion tracked to generate its motion trajectory.

2. The method according to claim 1, characterized in that, Each set of intermediate stitching parameters corresponds to each image stitching step in the predetermined stitching path of the multi-camera array. These intermediate stitching parameters are obtained through the following steps: The multi-camera array is controlled to acquire calibration images of the culture dish on which the calibration plate is placed at the same time to obtain a calibration image sequence. The culture dish on which the calibration plate is placed is the same size as the culture dish inoculated with the nematodes. The surface of the calibration plate has a standard pattern for feature detection. For each image stitching: Descriptive features of key points in the first and second images involved in the image stitching are extracted respectively. Based on the similarity between the descriptive features, the key points of the first image and the second image are matched to obtain matching point pairs. Based on the matching point pairs, calculate the coordinate transformation parameters that map the second image to the first image, and perform the transformation to obtain the transformed image; Based on the overlapping area between the transformed image and the first image, the stitching seam with the minimum visual difference is calculated using an energy minimization algorithm to obtain the position of the stitching seam.

3. The method according to claim 2, characterized in that, The coordinate transformation parameters include the homography matrix; The step of calculating the coordinate transformation parameters that map the second image to the first image based on the matching point pairs, and performing the transformation to obtain the transformed image, includes: A random sampling consensus algorithm is used to fit an initial homography matrix from the matching point pairs to map the second image to the coordinate system of the first image; Using the initial homography matrix as the initial value, the initial homography matrix is ​​optimized by minimizing the reprojection error of the matching point pairs to obtain the homography matrix. Based on the homography matrix, the second image is mapped to the coordinate system of the first image to obtain the transformed image that is geometrically aligned with the first image.

4. The method according to claim 2, characterized in that, The step of calling a pre-stored stitching template matching the location of each camera to stitch the image sequence to obtain a panoramic image includes: According to the predetermined stitching path, the intermediate stitching parameters corresponding to the image stitching are called sequentially, and multiple stitching steps are performed. The current splicing step in the multiple splicing steps includes: Based on the image source identifier targeted in the current stitching step, a first current image and a second current image are determined from the image sequence or the generated intermediate images; Based on the coordinate transformation parameters corresponding to the current stitching step, the second current image is transformed to obtain the transformed second current image; Based on the pixel values ​​in the overlapping area between the transformed second current image and the first current image, the brightness of the transformed second image is corrected so that the brightness of the corrected second image is the same as that of the first current image; Based on the stitching seam position corresponding to the current stitching step, the corrected second image and the first current image are stitched together to obtain the currently generated intermediate image; The intermediate image obtained from the final stitching step is determined as the panoramic image.

5. The method according to claim 1, characterized in that, The step of tracking the motion of the nematode based on the panoramic images corresponding to each of the multiple image sequences acquired at predetermined time intervals, and generating the motion trajectory of the nematode, includes: Based on the estimated movement region of the nematode in the culture dish, the estimated movement region in the panoramic image is locally enhanced in sequence to obtain the target image sequence. The target image sequence is input into the nematode behavior analysis tool, which outputs the nematode's movement trajectory.

6. The method according to any one of claims 1 to 5, characterized in that, The multi-camera array includes a master camera and multiple slave cameras; Sending acquisition commands to the multi-camera array includes: The acquisition command is sent to the main camera, causing the main camera to respond to the acquisition command and generate an exposure trigger signal; The exposure trigger signal is used to synchronously drive multiple slave cameras, so that the master camera and multiple slave cameras acquire images of the culture dish within the same exposure cycle, thereby obtaining the image sequence at the same moment.

7. The method according to claim 6, characterized in that, Each camera in the multi-camera array is an industrial camera of the same model and with the same parameters; The synchronization drive is obtained through the following steps: Configure the trigger port of the main camera to output mode, configure the trigger ports of each of the slave cameras to input mode, and connect the trigger port of the main camera to the trigger port of each of the slave cameras through physical signal lines.

8. A nematode motion tracking system based on a multi-camera array, characterized in that, The system includes: The client is used to send acquisition commands to the multi-camera array; A multi-camera array synchronous imaging device, which is communicatively connected to the client, is used to acquire images of a culture dish inoculated with nematodes from the locations of each camera in the multi-camera array at the same time in response to the acquisition command. An image processing device, communicatively connected to the multi-camera array synchronous imaging device, is used to respond to the image sequence acquired by the multi-camera array at the same time, call a pre-stored stitching template matching the location of each camera, and stitch the image sequence to obtain a panoramic image. The stitching template includes multiple sets of intermediate stitching parameters, including coordinate transformation parameters and stitching seam positions. The coordinate transformation parameters are used to describe the geometric mapping relationship between two images in any two-image stitching, and the stitching seam positions are used to indicate the stitching boundary. Based on the panoramic images corresponding to the multiple image sequences acquired at predetermined time intervals, the motion of the nematode is tracked to generate the motion trajectory of the nematode.

9. The system according to claim 8, characterized in that, The multi-camera array synchronous imaging device includes: A stage for placing the culture dish; A multi-camera array is positioned directly above the stage, with the optical axis of each camera in the array perpendicular to the plane of the culture dish, and overlapping areas between the viewing angles of adjacent cameras.

10. The system according to claim 9, characterized in that, The multi-camera array synchronous imaging device also includes: Multiple light sources are evenly arranged around the optical axes of multiple camera lenses and located between the stage and the multi-camera array.

Images