Target object capture method, apparatus, device and storage medium

Abstract

This invention provides a target object capture method, apparatus, device, and storage medium, belonging to the field of image recognition technology. The method includes: real-time acquisition of at least two images, including an execution terminal, wherein the at least two images are captured from different angles; identification of target objects in the at least two images, determination of the position of the same target object in each image, and determination of a target object to be captured based on the position information of the execution terminal and each target object in each image; control of the execution terminal to move towards the target object to be captured until the relative position between the execution terminal and the target object meets the capture condition, and control of the execution terminal to capture the target object. This invention effectively improves the target object capture success rate by real-time acquisition of at least two images, including an execution terminal, from different angles and determining the relative position of the target object and the execution terminal based on the identified position information of each target object in each image.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a target object capture method, a target object capture device, a terminal device, and a computer-readable storage medium. Background Technology

[0002] Microorganisms, as generally understood, include, but are not limited to, bacteria, viruses, and other microorganisms. Flowing liquids typically harbor a variety of organisms, such as bacteria, viruses, and microorganisms, in large numbers. Currently, the capture of microorganisms usually involves manual microscopic examination of specific microorganisms, but this manual method is inefficient. Furthermore, due to the fluidity of liquids and the random movement of microorganisms, existing visual methods, such as calibration-based automatic acquisition of specific microorganisms, often have low success rates and fail to achieve satisfactory results. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, device, and storage medium for capturing target objects, in order to solve the problem of low success rate in obtaining microorganisms in the prior art.

[0004] To achieve the above objectives, in a first aspect of the present invention, a method for capturing a target object is provided, comprising:

[0005] Real-time acquisition of at least two images, including those at the execution end, wherein the at least two images are captured from different angles and directions;

[0006] Identify target objects in at least two images, determine the position of the same target object in each image, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image;

[0007] The execution terminal is controlled to move towards the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition, and then the execution terminal is controlled to capture the target object to be captured.

[0008] Optionally, determining the target object to be captured based on the execution terminal and the position information of each target object in each image includes:

[0009] The presence of a target object is determined based on the execution endpoint and the position information of each target object in each image.

[0010] If not, the control execution end will randomly move to a new position.

[0011] Optionally, determining the target object to be captured based on the execution terminal and the position information of each target object in each image includes:

[0012] The presence of a target object is determined based on the execution endpoint and the position information of each target object in each image.

[0013] If the target object exists and includes a group of target objects, then control the execution terminal to separate the group of target objects.

[0014] Optionally, determining the target object to be captured based on the execution terminal and the position information of each target object in each image includes:

[0015] The presence of a target object is determined based on the execution endpoint and the position information of each target object in each image.

[0016] If the target object exists and includes discrete target objects, then based on the relative position of the execution terminal and each target object in the target space, and if there is a discrete target object located in any direct movement direction of the execution terminal, the target object to be captured is determined among the discrete target objects located in that direct movement direction of the execution terminal.

[0017] Based on the relative positions of the execution endpoint and each target object in the target space, and provided that there are no discrete target objects located in any direct movement direction of the execution endpoint, the discrete target object with the smallest relative distance to the execution endpoint is determined as the target object to be captured.

[0018] Optionally, identifying the target object in the at least two images and determining the position of the same target object in each image includes:

[0019] Using one image as the base image, based on the shooting angle relationship between the at least two images, it is determined whether multiple target objects in the base image will overlap in another image. If M target objects overlap in another image, the position information of each of the M target objects in the other image is determined based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of M target objects in the other image is called overlapping target objects.

[0020] The positional information of each of the M target objects on at least two images is determined as the positional information of the overlapping target objects.

[0021] Optionally, determining whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship of the at least two images includes:

[0022] Determine the region of the largest included angle formed by the line connecting the boundary point of the target object in the base image and the coordinate transformation point of the shooting position of the other image on the base image;

[0023] Determine whether the maximum included angle regions of multiple target objects overlap. If the maximum included angle regions of M target objects overlap, then it is determined that the M target objects overlap.

[0024] Optionally, identifying the target object in the at least two images includes:

[0025] Each of the images is preprocessed to obtain non-valid targets;

[0026] Each image is input into a trained recognition model for inference to obtain the initial target object;

[0027] Remove the invalid targets from the initial target objects to obtain the final target objects.

[0028] Optionally, determining the target object to be captured among the target objects based on the execution terminal and the position information of each target object in each image includes:

[0029] The relative positions of the execution terminal and each target object in the target space are determined based on the position information of the execution terminal and each target object in each image;

[0030] Based on the relative position of the execution terminal and each target object in the target space, the target object to be captured is determined among the target objects within the preset relative position threshold range.

[0031] Optionally, the shooting angles of the at least two images include at least a set of mutually perpendicular shooting angles.

[0032] Optionally, the target object is a microorganism, and the execution end is a straw capable of blowing air;

[0033] The step of controlling the execution terminal to separate the target object group includes: controlling the execution terminal to blow air into the target object group to separate the target objects;

[0034] The control of the execution terminal to capture the target object to be captured includes: controlling the execution terminal to absorb the target object to be captured.

[0035] In a second aspect of the invention, a target object capturing device is provided, comprising:

[0036] The image acquisition module is configured to acquire at least two images from the execution end in real time, wherein the at least two images are captured from different angles and directions.

[0037] The recognition module is configured to recognize target objects in the at least two images, determine the position of the same target object in each image, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image.

[0038] The capture module is configured to control the execution terminal to move toward the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition, and then control the execution terminal to capture the target object to be captured.

[0039] In a third aspect of the invention, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the target object capture method described above.

[0040] In a fourth aspect of the invention, a computer-readable storage medium is provided, the storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the target object capture method described above.

[0041] The above-described technical solution of the present invention has at least the following beneficial effects:

[0042] By acquiring at least two images, including the execution terminal, from different angles in real time, the same target object in each image is identified and determined. Based on the positional information of each target object in each image, the relative position of the target object and the execution terminal in the target space is further determined. The target object is then captured based on the relative position of the execution terminal and the target object. By calculating the relative position of the execution terminal and the target object to be captured in real time, the position of the target object is tracked, thereby controlling the execution terminal to gradually approach and capture the target object. Compared with existing technologies, this invention can achieve real-time tracking of moving target objects, thus effectively improving the success rate of target object capture.

[0043] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0045] Figure 1 This is a flowchart of a target object capture method provided by a preferred embodiment of the present invention;

[0046] Figure 2This is a schematic diagram of the capture device structure provided by a preferred embodiment of the present invention;

[0047] Figure 3 This is a target microbial absorption control logic diagram provided by a preferred embodiment of the present invention;

[0048] Figure 4 This is a flowchart of target object identification provided by a preferred embodiment of the present invention;

[0049] Figure 5 This is a flowchart of a method for determining the position of overlapping target objects according to a preferred embodiment of the present invention;

[0050] Figure 6 This is a positional relationship diagram provided by a preferred embodiment of the present invention, which determines the existence of overlapping phenomena based on the region with the largest included angle.

[0051] Figure 7 This is a positional relationship diagram based on the slope used to determine the existence of overlapping phenomena, provided by a preferred embodiment of the present invention.

[0052] Figure 8 This is a schematic diagram of a method for determining the maximum included angle region provided by a preferred embodiment of the present invention;

[0053] Figure 9a This is a schematic diagram of the shooting angle provided by a preferred embodiment of the present invention;

[0054] Figure 9b These are images captured by the imaging device 1 provided in a preferred embodiment of the present invention;

[0055] Figure 9c These are images captured by the imaging device 2 provided in a preferred embodiment of the present invention;

[0056] Figure 10 This is a schematic block diagram of a target object capturing device provided by a preferred embodiment of the present invention;

[0057] Figure 11 This is a schematic block diagram of a terminal device provided by a preferred embodiment of the present invention.

[0058] Explanation of reference numerals in the attached figures

[0059] 10-Terminal device, 100-Processor, 101-Memory, 102-Computer program, 21-First camera, 22-Second camera, 31-First guide rail, 32-Second guide rail, 33-Third guide rail, 34-Robotic arm, 35-Actuation end effector. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustrating and explaining the embodiments of the present invention and are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0061] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0062] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0063] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0064] For ease of understanding, before describing the embodiments of the present invention, the following explanations are provided: the target object includes, but is not limited to, bacteria, viruses or other microorganisms; the target object is located in the target environment, which can be a liquid environment, and the target space can refer to the three-dimensional coordinate system space where the liquid environment is located.

[0065] like Figure 1 As shown, in a first aspect of this embodiment, a target object capture method is provided, comprising:

[0066] S100. Real-time acquisition of at least two images, including the execution endpoint, with the at least two images captured from different angles. Each image includes the execution endpoint and a target object that may exist in the target environment.

[0067] S200. Identify target objects in at least two images, determine the position of the same target object in each image, and determine the target object to be captured in the target objects based on the execution end and the position information of each target object in each image.

[0068] S300: Control the execution end to move towards the target object to be captured until the relative position between the execution end and the target object to be captured meets the capture condition, and control the execution end to capture the target object to be captured.

[0069] Thus, this embodiment acquires at least two images, including the execution terminal, from different angles in real time, identifies and determines the same target object in each image, and further determines the relative position of the target object and the execution terminal in the target space based on the position information of each target object in each image. Based on the relative position of the execution terminal and the target object, it determines the target object to be captured. By calculating the relative position of the execution terminal and the target object to be captured in real time, the position tracking of the target object is achieved, thereby controlling the execution terminal to gradually approach the target object and capture the target object. This realizes real-time tracking of moving target objects, thereby effectively improving the success rate of target object capture.

[0070] The method of this embodiment can be applied to a target object capture system. The target object capture system of this embodiment includes at least two image acquisition devices, a capture device including an execution end, and a controller. The image acquisition device is a camera, used to acquire images of the execution end, including the capture device, in the target space from different angles. The controller is used to identify target objects in real time based on the images acquired by the camera, calculate the relative position of the execution end and each target object to determine the target object to be captured, and after determining the target object to be captured, control the execution end to move towards and capture the target object.

[0071] It should be noted that the target object can be either a discrete target object or a group of target objects. The terms "discrete target object" and "group of target objects" are two relative concepts. A discrete target object refers to target objects that are separated from each other, while a group of target objects refers to multiple target objects that are aggregated together.

[0072] like Figure 2As shown, in this embodiment, the target object capturing device includes: a first guide rail 31, a second guide rail 32, a third guide rail 33, a robotic arm 34, and an end effector 35. The second guide rail 32 is set at a certain angle to the first guide rail 31 and is slidably connected to the first guide rail 31; the third guide rail 33 is set at a certain angle to the second guide rail 32 and is slidably connected to the first guide rail 31; the end effector 35 is slidably connected to the third guide rail 33 via the robotic arm 34, and the end effector 35 is a suction tube capable of picking up the target object after receiving a capture command from the controller. To facilitate obtaining the position information of the target object in the target space, reduce the amount of computation, and improve computational efficiency, in a specific example of this embodiment, the first guide rail 31 and the second guide rail 32 are arranged parallel to the ground, and the second guide rail 32 is perpendicular to the first guide rail 31. The second guide rail 32 can move along the axial direction of the first guide rail 31. The third guide rail 33 is arranged perpendicular to the ground, and the third guide rail 33 is perpendicular to the second guide rail 32. The third guide rail 33 can move along the axial direction of the second guide rail 32. One end of the robotic arm 34 is fixedly connected to the end effector 35, and the other end is slidably connected to the third guide rail 33. The robotic arm 34 can move along the axial direction of the third guide rail 33. The robotic arm 34 can be a multi-link structure, so that the position of the end effector 35 can be adjusted according to the actual situation.

[0073] In step S100, to simplify the target object's position calculation process, at least two image acquisition devices include a first camera and a second camera, with the shooting angles of the first camera and the second camera perpendicular to each other. For example, the first camera is positioned directly above the target environment to acquire images of the target environment on a horizontal plane; the second camera is positioned to the side of the target environment to acquire images of the target environment on a vertical plane.

[0074] like Figure 3As shown, in use, the target object is first placed in a container with the target environment, for example, microorganisms are placed in a container filled with liquid. The capture device is placed on a parallel surface of the container. The initial position of the end effector in the container is determined by adjusting the robotic arm. The first camera is positioned directly above the container to capture images of the liquid environment inside the container on a horizontal plane, and the second camera is positioned on the left side of the container to capture images of the liquid environment inside the container on a vertical plane. When capturing microorganisms, the controller first controls the first and second cameras to capture images in real time at a preset shooting frequency and a predetermined angle, and then identifies the end effector in the captured images. Understandably, if the end effector is not identified in any of the captured images, the controller moves the end effector along the X-axis or Y-axis of the world coordinate system by controlling the movement of the second or third guide rail, and moves the end effector along the Z-axis by controlling the movement of the robotic arm, thereby adjusting the position of the end effector until it is identified in all the captured images. Understandably, in order to facilitate the calculation of the relative position between the execution end and the target object, the positions of the first and second cameras can be adjusted so that the execution end always remains within the field of view of the first and second cameras.

[0075] like Figure 4 As shown, in step S200, identifying the target object in at least two images includes:

[0076] S201. Preprocess each image to obtain non-valid targets:

[0077] S202. Input each image into the trained recognition model for reasoning to obtain the initial target object being recognized;

[0078] S203. Remove invalid targets from the initial target objects and use them as the final target objects.

[0079] In step S201, the image is preprocessed, including: removing invalid regions of the image; performing median filtering, color channel format conversion, and image sharpening on the obtained image to make the target object stand out more and be easier to identify; then performing edge detection on the current image; determining each connected component in the current image based on the edge detection results; calculating the contour size of each connected component; determining objects corresponding to connected components with contour sizes smaller than a first threshold or larger than a second threshold as invalid targets; and obtaining the edge coordinates of the invalid targets, where the first threshold is smaller than the second threshold. Through step S201, targets that are obviously too large or too small in the image can be identified. By reasonably setting the first and second thresholds, objects that are obviously not target objects can be identified, which are called invalid targets.

[0080] In step S202, the recognition model can be constructed based on machine learning algorithms, such as backpropagation (BP) neural networks or convolutional neural networks. When constructing the recognition model, a large number of image samples of the target object, such as a specific single discrete microorganism, are first collected as training samples. These training samples are then used as input to the neural network model for training. For example, the training samples are divided into training and test sets. The neural network model is trained using the training set and validated using the test set. The parameters of the neural network model are adjusted based on the model's prediction results. The model is continuously trained using samples until it converges, resulting in the trained recognition model. It is understood that by using different training samples to train the neural network model, a recognition model capable of identifying any target microorganism or target microbial community can be obtained. The training process of the neural network model is existing technology and will not be elaborated here.

[0081] Taking the trained recognition model for identifying target microorganisms (i.e., target objects) as an example, after preprocessing the image captured by the camera, the trained recognition model is input, and the trained recognition model is used to identify target microorganisms in the input image. The specific recognition result is: generating a detection box containing the target microorganism and the confidence level of the target microorganism within the detection box, where the confidence level is the probability of identifying the target microorganism as belonging to the target microorganism. After the recognition result, there are also post-processing steps for the recognition result: (1) Calculate the overlap of each detection box. If there is a detection box with an overlap greater than the overlap threshold, merge them into one detection box. For example, the overlap threshold can be set to 90%; (2) Compare all confidence levels with the preset confidence threshold, remove target microorganisms with confidence levels lower than the confidence threshold, and retain target microorganisms with confidence levels higher than the confidence threshold; (3) Calculate the outline size of each retained detection box and remove the target microorganisms corresponding to detection boxes that are too large or too small. In this way, the recognition accuracy of target microorganisms can be further improved through the post-processing process.

[0082] In step S203, the target microorganism identified in step S202 is used as the initial target object. The invalid targets obtained in step S201 are matched with the initial target object to determine whether there are invalid targets in the initial target object. If there are invalid targets in the initial target object, they are removed. It is understood that matching the initial target object with invalid targets can be based on the overlap of their contours, or it can be based on their center coordinates; this is not limited here. After removing the invalid targets from the initial target object, the remaining valid targets are the target microorganisms. The coordinates of the corresponding detection boxes are output, and the coordinates of the center point of the detection boxes are used as the coordinates of the target microorganisms. Since identifying the target microorganisms through step S202 may lead to over-detection, removing non-target microorganisms that differ significantly from the target microorganisms identified in step S201, based on the initial target object identified in step S202, effectively improves the accuracy of target microorganism identification.

[0083] Understandably, step S200 applies not only to identifying discrete target microorganisms but also to identifying target microbial communities. The identification process for target microbial communities is the same as that for identifying discrete target microorganisms, and will not be repeated here.

[0084] In this embodiment, the steps for determining the same target object in each image are as follows: First, obtain the two-dimensional coordinates of the target object in each image. For example, the coordinates of the center point of the detection box can be used as the two-dimensional coordinates of the identified target object. Second, perform coordinate matching on the two-dimensional coordinates of the target objects in different images. The image captured by the camera located directly above the container is the first image, and the image captured by the camera located to the left of the container is the second image. First, establish image coordinate systems for the first and second images. The origin of the image coordinate system can be the projection of the camera's optical axis onto the image, or it can be the lower left corner of the image; this is not limited here. In the first image, the two-dimensional coordinates of the target object are (X1, Y1), and in the second image, the two-dimensional coordinates of the target object are (X2, Y2). Based on the coordinate system settings of the first and second images, coordinate transformation can be performed to determine that Y2 in the second image corresponds to Y1 in the first image. Therefore, as long as Y1 and Y2 match, it can be determined that the target object in the first image and the target object in the second image are the same target object. At the same time, by using the two-dimensional coordinates of the same target object in each image, the three-dimensional coordinates of the target object in the target space can be obtained. Similarly, the three-dimensional coordinates of the execution end can be obtained.

[0085] In a specific example of this embodiment, determining the target object to be captured among the target objects based on the position information of the execution terminal and each target object in each image includes: determining the relative position of the execution terminal and each target object in the target space based on the position information of the execution terminal and each target object in each image; and determining the target object to be captured among the target objects within a preset relative position threshold range based on the relative position of the execution terminal and each target object in the target space. Using this method, it is possible to find the target object to be captured in the vicinity of the execution terminal, rather than searching for the target object in the entire target space, thus resulting in a faster search speed.

[0086] In step S200, the target object to be captured is determined based on the position information of the execution terminal and each target object in each image, including: determining whether a target object exists based on the position information of the execution terminal and each target object in each image; if not, controlling the execution terminal to move randomly to a new position.

[0087] In this embodiment, after identifying target objects in at least two images, the system further checks whether a target object exists within a preset relative position threshold range from the execution endpoint. If no target object is found, the execution endpoint is randomly moved to a new position along the X, Y, or Z axis of the world coordinate system, and the image recognition steps are repeated until target objects are present in all target spatial regions within the preset relative position threshold range from the execution endpoint. The target spatial region can be a rectangular spatial region including the execution endpoint, such as a rectangular spatial region centered on the endpoint of the execution endpoint, or a fan-shaped or circular spatial region including the execution endpoint; no limitation is made here. It is understood that, to reduce computational load, in another specific example of this embodiment, after preprocessing the images, the target spatial region within the preset relative position threshold range from the execution endpoint can be determined first, and then target object recognition can be performed only on the target spatial region, without needing to perform target object recognition on all regions of the acquired images, thereby further improving recognition efficiency.

[0088] In this embodiment, the target object includes target microorganisms or target microbial communities, and the execution end is a straw capable of blowing air. Step S200, determining the target object to be captured based on the position information of the execution end and each target object in each image, includes: determining whether a target object exists based on the position information of the execution end and each target object in each image; if it exists and the target object includes a target object group, then controlling the execution end to separate the target object group. Specifically, if a target object exists within a target spatial region within a preset relative position threshold range from the execution end, and a target microbial community is determined to exist within the target spatial region by target object identification, the straw is controlled to blow air onto the target microbial community to separate the target microorganisms. More specifically, in practical applications, it is also possible to: if only the target microbial community exists within the target spatial region, and no discrete target microorganisms exist, control the straw to blow air onto the target microbial community to separate the target microorganisms, repeating the above steps until discrete target microorganisms exist within the target spatial region within a preset relative position threshold range from the execution end in the acquired image.

[0089] In a specific example of this embodiment, step S200, determining the target object to be captured based on the position information of the execution terminal and each target object in each image, includes: determining whether a target object exists based on the position information of the execution terminal and each target object in each image; if it exists and the target object includes discrete target objects, then, based on the relative position of the execution terminal and each target object in the target space and the existence of discrete target objects located in any direct movement direction of the execution terminal, determining the target object to be captured among the discrete target objects located in the direct movement direction of the execution terminal; if, based on the relative position of the execution terminal and each target object in the target space and the absence of discrete target objects located in any direct movement direction of the execution terminal, determining the discrete target object with the smallest relative distance to the execution terminal as the target object to be captured. The term "direct movement direction" refers to the direction in which the execution end can directly reach the discrete target object by moving along only one direction. In this embodiment, there are three direct movement directions: the first direct movement direction is the direction in which the execution end can directly reach the discrete target object by moving only along the X-axis; the second direct movement direction is the direction in which the execution end can directly reach the discrete target object by moving only along the Y-axis; and the third direct movement direction is the direction in which the execution end can directly reach the discrete target object by moving only along the Z-axis. More specifically, the three direct movement directions are the directions of the first guide rail 31, the second guide rail 32, and the third guide rail 33.

[0090] Specifically, if discrete target microorganisms are identified within a target space region within a preset relative position threshold range from the execution end through target object identification, it is further determined whether discrete target microorganisms located in any direct movement direction of the execution end exist within the target space region. When controlling the execution end to absorb target microorganisms, if discrete target microorganisms exist in any direct movement direction of the execution end, the target microorganism to be captured is preferentially identified among the discrete target microorganisms in that direct movement direction of the execution end; if multiple discrete target microorganisms exist in the same direct movement direction of the execution end, the discrete target microorganism closest to the execution end is identified as the target microorganism to be captured.

[0091] Understandably, the distance between the target microorganism and the execution terminal can be calculated and determined using the coordinates of the target microorganism and the execution terminal in at least two images. For example, using the coordinates of the center point of the detection box corresponding to the target microorganism as the two-dimensional coordinates of the target microorganism in each image, the three-dimensional coordinates of the target microorganism in the target space can be determined by performing coordinate transformation based on the two-dimensional coordinates of the target microorganism in each image. Similarly, the three-dimensional coordinates of the execution terminal in the same target space can be obtained.

[0092] If there are no discrete target microorganisms located in any direct movement direction of the execution end within the target area, the Euclidean distance between the target microorganism and the execution end is calculated based on the three-dimensional coordinates of the target microorganism and the execution end, and the target microorganism with the closest Euclidean distance to the execution end is selected as the target microorganism to be captured.

[0093] In step S300, after identifying the target microorganism to be captured, the execution terminal is controlled to move towards the target microorganism by a preset step size, continuously acquiring at least two images including the execution terminal and the target microorganism. The three-dimensional coordinates of the execution terminal and the target microorganism in the target space are determined based on their two-dimensional coordinates in the at least two images. The relative position between the execution terminal and the target microorganism is calculated based on their three-dimensional coordinates. It is then determined whether the relative position between the execution terminal and the target microorganism meets the capture condition. If not, the execution terminal continues to move towards the target microorganism by a preset step size, acquiring at least two images including both the execution terminal and the target microorganism, and calculating their relative position. This process is repeated until the relative position between the execution terminal and the target microorganism meets the capture condition, at which point the execution terminal is controlled to absorb the target microorganism. The capture condition can be that the Euclidean distance between the execution terminal and the target microorganism is less than an absorption threshold.

[0094] like Figure 5As shown, in this embodiment, identifying a target object in at least two images and determining the position of the same target object in each image further includes:

[0095] S204. Using one image as the base image, based on the shooting angle relationship of at least two images, determine whether multiple target objects in the base image will overlap in another image. If M target objects overlap in another image, determine the position information of each of the M target objects in that other image based on the position information of the overlapping target objects, where M is an integer greater than or equal to 2. The imaging of M target objects in another image is called overlapping target objects. The overlap phenomenon includes partial overlap or complete overlap. Partial overlap refers to a relationship where at least two target objects partially occlude or are partially occluded in the shooting direction when shooting another image; complete overlap refers to a relationship where at least two target objects completely occlude or are completely occluded in the shooting direction when shooting another image.

[0096] S205. Determine the position information of each of the M target objects on at least two images as the position information of the overlapping target objects.

[0097] The method for determining the location of overlapping target objects provided by this invention can be applied to a variety of scenarios. The target object and its environment refer to different things depending on the application scenario. For example, in a microbial experimental environment, the target object includes, but is not limited to, bacteria, viruses and microorganisms, and the environment includes petri dishes, containers and fluid conduits.

[0098] The method for identifying target objects in each image can be implemented using existing technologies such as image recognition models, and the specific implementation method is not improved in this invention. Due to different shooting angles, multiple target objects that overlap in one image will not overlap in another image. Therefore, in this embodiment, based on the shooting angles of at least two images and the target objects identified in at least two images, it is possible to determine that M target objects appearing in the base image overlap in another image. These M target objects are called overlapping target objects. Therefore, the M target objects overlap in another image to form a target object N. The coordinates of target object N in the other image can be used as the coordinates of the M target objects in the other image. Thus, the coordinates of the M target objects in each image are known, and the coordinates of the M overlapping target objects in each image are determined, thereby further determining the positions of the M overlapping target objects in the target space.

[0099] Therefore, in the process of capturing target objects, the overlapping target object location determination method provided by this invention can also identify overlapping target microorganisms based on the fact that multiple target objects are displayed on one image but only one target object is partially or even completely overlapped on another image. Thus, in the process of automatically capturing target objects, based on the specific location of the overlapping target object, it is possible to determine a large number of target objects in the vicinity of the execution end (e.g., a straw). In this way, it is possible to determine the target object closest to the execution end among the many target objects as the target object to be captured. The execution end can then quickly and accurately capture the target object to be captured, thereby improving the capture efficiency and accuracy of the target object.

[0100] In this embodiment of the invention, step S204 specifically includes: determining the maximum angle region formed by connecting the boundary points of each target object in the base image to the coordinate transformation point of the shooting position of another image on the base image; determining whether the maximum angle regions of multiple target objects overlap; if the maximum angle regions of M target objects overlap, then it is determined that the M target objects overlap. The coordinate transformation point refers to the position of the shooting position coordinates of another image transformed to the base image; the maximum angle region can be understood as the shooting area of ​​the shooting device of another image transformed to the shooting area on the base image.

[0101] In step S204, determining whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship of at least two images is determined by the maximum angle region between the coordinate transformation points of the target object and the shooting position of the other image in the base image. This shooting position can be, but is not limited to, the optical center position of the camera.

[0102] To simplify the description, this embodiment uses two images as an example, captured by two separate cameras at perpendicular angles. One camera captures the image vertically, while the other captures it horizontally. In this case, the coordinate transformation point is essentially the coordinate point of the other image's location projected onto the base image.

[0103] Specifically, Figure 6 This schematically illustrates a positional relationship diagram based on the region of maximum included angle, according to an embodiment of the present invention. For example... Figure 6As shown, the maximum angle region formed by the line connecting the boundary points of the target objects in the base image and the coordinate transformation is determined, i.e., the multiple shaded areas in the figure; the boundary of this maximum angle region is related to the boundary of the target object. It is determined whether the maximum angle regions of multiple target objects overlap. If the maximum angle regions of M target objects overlap, then the M target objects are determined to overlap. For example, if the maximum angle regions (shaded areas) of two target objects located slightly above in the figure overlap, then these two target objects are determined to overlap. In the specific algorithm judgment, slope and angle can be used to judge the overlap phenomenon, which will be explained in subsequent embodiments.

[0104] I. Using slope to determine overlap. In this embodiment of the invention, the maximum and minimum slopes of the maximum included angle region of the target object are determined; the slope range of the target object is calculated based on the maximum and minimum slopes; it is determined whether the slope ranges of multiple target objects overlap. If the slope ranges of M target objects overlap, then it is determined that the M target objects overlap. Specifically, Figure 7 This schematically illustrates a positional relationship diagram based on slope, according to an embodiment of the present invention. In this embodiment, Figure 7 The coordinate system is constructed on the plane of the base image, with the origin being the coordinate transformation point of the shooting position of another image on the base image. Taking a target object as an example, the linear function relationship between the center point of the target object and the origin can be expressed as: y = kx (not shown in the figure). The boundary points of the target object and the origin also form a similar linear function relationship to y = kx, the only difference being the value of k. The possible values ​​of k are within a certain range, such as [k1, k2] and [k3, k4]. Thus, the maximum included angle region of each target object can be mapped to a slope range.

[0105] II. Judging Overlap by Angle. In this embodiment of the invention, judging whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum and minimum angles of the maximum included angle regions of the target objects; calculating the angle range of the target objects based on the maximum and minimum angles; judging whether the angle ranges of multiple target objects overlap. If the angle ranges of M target objects overlap, then it is determined that the M target objects overlap. Specifically, this method actually maps the boundary line of the maximum included angle region to the angle with a reference line, thereby obtaining the maximum and minimum angles of the maximum included angle region; calculating the angle range of each target object based on the maximum and minimum angles of each target object. The existence of overlap is determined by whether the angle ranges overlap.

[0106] Figure 8A schematic diagram illustrating a method for determining the maximum included angle region according to an embodiment of the present invention is shown. Figure 8 As shown in the embodiment of the present invention, a method for determining the maximum included angle region is provided. This method for determining the maximum included angle region formed by connecting the boundary point of the target object in the base image with the coordinate transformation point includes: determining the center point of the target object in the base image; determining a first line connecting the coordinate transformation point and the center point in the base image; determining a second line perpendicular to the first line in the base image at the center point; determining the intersection point where the second line coincides with the boundary point of the target object in the base image, which in this embodiment are intersection point 1 and intersection point 2 in the figure; and connecting the intersection point with the coordinate transformation point to form the maximum included angle region. In practical applications, since the target object may be a microorganism with irregular edges, the number of intersection points may be greater than two. Determining the maximum included angle region based on multiple intersection points can maximize the identification of overlapping target microorganisms, thereby further improving the efficiency of capturing target microorganisms.

[0107] In this embodiment, the above content discloses a technical means of determining whether two or more target objects overlap by using the area of ​​the largest included angle.

[0108] The following details the process of combining and judging multiple target objects identified on a base image. Specifically, determining whether multiple target objects in the base image will overlap in another image includes: arbitrarily selecting one target object in the base image as the current target object; comparing the other target objects in the base image with the current target object to determine if they overlap; and from the remaining target objects in the base image, arbitrarily selecting another target object in the base image as the current target object, and comparing the other remaining target objects with the current target object to determine if they overlap. The remaining target objects refer to the other target objects in the base image that have been excluded from being selected as the current target object or identified as overlapping target objects.

[0109] Here's an example: If four target objects (1-4) are identified in the base image, then target object 1 is arbitrarily selected as the current target object. Target objects 2-4 are compared with target object 1, and it is determined whether they overlap. If target object 2 overlaps with target object 1, then the remaining target objects include target objects 3 and 4. Target object 3 is then arbitrarily selected as the next current target object, and it is determined whether the remaining target object 4 overlaps with target object 3. The process ends here. In another example, target object 1 is arbitrarily selected as the current target object. Target objects 2-4 are compared with target object 1, and it is determined whether they overlap. If both target objects 2 and 3 overlap with target object 1, then target objects 1-3 are overlapping target objects, and the remaining target objects only include target object 4. The process ends here.

[0110] Preferably, determining whether multiple target objects in the base image overlap in another image includes: determining whether multiple target objects in a preset region of the base image overlap in another image. The preset region can be defined as the region where the execution endpoint is located. This way, only overlapping target objects around the region near the execution endpoint need to be determined, which reduces the determination time compared to determining whether all target objects in the base image overlap, thus improving capture efficiency. Furthermore, since the position of a target object changes over time within its environment, reducing the determination time means reducing the distance the target object moves, thereby improving the accuracy and precision of the capture.

[0111] The following is combined Figures 9a-9c The invention will be explained in detail. Figure 9a A schematic diagram illustrating the shooting angle according to an embodiment of the present invention is shown. Figure 9a As shown, at least two images are captured from a set of mutually perpendicular angles. The reason for choosing mutually perpendicular angles is that the coordinate relationship between the images is more compatible with a Cartesian coordinate system, reducing the computational complexity of coordinate transformations when calculating the target object's spatial position. Of course, in practical applications, there can be three or more shooting angles. The choice can be made between increasing the accuracy of the target object's position (higher cost) with more shooting angles and decreasing the cost with fewer shooting angles, depending on the specific needs. Here, the mutually perpendicular shooting angles are preferably horizontal and vertical angles, as this simplifies the calculation of the target object's position information.

[0112] like Figure 9a As shown, if the shooting angle of shooting device 1 is along the horizontal direction, the obtained image is in the XY coordinate system. If the shooting angle of shooting device 2 is along the vertical direction, the obtained image is in the YZ coordinate system. Figure 9b The image captured by the imaging device 1 is shown schematically. A target object D exists in the image captured by the imaging device 1, and the coordinates of the target object D are (x1, y1). Figure 9c The image captured by imaging device 2 is illustrated schematically. The image captured by imaging device 2 contains independent images of target objects A, B, and C. The coordinates of target object A are (y1, z1), the coordinates of target object B are (y1, z2), and the coordinates of target object C are (y1, z3). Since the y-axis coordinates of target objects A, B, C, and D are all y1 based on the angular relationship between imaging devices 1 and 2, the coordinate system settings, and the imaging parameters, it is determined that target objects with the same y1 coordinate overlap in another image. Target objects A, B, and C are referred to as overlapping target objects, and the target object formed by the overlap in another image is target object D. Therefore, the coordinates of target object A are saved as (x1, y1, z1), the coordinates of target object B are saved as (x1, y1, z2), and the coordinates of target object C are saved as (x1, y1, z3), thus determining the positional information of the overlapping target objects.

[0113] It should be noted that the above is just a simple example. In practical applications, since the imaging area of ​​the imaging device is a conical space formed by the emitted rays, to determine whether the target objects in the two images overlap, it is necessary to first transform the coordinates of the target objects in the images captured by the two imaging devices according to the trigonometric theorem, and then determine whether there is an overlap based on the transformed coordinates.

[0114] The above describes the action of determining overlapping target objects when there is overlap. Of course, in practical applications, if there is no overlap, the coordinates of the target object in each image are directly saved.

[0115] The following describes the implementation method of the target object capture method, taking microorganisms in solution as the target object and a pipette of a robotic arm as the end effector:

[0116] Step 1: Simultaneously take pictures with imaging device 1 and imaging device 2 to obtain images of the microorganisms in the entire solution and the pipette at the end of the robotic arm. The images taken here require that the resolution of the specific microorganism in the image is not less than 20 pixels and there is no obvious distortion.

[0117] Step 2: Obtain the manually defined size range for specific microorganisms. Based on the imaging adjustments made to the specific microorganism by the imaging device in the previous step, the size range of that specific microorganism in the image is statistically analyzed. Manual setting is relatively simple and does not require a learning-based statistical approach. Since the positions of the imaging device and the container containing the microorganism are fixed, the size can be set relatively large to ensure the microorganism's pixels are clear.

[0118] Step 3: Extraction of Edge Coordinates of Invalid Targets. Image preprocessing is performed on the images from both imaging devices 1 and 2 to obtain the coordinates of invalid targets, facilitating subsequent removal. The image preprocessing steps are: removing invalid regions → median filtering → image sharpening → edge detection → calculating the size of connected components in each region → marking connected components that do not conform to the specific microbial size range specified in Step 4 as invalid target regions → obtaining the edge coordinates (x, y) of invalid targets; and using the remaining connected components as the edge coordinates (X, Y) of valid microorganisms.

[0119] Step 4: Perform image preprocessing on the images from imaging device 1 and imaging device 2 respectively to make the specific microorganisms stand out more in the images for easier identification. This results in two images where the specific microorganisms are clearly visible. The image preprocessing steps for this part are: removing invalid regions → median filtering → color channel format conversion → image sharpening.

[0120] Step 5: Calculate the position of the straw end in the image. Since the size of the robotic arm end effector and the angle of the robotic arm are fixed and the robotic arm is obvious, it is easy to use template matching to calculate the position of the robotic arm end effector in the two images acquired by imaging device 1 and imaging device 2 respectively.

[0121] Step Six: Use the trained offline model to identify the two images obtained in the previous step, obtaining the identification results of specific microorganisms distributed on the two images. The trained offline model can be obtained as follows: collect a large number of image samples of specific individual microorganisms and microbial communities; train the model on the collected image samples using a deep learning algorithm for object detection until the model converges, obtaining a good model that can identify specific microorganisms from multiple microorganisms coexisting in the container and locate the microorganism's position on the image. This process is an offline learning process, mainly to obtain a reliable algorithm model for microbial target identification and localization.

[0122] The steps in this section include: normalizing the image based on the previous step → extracting features using an offline model and obtaining the recognition result (the coordinates of the rectangle containing the microorganism and the probability value that the model considers the recognition result to be a specific microorganism) → post-processing the result to obtain the final result. The main post-processing methods are: 1) removing recognition results with low probability values; 2) calculating the size of the rectangle containing the microorganism and removing recognition results that do not meet the aforementioned constraints.

[0123] Step 7: Confirmation of final specific microorganism identification results. Combining steps 3 and 6, invalid targets are eliminated, and the remaining targets are taken as the final specific microorganism region. At the same time, the coordinates of the corresponding rectangle and the coordinates of the center point are obtained.

[0124] Step 8: Calculate the relative position of the microorganism and the robotic arm end effector in 3D space, and control the robotic arm end effector to move toward the target microorganism based on the calculation results. When the relative position between the robotic arm end effector and the target microorganism meets the capture conditions, control the robotic arm end effector to capture the target microorganism.

[0125] Through the above implementation methods, it is possible to quickly and accurately operate on specific microorganisms, whether they are completely or partially obscured.

[0126] like Figure 10 As shown, in a second aspect of the present invention, a target object capturing device is provided, comprising:

[0127] The image acquisition module is configured to acquire at least two images at the end of the execution process in real time, wherein the at least two images are captured from different angles.

[0128] The recognition module is configured to recognize target objects in at least two images, determine the position of the same target object in each image, and determine the target object to be captured in the target objects based on the execution end and the position information of each target object in each image.

[0129] The capture module is configured to control the execution end to move towards the target object to be captured until the relative position between the execution end and the target object meets the capture condition, and then control the execution end to capture the target object.

[0130] Optionally, the recognition module is configured as follows:

[0131] The system determines whether a target object exists based on the position information of the execution terminal and each target object in each image; if not, it controls the execution terminal to move randomly to a new position.

[0132] Optionally, the recognition module is configured as follows:

[0133] The presence of a target object is determined based on the position information of the execution terminal and each target object in each image; if it exists and the target object only includes a group of target objects, the execution terminal is controlled to separate the group of target objects.

[0134] Optionally, the recognition module is configured as follows:

[0135] The presence of a target object is determined based on the position information of the execution terminal and each target object in each image. If a target object exists and includes discrete target objects, then if the relative position of the execution terminal and each target object in the target space determines that there is a discrete target object located in the first direct movement direction or the second direct movement direction of the execution terminal, the discrete target object located in the first direct movement direction or the second direct movement direction of the execution terminal is determined as the target object to be captured. If the relative position of the execution terminal and each target object in the target space determines that there is no discrete target object located in the first direct movement direction or the second direct movement direction of the execution terminal, the discrete target object with the smallest distance from the execution terminal is determined as the target object to be captured.

[0136] Optionally, the recognition module is configured as follows:

[0137] Each image is preprocessed to obtain invalid targets; each image is then input into a trained recognition model for inference to obtain the initial target object; invalid targets are removed from the initial target object to obtain the final target object.

[0138] Optionally, the recognition module is configured as follows:

[0139] The relative positions of the execution terminal and each target object in the target space are determined based on the position information of the execution terminal and each target object in each image; based on the relative positions of the execution terminal and each target object in the target space, the target objects to be captured are determined among the target objects within the preset relative position threshold range.

[0140] Optionally, the shooting angles of at least two images include at least one set of mutually perpendicular shooting angles.

[0141] Optionally, the target object is a microorganism, and the execution end is a straw capable of blowing air; the capture module is configured to: control the execution end to blow air onto the target object group to separate the target object; and control the execution end to suck up the target object to be captured.

[0142] Optionally, the recognition module is further configured to use one image as a base image, and based on the shooting angle relationship of at least two images, determine whether multiple target objects in the base image will overlap in another image. If M target objects overlap in another image, determine the position information of each of the M target objects in that other image based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the imaging of M target objects in another image is called overlapping target objects; and determine the position information of each of the M target objects in at least two images as the position information of overlapping target objects.

[0143] Through the above technical solution, the device for determining the position of overlapping target objects provided by the present invention can quickly and accurately determine the position of overlapping target objects.

[0144] Optionally, based on the shooting angle relationship of at least two images, it is determined whether multiple target objects in the base image will overlap on another image, including: determining the maximum angle region formed by the lines connecting the boundary points of each target object in the base image and the projection points of the imaging reference points of the other image on the base image; determining whether the maximum angle regions of multiple target objects overlap; if the maximum angle regions of M target objects overlap, then it is determined that the M target objects overlap.

[0145] Optionally, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum slope and minimum slope of the maximum included angle region; calculating the slope range of each target object based on the maximum slope and minimum slope of each target object; determining whether the slope ranges of multiple target objects overlap; if the slope ranges of M target objects overlap, then determining that the M target objects overlap.

[0146] Optionally, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum and minimum angles of the maximum included angle regions; calculating the angle range of each target object based on the maximum and minimum angles of each target object; determining whether the angle ranges of multiple target objects overlap; if the angle ranges of M target objects overlap, then determining that the M target objects overlap.

[0147] Optionally, determining the region with the maximum angle formed by connecting the boundary points of multiple target objects in the base image with the imaging reference point of another image includes: determining the center point of the target objects in the base image; determining a first line connecting the imaging reference point and the center point; determining a second line on the base image perpendicular to the first line at the center point; determining two intersection points of the second line coinciding with the boundary points of the target objects in the base image; and determining the region with the maximum angle formed by connecting each of the two intersection points with the imaging reference point.

[0148] Optionally, determining whether the maximum included angle regions of multiple target objects overlap includes: selecting one target object in the base image as the current target object; comparing the maximum included angle regions of other target objects in the base image with the maximum included angle region of the current target object one by one to determine whether there is an overlap; if there is an overlap with the current target object.

[0149] Optionally, determining whether the maximum included angle regions of multiple target objects overlap further includes: reselecting another target object in the base image as the current target object; comparing the maximum included angle regions of the other remaining target objects in the base image with the maximum included angle region of the current target object one by one to determine whether there is an overlap; if there is an overlap with the current target object.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] In a third aspect of the invention, a terminal device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the target object capture method described above.

[0152] like Figure 11 The diagram shown is a schematic representation of a terminal device provided in an embodiment of the present invention. Figure 11 As shown, the terminal device 10 of this embodiment includes a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100. When the processor 100 executes the computer program 102, it implements the steps in the above method embodiments. Alternatively, when the processor 100 executes the computer program 102, it implements the functions of each module / unit in the above device embodiments.

[0153] For example, computer program 102 can be divided into one or more modules / units, one or more of which are stored in memory 101 and executed by processor 100 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 102 in terminal device 10. For example, computer program 102 can be divided into an image acquisition module, a recognition module, and a capture module.

[0154] Terminal device 10 may be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 10 may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that... Figure 11 This is merely an example of terminal device 10 and does not constitute a limitation on terminal device 10. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device may also include input / output devices, network access devices, buses, etc.

[0155] The processor 100 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0156] The memory 101 can be an internal storage unit of the terminal device 10, such as a hard disk or RAM of the terminal device 10. The memory 101 can also be an external storage device of the terminal device 10, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 10. Furthermore, the memory 101 can include both internal and external storage units of the terminal device 10. The memory 101 is used to store computer programs and other programs and data required by the terminal device 10. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0157] In a fourth aspect of the invention, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the steps of the target object capture method described above.

[0158] In summary, this invention acquires at least two images, including the execution terminal, from different angles in real time, identifies and determines the same target object in each image, further determines the relative position of the target object and the execution terminal in the target space based on the position information of each target object in each image, and determines the target object to be captured based on the relative position of the execution terminal and the target object. By calculating the relative position of the execution terminal and the target object to be captured in real time, the position tracking of the target object is achieved, thereby controlling the execution terminal to gradually approach and capture the target object. Compared with the prior art, this invention can achieve real-time tracking of moving target objects, thereby effectively improving the success rate of target object capture.

[0159] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0160] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0161] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0162] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the embodiments of the present invention, and should also be regarded as the content disclosed by the embodiments of the present invention.

Claims

1. A method for capturing a target object, characterized in that, include: Real-time acquisition of at least two images, including those at the execution end, wherein the at least two images are captured from different angles and directions; Identify target objects in at least two images, determine the position of the same target object in each image, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image; The execution terminal is controlled to move towards the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition, and then the execution terminal is controlled to capture the target object to be captured. The step of identifying target objects in at least two images and determining the position of the same target object in each image includes: Using one image as a base image, determine the region of the largest included angle formed by the line connecting the boundary point of the target object in the base image and the coordinate transformation point of the shooting position of another image on the base image; Determine whether the maximum included angle regions of multiple target objects overlap. If the maximum included angle regions of M target objects overlap, then it is determined that the M target objects overlap. Determine the position information of the overlapping target objects. The imaging of the M target objects in the other image is called the overlapping target objects.

2. The target object capture method according to claim 1, characterized in that, The step of determining the target object to be captured based on the execution terminal and the position information of each target object in each image includes: Based on the position information of the execution terminal and each target object in each image, determine whether there is a target object within a preset relative position threshold range from the execution terminal; If not, the control execution end will randomly move to a new position.

3. The target object capture method according to claim 1, characterized in that, The step of determining the target object to be captured based on the execution terminal and the position information of each target object in each image includes: Based on the position information of the execution terminal and each target object in each image, determine whether there is a target object within a preset relative position threshold range from the execution terminal; If the target object exists and includes a group of target objects, then control the execution terminal to separate the group of target objects.

4. The target object capture method according to claim 1, characterized in that, The step of determining the target object to be captured based on the execution terminal and the position information of each target object in each image includes: Based on the position information of the execution terminal and each target object in each image, determine whether there is a target object within a preset relative position threshold range from the execution terminal; If the target object exists and includes discrete target objects, then based on the relative position of the execution terminal and each target object in the target space, and if there is a discrete target object located in any direct movement direction of the execution terminal, the target object to be captured is determined among the discrete target objects located in that direct movement direction of the execution terminal. Based on the relative positions of the execution endpoint and each target object in the target space, and provided that there are no discrete target objects located in any direct movement direction of the execution endpoint, the discrete target object with the smallest relative distance to the execution endpoint is determined as the target object to be captured.

5. The target object capture method according to claim 2, characterized in that, The step of identifying target objects in at least two images and determining the position of the same target object in each image further includes: Based on the position information of the overlapping target objects in the other image, determine the position information of each of the M target objects in the other image, where M is an integer greater than or equal to 2; The positional information of each of the M target objects on at least two images is determined as the positional information of the overlapping target objects.

6. The target object capture method according to any one of claims 1-5, characterized in that, The identification of the target object in the at least two images includes: Each of the images is preprocessed to obtain non-valid targets; Each image is input into a trained recognition model for inference to obtain the initial target object; Remove the invalid targets from the initial target objects to obtain the final target objects.

7. The target object capture method according to any one of claims 1-5, characterized in that, The step of determining the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image includes: The relative positions of the execution terminal and each target object in the target space are determined based on the position information of the execution terminal and each target object in each image; Based on the relative position of the execution terminal and each target object in the target space, the target object to be captured is determined among the target objects within the preset relative position threshold range.

8. The target object capture method according to claim 1, characterized in that, The shooting angles of the at least two images include at least one set of mutually perpendicular shooting angles.

9. The target object capture method according to claim 3, characterized in that, The target object is microorganisms, and the execution end is a straw capable of blowing air; The step of controlling the execution terminal to separate the target object group includes: controlling the execution terminal to blow air into the target object group to separate the target objects; The control of the execution terminal to capture the target object to be captured includes: controlling the execution terminal to absorb the target object to be captured.

10. A target object capturing device, employing the target object capturing method according to any one of claims 1-9, characterized in that, The device includes: The image acquisition module is configured to acquire at least two images, including the execution end, in real time, wherein the at least two images are captured from different angles and directions. The recognition module is configured to recognize target objects in the at least two images, determine the position of the same target object in each image, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image. The capture module is configured to control the execution terminal to move toward the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition, and then control the execution terminal to capture the target object to be captured.

11. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the target object capture method according to any one of claims 1-9.

12. A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the target object capture method according to any one of claims 1 to 9.

Images