Position determination method, device and equipment for overlapping target objects

Abstract

This invention relates to the field of image recognition technology, and provides a method, apparatus, and device for determining the location of overlapping target objects. One method for determining the location of overlapping target objects includes: acquiring at least two images including the environment in which the target object is located; identifying the target object in each of the images; using one image as a base image, determining whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship of the at least two images; if M target objects overlap in another image, determining the location information of each of the M target objects in that other image based on the location information of the overlapping target objects in that other image; and determining the location information of each of the M target objects in the at least two images as the location information of the overlapping target objects. This invention improves the speed and accuracy of target object recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a method for determining the position of overlapping target objects, a device for determining the position of overlapping target objects, a equipment for determining the position of overlapping target objects, and a computer-readable storage medium. Background Technology

[0002] Microorganisms typically include, but are not limited to, bacteria, viruses, and other tiny organisms. In flowing liquids, there are usually multiple organisms such as bacteria, viruses, and tiny organisms, and their numbers are often quite large. Currently, the capture of microorganisms usually involves manually picking specific microorganisms using a microscope, but this manual method is inefficient. Furthermore, due to the fluidity of liquids and the random movement of microorganisms, the success rate of capturing target microorganisms using existing visual methods is low, making it difficult to achieve satisfactory results. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, and device for determining the position of overlapping target objects, so as to solve some of the above-mentioned problems.

[0004] To achieve the above objective, a first aspect of the present invention provides a method for determining the position of overlapping target objects, comprising: acquiring at least two images including the environment in which the target objects are located, wherein the at least two images are taken from different angles; identifying the target objects in each of the images; using one image as a base image, determining, based on the relationship between the shooting angles of the at least two images, whether multiple target objects in the base image will overlap in another image; if M target objects overlap in another image, determining the position information of each of the M target objects in the other image based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of the M target objects in the other image is called overlapping target objects; and determining the position information of each of the M target objects in the at least two images as the position information of the overlapping target objects.

[0005] Preferably, determining whether multiple target objects in the base image will overlap on another image based on the shooting angle relationship of the at least two images includes: determining the maximum angle region formed by the line connecting the boundary point of the target object in the base image and the coordinate transformation point of the shooting position of the other image on the base image; determining whether the maximum angle regions of multiple target objects overlap; if the maximum angle regions of M target objects overlap, then it is determined that the M target objects overlap.

[0006] Preferably, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum slope and minimum slope of the maximum included angle region of the benchmark object; determining the slope range of the target object based on the maximum slope and minimum slope of the target object; determining whether the slope ranges of multiple target objects overlap; if the slope ranges of M target objects overlap, then determining that the M target objects overlap.

[0007] Preferably, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum angle and minimum angle of the maximum included angle region of the target objects; calculating the angle range of the target objects based on the maximum angle and minimum angle of the target objects; determining whether the angle ranges of multiple target objects overlap; if the angle ranges of M target objects overlap, then determining that the M target objects overlap.

[0008] Preferably, determining the region with the maximum included angle formed by the line connecting the boundary point of the target object in the base image and the coordinate transformation point includes: determining the center point of the target object in the base image; determining a first line connecting the coordinate transformation point and the center point in the base image; determining a second line perpendicular to the first line in the base image at the center point; determining the intersection point of the second line and the boundary point of the target object; and connecting the intersection point with the coordinate transformation point to form the region with the maximum included angle.

[0009] Preferably, determining whether multiple target objects in the base image overlap in another image includes: arbitrarily selecting one target object in the base image as the current target object; comparing the other target objects in the base image with the current target object to determine whether they overlap; arbitrarily selecting another target object in the base image as the current target object from the remaining target objects in the base image, comparing the other target objects in the remaining target objects with the current target object to determine whether they overlap; the remaining target objects refer to the other target objects in the base image excluding those already selected as the current target object and those determined to be overlapping target objects.

[0010] Preferably, determining whether multiple target objects in the base image will overlap in another image includes: determining whether multiple target objects in a preset region of the base image will overlap in another image.

[0011] Preferably, the shooting angles of the at least two images include at least one set of mutually perpendicular shooting angles.

[0012] In a second aspect of the present invention, a target object capture method is also provided, comprising the following steps: acquiring at least two images including an execution terminal in real time, wherein the at least two images are captured from different angles; identifying a target object in the at least two images; determining the position of the same target object in each image according to the overlapping target object position determination method according to any one of claims 1-9; determining a target object to be captured in the target object based on the execution terminal and the position information of each target object in each image; controlling the execution terminal to move toward the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition; and controlling the execution terminal to capture the target object to be captured.

[0013] In a third aspect of the invention, a device for determining the position of overlapping target objects is also provided, comprising:

[0014] An image acquisition module is used to acquire at least two images including the environment where the target object is located, wherein the at least two images are captured from different angles; a target recognition module is used to identify the target object in each of the images; an overlap determination module is used to use one image as a base image and, based on the shooting angle relationship between the at least two images, determine whether multiple target objects in the base image will overlap in another image; if M target objects overlap in another image, determine the position information of each of the M target objects in the other image based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of M target objects in the other image is called overlapping target objects; and a position determination module is used to determine the position information of each of the M target objects in the at least two images as the position information of overlapping target objects.

[0015] In a fourth aspect of the present invention, a target object capture device is also provided, comprising: an image acquisition module for acquiring at least two images, including an execution end, in real time, wherein the at least two images are captured from different angles; a target determination module for identifying target objects in the at least two images, determining the position of the same target object in each image according to the aforementioned overlapping target object position determination method, and determining a target object to be captured in the target objects based on the execution end and the position information of each target object in each image; and an end-point control module for controlling the execution end to move toward the target object to be captured until the relative position between the execution end and the target object to be captured satisfies the capture condition, and controlling the execution end to capture the target object to be captured.

[0016] In a fifth aspect of the invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the aforementioned overlapping target object position determination method and / or the steps of the aforementioned target object capture method.

[0017] In a sixth aspect of the invention, a storage medium is also provided, the storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the aforementioned method for determining the position of overlapping target objects and / or the steps of the aforementioned method for capturing target objects.

[0018] A seventh aspect of the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the aforementioned method for determining the position of an overlapping target object and / or the steps of the aforementioned method for capturing a target object.

[0019] The above technical solution has at least the following beneficial effects:

[0020] This invention utilizes a method for determining the location of overlapping target objects to pinpoint the specific locations of multiple overlapping target objects in an image. This allows for the calculation of numerous target object locations, facilitating the automatic capture of target objects. Based on these overlapping locations, a greater number of target objects can be identified in the vicinity of the execution endpoint (e.g., an eyedropper). This allows the execution endpoint to quickly and accurately capture the target object closest to the execution endpoint, thereby improving capture efficiency and accuracy.

[0021] Other features and advantages of this embodiment of the invention, including the determination of the position of overlapping target objects, will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0023] Figure 1 The diagram illustrates the steps of a method for determining the position of overlapping target objects according to an embodiment of the present invention.

[0024] Figure 2 This schematic diagram illustrates the positional relationship based on the region of maximum included angle, according to an embodiment of the present invention.

[0025] Figure 3 This schematic diagram illustrates the positional relationship based on the slope, according to an embodiment of the present invention.

[0026] Figure 4 A schematic diagram illustrating a method for determining the maximum included angle region according to an embodiment of the present invention is shown.

[0027] Figure 5a A schematic diagram illustrating the shooting angle according to an embodiment of the present invention is shown;

[0028] Figure 5b The image captured by the imaging device 1 is shown schematically.

[0029] Figure 5c The image captured by the imaging device 2 is shown schematically.

[0030] Figure 6 The schematic diagram illustrates the steps of a target object capture method according to an embodiment of the present invention;

[0031] Figure 7 This schematic diagram illustrates the structure of a device for determining the position of overlapping target objects according to an embodiment of the present invention;

[0032] Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0033] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0034] For ease of understanding, before describing the embodiments of the present invention, the following explanations are provided: The target object includes, but is not limited to, bacteria, viruses, or other microorganisms; the target object is located in a target environment, which can be a liquid environment, and the target space can refer to the three-dimensional coordinate system space where the liquid environment is located. The target object can be a discrete target object or a group of target objects; it can also include both discrete target objects and a group of target objects. Discrete target objects and a group of target objects are two relative concepts. A discrete target object is a target object that is mutually discrete, while a group of target objects is multiple target objects aggregated together.

[0035] In the following text, to improve the accuracy of capturing target objects, it is preferable to capture discrete target objects. Therefore, the target objects will be described as discrete target objects in the following text. The principle of target objects being a group of target objects or including both target object groups and discrete target objects is similar and will not be elaborated further in the following text.

[0036] Figure 1The diagram schematically illustrates the steps of a method for determining the position of overlapping target objects according to an embodiment of the present invention. Figure 1 As shown, it includes:

[0037] S01. Acquire at least two images including the environment where the target object is located, wherein the at least two images are taken from different angles;

[0038] S02. Identify the target object in each of the images;

[0039] S03. Using one image as the base image, based on the shooting angle relationship between the at least two images, determine whether multiple target objects in the base image will overlap in another image. If M target objects overlap in another image, determine the position information of each of the M target objects in that other image based on the position information of the overlapping target objects, where M is an integer greater than or equal to 2. The imaging of M target objects in another image is called overlapping target objects. The overlapping phenomenon includes partial overlap or complete overlap. Partial overlap refers to a relationship where at least two target objects partially occlude or are partially occluded in the shooting direction when shooting another image; complete overlap refers to a relationship where at least two target objects completely occlude or are completely occluded in the shooting direction when shooting another image.

[0040] S04. Determine the position information of each of the M target objects on at least two images as the position information of overlapping target objects.

[0041] The method for determining the location of overlapping target objects provided by this invention can be applied to a variety of scenarios. The target object and its environment refer to different things depending on the application scenario. For example, in a microbial experimental environment, the target object includes, but is not limited to, bacteria, viruses and microorganisms, and the environment includes petri dishes, containers and fluid conduits.

[0042] The method for identifying target objects in each image in step S02 can be implemented using existing technologies such as image recognition models, and the specific implementation method is not improved in this invention. Due to different shooting angles, multiple target objects that overlap in one image will not overlap in another image. Therefore, in this embodiment, based on the shooting angles of at least two images and the target objects identified in at least two images, it is possible to determine that M target objects appearing in the base image overlap in another image. These M target objects are called overlapping target objects. Therefore, the M target objects overlap in another image to form a target object N. The coordinates of the target object N in the other image can be used as the coordinates of the M target objects in the other image. Thus, the coordinates of the M target objects in each image are known, and the coordinates of the M overlapping target objects in each image are determined, thereby further determining the positions of the M overlapping target objects in the target space.

[0043] Therefore, in the process of capturing target objects, the overlapping target object location determination method provided by this invention can also identify overlapping target microorganisms based on the fact that multiple target objects are displayed on one image but only one target object is partially or even completely overlapped on another image. Thus, in the process of automatically capturing target objects, based on the specific location of the overlapping target object, it is possible to determine a large number of target objects in the vicinity of the execution end (e.g., a straw). In this way, it is possible to determine the target object closest to the execution end among the many target objects as the target object to be captured. The execution end can then quickly and accurately capture the target object to be captured, thereby improving the capture efficiency and accuracy of the target object.

[0044] In this embodiment of the invention, step 03 specifically includes: determining the maximum angle region formed by connecting the boundary points of each target object in the base image to the coordinate transformation point of the shooting position of another image on the base image; determining whether the maximum angle regions of multiple target objects overlap; if the maximum angle regions of M target objects overlap, then it is determined that the M target objects overlap. The coordinate transformation point refers to the position of the shooting position coordinates of another image transformed to the base image; the maximum angle region can be understood as the shooting area of ​​the shooting device of another image transformed to the shooting area on the base image.

[0045] In step S03, determining whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship of at least two images is done by considering the maximum angle region between the coordinate transformation points of the target object and the shooting position in the other image on the base image. This shooting position can be, but is not limited to, the optical center position of the camera.

[0046] To simplify the description, this embodiment uses two images as an example, captured by two separate cameras at perpendicular angles. One camera captures the image vertically, while the other captures it horizontally. In this case, the coordinate transformation point is essentially the coordinate point of the other image's location projected onto the base image.

[0047] Specifically, Figure 2 This schematically illustrates a positional relationship diagram based on the region of maximum included angle, according to an embodiment of the present invention. For example... Figure 2 As shown, the maximum angle region formed by the line connecting the boundary points of the target objects in the base image and the coordinate transformation is determined, i.e., the multiple shaded areas in the figure; the boundary of this maximum angle region is related to the boundary of the target object. It is determined whether the maximum angle regions of multiple target objects overlap. If the maximum angle regions of M target objects overlap, then the M target objects are determined to overlap. For example, if the maximum angle regions (shaded areas) of two target objects located slightly above in the figure overlap, then these two target objects are determined to overlap. In the specific algorithm judgment, slope and angle can be used to judge the overlap phenomenon, which will be explained in subsequent embodiments.

[0048] First, the slope is used to determine the overlap phenomenon. In this embodiment of the invention, the maximum and minimum slopes of the maximum included angle region of the target object are determined; the slope range of the target object is calculated based on the maximum and minimum slopes; it is determined whether the slope ranges of multiple target objects overlap. If the slope ranges of M target objects overlap, then it is determined that the M target objects overlap. Specifically, Figure 3 This schematically illustrates a positional relationship diagram based on slope, according to an embodiment of the present invention. In this embodiment, Figure 3 The coordinate system is constructed on the plane of the base image, with the origin being the coordinate transformation point of the shooting position of another image on the base image. Taking a target object as an example, the linear function relationship between the center point of the target object and the origin can be expressed as: y = kx (not shown in the figure). The boundary points of the target object and the origin also form a similar linear function relationship to y = kx, the only difference being the value of k. The possible values ​​of k are within a certain range, such as [k1, k2] and [k3, k4]. Thus, the maximum included angle region of each target object can be mapped to a slope range.

[0049] Second, the overlap phenomenon is judged by angle. In this embodiment of the invention, judging whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum and minimum angles of the maximum included angle regions of the target objects; calculating the angle range of the target objects based on the maximum and minimum angles; judging whether the angle ranges of multiple target objects overlap. If the angle ranges of M target objects overlap, then it is determined that the M target objects overlap. Specifically, this method actually maps the boundary line of the maximum included angle region to the angle with a reference line, thereby obtaining the maximum and minimum angles of the maximum included angle region; calculating the angle range of each target object based on the maximum and minimum angles of each target object. The existence of overlap is determined by whether the angle ranges overlap.

[0050] Figure 4 A schematic diagram illustrating a method for determining the maximum included angle region according to an embodiment of the present invention is shown. Figure 4 As shown in the embodiment of the present invention, a method for determining the maximum included angle region is provided. This method for determining the maximum included angle region formed by connecting the boundary point of the target object in the base image with the coordinate transformation point includes: determining the center point of the target object in the base image; determining a first line connecting the coordinate transformation point and the center point in the base image; determining a second line perpendicular to the first line in the base image at the center point; determining the intersection point where the second line coincides with the boundary point of the target object in the base image, which in this embodiment are intersection point 1 and intersection point 2 in the figure; and connecting the intersection point with the coordinate transformation point to form the maximum included angle region. In practical applications, since the target object may be a microorganism with irregular edges, the number of intersection points may be greater than two. Determining the maximum included angle region based on multiple intersection points can maximize the identification of overlapping target microorganisms, thereby further improving the efficiency of capturing target microorganisms.

[0051] In this embodiment, the above content discloses a technical means of determining whether two or more target objects overlap by using the area of ​​the largest included angle.

[0052] The following details the process of combining and judging multiple target objects identified on a base image. Specifically, determining whether multiple target objects in the base image will overlap in another image includes: arbitrarily selecting one target object in the base image as the current target object; comparing the other target objects in the base image with the current target object to determine if they overlap; and from the remaining target objects in the base image, arbitrarily selecting another target object in the base image as the current target object, and comparing the other remaining target objects with the current target object to determine if they overlap. The remaining target objects refer to the other target objects in the base image that have been excluded from being selected as the current target object or identified as overlapping target objects.

[0053] Here's an example: If four target objects (1-4) are identified in the base image, then target object 1 is arbitrarily selected as the current target object. Target objects 2-4 are compared with target object 1, and it is determined whether they overlap. If target object 2 overlaps with target object 1, then the remaining target objects include target objects 3 and 4. Target object 3 is then arbitrarily selected as the next current target object, and it is determined whether the remaining target object 4 overlaps with target object 3. The process ends here. In another example, target object 1 is arbitrarily selected as the current target object. Target objects 2-4 are compared with target object 1, and it is determined whether they overlap. If both target objects 2 and 3 overlap with target object 1, then target objects 1-3 are overlapping target objects, and the remaining target objects only include target object 4. The process ends here.

[0054] Preferably, determining whether multiple target objects in the base image overlap in another image includes: determining whether multiple target objects in a preset region of the base image overlap in another image. The preset region can be defined as the region where the execution endpoint is located. This way, only overlapping target objects around the region near the execution endpoint need to be determined, which reduces the determination time compared to determining whether all target objects in the base image overlap, thus improving capture efficiency. Furthermore, since the position of a target object changes over time within its environment, reducing the determination time means reducing the distance the target object moves, thereby improving the accuracy and precision of the capture.

[0055] The following is combined Figures 5a-5c The invention will be explained in detail. Figure 5a A schematic diagram illustrating the shooting angle according to an embodiment of the present invention is shown. Figure 5aAs shown, the shooting angles of the at least two images include a set of mutually perpendicular shooting angles. The reason for choosing mutually perpendicular shooting angles is primarily because the coordinate relationship between the images is more compatible with a Cartesian coordinate system, reducing the computational complexity of coordinate transformations when calculating the target object's spatial position. Of course, in practical applications, there can be three or more shooting angles. The choice can be made based on actual needs, considering both the higher accuracy but cost of more shooting angles and the lower cost of fewer shooting angles. Here, the mutually perpendicular shooting angles are preferably horizontal and vertical angles, as this simplifies the calculation of the target object's position information.

[0056] like Figure 5a As shown, if the shooting angle of shooting device 1 is along the horizontal direction, the obtained image is in the XY coordinate system. If the shooting angle of shooting device 2 is along the vertical direction, the obtained image is in the YZ coordinate system. Figure 5b The image captured by the imaging device 1 is shown schematically. A target object D exists in the image captured by the imaging device 1, and the coordinates of the target object D are (x1, y1). Figure 5c The image captured by imaging device 2 is illustrated schematically. The image captured by imaging device 2 contains independent images of target objects A, B, and C. The coordinates of target object A are (y1, z1), the coordinates of target object B are (y1, z2), and the coordinates of target object C are (y1, z3). Since the y-axis coordinates of target objects A, B, C, and D are all y1 based on the angular relationship between imaging devices 1 and 2, the coordinate system settings, and the imaging parameters, it is determined that target objects with the same y1 coordinate overlap in another image. Target objects A, B, and C are referred to as overlapping target objects, and the target object formed by the overlap in another image is target object D. Therefore, the coordinates of target object A are saved as (x1, y1, z1), the coordinates of target object B are saved as (x1, y1, z2), and the coordinates of target object C are saved as (x1, y1, z3), thus determining the positional information of the overlapping target objects.

[0057] It should be noted that the above is just a simple example. In practical applications, since the imaging area of ​​the imaging device is a conical space formed by the emitted rays, to determine whether the target objects in the two images overlap, it is necessary to first transform the coordinates of the target objects in the images captured by the two imaging devices according to the trigonometric theorem, and then determine whether there is an overlap based on the transformed coordinates.

[0058] The above describes the action of determining overlapping target objects when there is overlap. Of course, in practical applications, if there is no overlap, the coordinates of the target object in each image are directly saved.

[0059] Figure 6 The diagram schematically illustrates the steps of a target object capture method according to an embodiment of the present invention. Figure 6 As shown, in this embodiment, the method includes the following steps:

[0060] S11. Real-time acquisition of at least two images, including the execution end, with at least two images taken from different angles;

[0061] S12. Identify target objects in at least two images, determine the position of the same target object in each image according to the aforementioned method for determining the position of overlapping target objects, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image.

[0062] S13. Control the execution end to move towards the target object to be captured until the relative position between the execution end and the target object to be captured meets the capture condition, and control the execution end to capture the target object to be captured.

[0063] In step S12, the aforementioned method for determining the position of overlapping target objects is used to determine the position of the same target object in each image, that is, to determine the position information of overlapping target objects.

[0064] The following describes the implementation of the target object capture method, taking microorganisms in solution as the target object and a pipette from a robotic arm as the end effector:

[0065] Step 1: Simultaneously take pictures with imaging device 1 and imaging device 2 to obtain images of the microorganisms in the entire solution and the pipette at the end of the robotic arm. The images taken here require that the resolution of the specific microorganisms in the image is not less than 20 pixels and there is no obvious distortion.

[0066] Step 2: Obtain the manually defined size range for specific microorganisms. Based on the imaging adjustments made to the specific microorganism by the imaging device in the previous step, the size range of that specific microorganism in the image is statistically analyzed. Manual setting is relatively simple and does not require a learning-based statistical approach. Since the positions of the imaging device and the container containing the microorganism are fixed, the size can be set relatively large to ensure the microorganism's pixels are clear.

[0067] Step 3: Extraction of Edge Coordinates of Invalid Targets. Image preprocessing is performed on the images from both imaging devices 1 and 2 to obtain the coordinates of invalid targets, facilitating subsequent removal. The image preprocessing steps are: removing invalid regions → median filtering → image sharpening → edge detection → calculating the size of connected components in each region → marking connected components that do not conform to the specific microbial size range specified in Step 4 as invalid target regions → obtaining the edge coordinates (x, y) of invalid targets; and using the remaining connected components as the edge coordinates (X, Y) of valid microorganisms.

[0068] Step 4: Perform image preprocessing on the images from imaging device 1 and imaging device 2 respectively to make the specific microorganisms stand out more in the images for easier identification. This results in two images where the specific microorganisms are clearly visible. The image preprocessing steps for this part are: removing invalid regions → median filtering → color channel format conversion → image sharpening.

[0069] Step 5: Calculate the position of the straw end in the image. Since the size of the robotic end effector and the angle of the robotic arm are fixed and the robotic arm is obvious, it is easy to use template matching to calculate the position of the robotic end effector in the two images acquired by imaging device 1 and imaging device 2 respectively.

[0070] Step Six: Use the trained offline model to identify the two images obtained in the previous step, obtaining the identification results of specific microorganisms distributed on the two images. The trained offline model can be obtained as follows: collect a large number of image samples of specific individual microorganisms and microbial communities; train the model on the collected image samples using a deep learning algorithm for object detection until the model converges, obtaining a good model that can identify specific microorganisms from multiple microorganisms coexisting in the container and locate the microorganism's position on the image. This process is an offline learning process, mainly to obtain a reliable algorithm model for microbial target identification and localization.

[0071] The steps in this section include: normalizing the image based on the previous step → extracting features using an offline model and obtaining the recognition result (the coordinates of the rectangle containing the microorganism and the probability value that the model considers the recognition result to be a specific microorganism) → post-processing the result to obtain the final result. The main post-processing methods are: 1) removing recognition results with low probability values; 2) calculating the size of the rectangle containing the microorganism and removing recognition results that do not meet the aforementioned constraints.

[0072] Step 7: Confirmation of final specific microorganism identification results. Combining steps 3 and 6, invalid targets are eliminated, and the remaining targets are taken as the final specific microorganism region. At the same time, the coordinates of the corresponding rectangle and the coordinates of the center point are obtained.

[0073] Step 8: Calculate the relative position of the microorganism and the robotic arm end effector in 3D space, and control the robotic arm end effector to move toward the target microorganism based on the calculation results. When the relative position between the robotic arm end effector and the target microorganism meets the capture conditions, control the robotic arm end effector to capture the target microorganism.

[0074] Through the above implementation methods, it is possible to quickly and accurately operate on specific microorganisms, whether they are completely or partially obscured.

[0075] Figure 7 A schematic diagram illustrating the structure of a device for determining the position of overlapping target objects according to an embodiment of the present invention is shown. Figure 7 As shown, this embodiment of the invention provides a device for determining the position of overlapping target objects. The device includes: an image acquisition module 701, used to acquire at least two images including the environment where the target object is located, wherein the at least two images are captured from different angles; a target recognition module 702, used to identify the target object in each of the images; an overlap determination module 703, used to use one image as a base image, and based on the shooting angle relationship between the at least two images, determine whether multiple target objects in the base image will overlap in another image; if M target objects overlap in another image, determine the position information of each of the M target objects in the other image based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of the M target objects in the other image is called overlapping target objects; and a position determination module 704, used to determine the position information of each of the M target objects in the at least two images as the position information of overlapping target objects.

[0076] Through the above technical solution, the device for determining the position of overlapping target objects provided by the present invention can quickly and accurately determine the position of overlapping target objects.

[0077] In some optional embodiments, determining whether multiple target objects in the base image will overlap on another image based on the shooting angle relationship of the at least two images includes: determining the maximum angle region formed by the lines connecting the boundary points of each target object in the base image and the projection points of the imaging reference points of the other image on the base image; determining whether the maximum angle regions of multiple target objects overlap; if the maximum angle regions of M target objects overlap, then determining that the M target objects overlap.

[0078] In some optional embodiments, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum slope and minimum slope of the maximum included angle region; calculating the slope range of each target object based on the maximum slope and minimum slope of each target object; determining whether the slope ranges of multiple target objects overlap; if the slope ranges of M target objects overlap, then determining that the M target objects overlap.

[0079] In some optional embodiments, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum and minimum angles of the maximum included angle regions; calculating the angle range of each target object based on the maximum and minimum angles of each target object; determining whether the angle ranges of multiple target objects overlap; if the angle ranges of M target objects overlap, then determining that the M target objects overlap.

[0080] In some optional embodiments, determining the maximum angle region formed by connecting the boundary points of the plurality of target objects in the base image with the imaging reference point of the other image includes: determining the center point of the target object in the base image; determining a first line connecting the imaging reference point and the center point; determining a second line on the base image perpendicular to the first line at the center point; determining two intersection points of the second line coinciding with the boundary points of the target object in the base image; and determining that each of the two intersection points forms the maximum angle region with the imaging reference point.

[0081] In some optional embodiments, determining whether the maximum included angle regions of multiple target objects overlap includes: selecting one of the target objects in the base image as the current target object; comparing the maximum included angle regions of the other target objects in the base image with the maximum included angle region of the current target object one by one to determine whether there is an overlap; if there is an overlap with the current target object.

[0082] In some optional embodiments, determining whether the maximum included angle regions of multiple target objects overlap further includes: reselecting and determining another target object in the base image as the current target object; comparing the maximum included angle regions of the other remaining target objects in the base image with the maximum included angle region of the current target object one by one to determine whether there is an overlap, and if there is an overlap with the current target object.

[0083] In some alternative embodiments, the shooting angles of the at least two images include at least a set of mutually perpendicular shooting angles.

[0084] Figure 8A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. The present invention provides an image signal generator, including: a memory 810 configured to store instructions; and a processor 820 configured to retrieve instructions from the memory 810 and, when executing the instructions, to implement the steps of the above-described overlapping target object position determination method and / or the steps of the above-described target object capture method.

[0085] In this embodiment of the invention, the processor 820 can be configured to: acquire at least two images including the environment where the target object is located, wherein the at least two images are captured from different angles; identify the target object in each of the images; using one image as a base image, determine whether multiple target objects in the base image will overlap in another image based on the relationship between the shooting angles of the at least two images; if M target objects overlap in another image, determine the position information of each of the M target objects in the other image based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of the M target objects in the other image is called overlapping target objects; and determine the position information of each of the M target objects in the at least two images as the position information of the overlapping target objects.

[0086] Furthermore, the processor 820 can also be configured such that, preferably, determining whether multiple target objects in the base image will overlap on another image based on the shooting angle relationship of the at least two images includes: determining the maximum angle region formed by the lines connecting the boundary points of each target object in the base image and the projection points of the imaging reference points of the other image on the base image; determining whether the maximum angle regions of multiple target objects overlap; if the maximum angle regions of M target objects overlap, then determining that the M target objects overlap.

[0087] Preferably, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum slope and minimum slope of the maximum included angle region; calculating the slope range of each target object based on the maximum slope and minimum slope of each target object; determining whether the slope ranges of multiple target objects overlap; if the slope ranges of M target objects overlap, then determining that the M target objects overlap.

[0088] Preferably, determining whether the maximum included angle regions of multiple target objects overlap includes: determining the maximum and minimum angles of the maximum included angle regions; calculating the angle range of each target object based on the maximum and minimum angles of each target object; determining whether the angle ranges of multiple target objects overlap; if the angle ranges of M target objects overlap, then determining that the M target objects overlap.

[0089] Preferably, determining the maximum angle region formed by connecting the boundary points of multiple target objects in the base image with the imaging reference point of the other image includes: determining the center point of the target object in the base image; determining a first line connecting the imaging reference point and the center point; determining a second line perpendicular to the first line on the base image at the center point; determining two intersection points where the second line coincides with the boundary points of the target object in the base image; and determining that each of the two intersection points forms the maximum angle region with the imaging reference point.

[0090] Preferably, determining whether the maximum included angle regions of multiple target objects overlap includes: selecting one of the target objects in the base image as the current target object; comparing the maximum included angle regions of the other target objects in the base image with the maximum included angle region of the current target object one by one to determine whether there is an overlap; if there is an overlap with the current target object.

[0091] Preferably, determining whether the maximum included angle regions of multiple target objects overlap further includes: reselecting and determining another target object in the base image as the current target object; comparing the maximum included angle regions of the other remaining target objects in the base image with the maximum included angle region of the current target object one by one to determine whether there is an overlap; if there is an overlap with the current target object.

[0092] Preferably, the shooting angles of the at least two images include at least one set of mutually perpendicular shooting angles.

[0093] Furthermore, the processor 820 can also be configured to: acquire in real time at least two images, including the execution terminal, wherein the at least two images are captured from different angles; identify target objects in the at least two images; determine the position of the same target object in each image according to the aforementioned method for determining the position of overlapping target objects; determine the target object to be captured in the target objects based on the position information of the execution terminal and each target object in each image; control the execution terminal to move towards the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition; and control the execution terminal to capture the target object to be captured.

[0094] This invention also provides a machine-readable storage medium storing instructions that cause a machine to perform the steps of the aforementioned overlapping target object location determination method and / or the steps of the aforementioned target object capture method.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0100] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0101] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0102] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0103] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for determining the position of overlapping target objects, characterized in that, include: Acquire at least two images, including the environment in which the target object is located, wherein the at least two images are captured from different angles; Identify the target object in each of the images; Using one image as the base image, based on the shooting angle relationship between the at least two images, it is determined whether multiple target objects in the base image will overlap in another image. If M target objects overlap in another image, the position information of each of the M target objects in the other image is determined based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of M target objects in the other image is called overlapping target objects. Determining the position information of each of the M target objects on at least two images as the position information of overlapping target objects; the step of determining whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship of the at least two images includes: Determine the region of the largest included angle formed by the line connecting the boundary point of the target object in the base image and the coordinate transformation point of the shooting position of the other image on the base image; Determine whether the maximum included angle regions of multiple target objects overlap. If the maximum included angle regions of M target objects overlap, then it is determined that the M target objects overlap.

2. The method according to claim 1, characterized in that, The determination of whether the maximum included angle regions of multiple target objects overlap includes: Determine the maximum and minimum slopes of the maximum included angle region of the target object; The slope range of the target object is determined based on the maximum and minimum slopes of the target object; Determine whether the slope ranges of multiple target objects overlap. If the slope ranges of M target objects overlap, then it is determined that the M target objects overlap.

3. The method according to claim 1, characterized in that, The determination of whether the maximum included angle regions of multiple target objects overlap includes: Determine the maximum and minimum angles of the maximum included angle region of the target object; Calculate the angle range of the target object based on its maximum and minimum angles; Determine whether the angular ranges of multiple target objects overlap. If the angular ranges of M target objects overlap, then it is determined that the M target objects overlap.

4. The method according to claim 1, characterized in that, Determining the region with the largest included angle formed by the lines connecting the boundary points of the target object in the base image and the coordinate transformation points includes: Determine the center point of the target object in the base image on the base image; Determine a first line connecting the coordinate transformation point and the center point on the base image; At the center point, determine a second line on the base image that is perpendicular to the first line; Determine the intersection point where the second connecting line coincides with the boundary point of the target object; Connect the intersection point with the coordinate transformation point to form the region with the largest included angle.

5. The method according to claim 1, characterized in that, The step of determining whether multiple target objects in the base image will overlap in another image includes: Arbitrarily select one of the target objects in the base image as the current target object; Compare the other target objects in the base image with the current target object to determine whether there is any overlap with the current target object; Among the remaining target objects in the base image, arbitrarily select another target object in the base image as the current target object, and compare the other target objects among the remaining target objects with the current target object to determine whether there is an overlap with the current target object; The remaining target objects refer to the other target objects in the base image that have been excluded from being the current target object or identified as overlapping target objects.

6. The method according to claim 1, characterized in that, The step of determining whether multiple target objects in the base image will overlap in another image includes: Determine whether multiple target objects in a preset region of the base image will overlap in another image.

7. The method according to claim 1, characterized in that, The shooting angles of the at least two images include at least one set of mutually perpendicular shooting angles.

8. A method for capturing a target object, characterized in that, Includes the following steps: Real-time acquisition of at least two images, including those at the execution end, wherein the at least two images are captured from different angles; Identify target objects in at least two images, determine the position of the same target object in each image according to the overlapping target object position determination method according to any one of claims 1-7, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image; The execution terminal is controlled to move toward the target object to be captured until the relative position between the execution terminal and the target object to be captured meets the capture condition, and then the execution terminal is controlled to capture the target object to be captured.

9. A device for determining the position of overlapping target objects, characterized in that, include: An image acquisition module is used to acquire at least two images, including the environment in which the target object is located, wherein the at least two images are captured from different angles; The target recognition module is used to identify the target object in each of the images; The overlap determination module is used to determine whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship between the at least two images, using one image as the base image. If M target objects overlap in another image, the position information of each of the M target objects in the other image is determined based on the position information of the overlapping target objects in the other image, where M is an integer greater than or equal to 2; the image of M target objects in the other image is called overlapping target objects. as well as The position determination module is used to determine the position information of each of the M target objects on at least two images as the position information of overlapping target objects; The step of determining whether multiple target objects in the base image will overlap in another image based on the shooting angle relationship of the at least two images includes: Determine the region of the largest included angle formed by the line connecting the boundary point of the target object in the base image and the coordinate transformation point of the shooting position of the other image on the base image; Determine whether the maximum included angle regions of multiple target objects overlap. If the maximum included angle regions of M target objects overlap, then it is determined that the M target objects overlap.

10. A target object capturing device, characterized in that, include: An image acquisition module is used to acquire at least two images, including the execution end, in real time, wherein the at least two images are captured from different angles; The target determination module is used to identify target objects in the at least two images, determine the position of the same target object in each image according to the overlapping target object position determination method according to any one of claims 1-7, and determine the target object to be captured in the target objects based on the execution terminal and the position information of each target object in each image; as well as The end-point control module is used to control the execution end to move towards the target object to be captured until the relative position between the execution end and the target object to be captured meets the capture condition, and to control the execution end to capture the target object to be captured.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for determining the position of overlapping target objects as described in any one of claims 1 to 7 and / or the steps of the target object capture method as described in claim 8.

12. A storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the method for determining the position of an overlapping target object as claimed in any one of claims 1 to 7 and / or the steps of the method for capturing a target object as claimed in claim 8.

Images