Vision-based sample tube selection and exception decision processing method and system

Abstract

The invention belongs to the field of biological sample storage, and particularly relates to a vision-based sample tube selection and abnormity decision processing method and system. Before a tube selection clamping jaw extracts a sample tube from a cryopreservation box and after tube selection abnormity is detected in a common abnormity detection process, the tube selection clamping jaw can be used for selecting the sample tube from the cryopreservation box; the arrangement condition of the sample tubes in the cryopreservation box is photographed through the visual recognition assembly, scene images are captured, different types of capturing scenes are judged, and according to the different types of capturing scenes, action strategies which should be adopted by the mechanical arm tube picking system are judged to process abnormal sample tubes. The degree of freedom of action execution is increased, the pipe picking action in the normal state can be increased, the pipe picking action in the abnormal state can be achieved, the self-adaptability and intelligence of the whole system are improved, independent work can be achieved, the existing anomaly detection process can be improved, manual disposal can be avoided in the anomaly disposal decision process, and the labor intensity of workers is reduced. The response time is shortened, the exposure time of the sample in a high-temperature environment is shortened, and the activity of the sample in the sample is protected.

Description

Technical Field

[0001] The present invention belongs to the field of biological sample storage, and in particular relates to a vision-based sample tube selection and abnormal decision-making disposal method and system. Background Art

[0002] The existing cryobox picking mechanism generally adds a camera at the bottom to identify the presence or absence of cryotubes. When working, the camera can record and analyze the changes in the empty space of the cryobox before and after the tube is picked, and compare them with the pre-written value. If the detection value after picking the tube is not empty or the detection value before picking the tube is empty, it means that an abnormality has been detected.

[0003] Since only the vacancy changes can be detected, when the abnormal cryopreservation tube status is complex, the existing solution is helpless and can only handle the abnormal samples manually. This is time-consuming and labor-intensive, and affects the survival rate of the samples.

[0004] When the box picking and placing mechanism operates for a long time in a deep cryogenic environment, frost will gradually form on the surface of the cryobox, increasing the positioning error of the sample tube, causing the robot arm to fail to pick up the tube, and the sample tube that was not successfully taken out will show an abnormal posture and block other sample tubes. If this abnormal situation is not handled in time, it will affect the subsequent tube picking tasks.

[0005] Chinese patent CN117262658A discloses a device and method for high-precision intelligent positioning and guidance of a storage mechanism access channel, which can accurately align with a cryobox access channel or a deformed cryobox access channel. The cryobox access plate can smoothly access the cryobox, thereby improving the positioning accuracy of the cryobox access channel and reducing manual intervention and maintenance costs. However, when the abnormal state of the cryotube is relatively complex, this patent cannot solve the problem of cryotubes with abnormal postures. Summary of the invention

[0006] The present invention proposes a vision-based sample tube selection and abnormal decision-making method and system, which solves the above-mentioned problem by taking images and prompt texts as inputs of a multimodal model, processing and generating intelligent decision-making information in a large model, and guiding the robot to reasonably handle cryopreservation tubes with abnormal postures.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows: In a first aspect, a method for selecting sample tubes and handling abnormal decisions based on vision comprises the following steps: Step S1: before the tube picking gripper extracts the sample tube from the cryopreservation box, after the common abnormality detection process detects the abnormality of tube picking, the visual recognition component takes a picture of the arrangement of the sample tubes in the cryopreservation box to capture the scene image; Step S2: the image is passed into the multimodal model for recognition, and the image content is analyzed with the help of prompt text to determine different types of grabbing scenarios; Step S3: Determine the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes according to different types of grasping scenarios.

[0008] Preferably, step S2 comprises the following steps: Step S21: passing the image into the multimodal model for recognition; Step S22: Analyzing the image content with the aid of a prompt text, the prompt text explaining the background information of the image and providing an example of the format of the output result, including the output of the number, posture and spatial relationship fields of the abnormal sample tubes; Step S23: Determine different types of grabbing scenarios based on the number, arrangement posture and spatial relationship of the returned abnormal sample tubes.

[0009] Preferably, step S3 determines the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes according to different types of grabbing scenarios, including: According to the number of abnormal sample tubes, determine whether the tube picking system needs to adopt the abnormal processing mode. If the number of abnormal sample tubes is 0, the tube picking system will be positioned normally. If the number of abnormal sample tubes is ≥ 1, the abnormal processing mode needs to be adopted. If the number of abnormal sample tubes is greater than 1, determine whether the abnormal sample tubes are in contact with each other. If the abnormal sample tubes are in contact with each other, further determine the spatial relationship between the abnormal sample tubes. If the abnormal sample tubes are crossed, the robotic arm tube picking system pushes the upper sample tube away and grabs the sample tube. If the abnormal sample tubes are parallel, the robotic arm tube picking system pushes any sample tube away from the front of the tube cover and grabs the sample tube. If the abnormal sample tubes are not in contact with each other, the robotic arm tube picking system will directly grab any sample tube; If the number of abnormal sample tubes is 1, determine the posture of the abnormal sample tubes. If the abnormal sample tubes are in a horizontal position, the robotic arm tube picking system grabs the middle of the sample tube. If the abnormal sample tubes are in a reclining position, grab them from above the sample tube cover.

[0010] Preferably, the multimodal model is GPT-4 or Gemini 1.5.

[0011] In a second aspect, a vision-based sample tube selection and abnormal decision-making processing system includes: A storage mechanism, wherein a freezing box is disposed on the top of the storage mechanism, and a plurality of sample tubes are disposed in the freezing box; A visual recognition component, the visual recognition component includes a visual recognition unit, a USB data cable and a computer, the USB data cable is used to transmit visual images between the visual recognition unit and the computer, and the visual recognition component is located above the tube selection area; A robotic arm tube picking system, the robotic arm tube picking system comprising a robotic arm and a tube picking claw, the tube picking claw being fixed on the robotic arm, and the tube picking claw being used to take out a sample tube from a storage mechanism; Also includes multimodal model software.

[0012] According to a third aspect, a computing device includes: one or more processors; The storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described above.

[0013] In a fourth aspect, a computer-readable storage medium stores a program, and when the program is executed by a processor, any of the methods described above is implemented.

[0014] The beneficial effects of the present invention are: Before the tube picking gripper of the present invention extracts the sample tube from the cryopreservation box, after the ordinary abnormality detection process detects the abnormality of the tube picking, the arrangement of the sample tubes in the cryopreservation box is photographed and captured by the visual recognition component, and the image is passed into the multimodal model for recognition, and the image content is analyzed with the help of prompt text to determine different types of grabbing scenes, and the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes is determined according to different types of grabbing scenes. The method provided by the patent of the present invention can work independently or can be an improvement of the existing abnormality detection process. The abnormality handling decision process can avoid manual handling, shorten the response time, reduce the exposure time of samples in high temperature environment, and protect the activity of the samples therein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a vision-based sample tube selection and abnormal decision-making method provided by an embodiment of the present invention.

[0016] Figure 2 It is a capture scene judgment flow chart of a vision-based sample tube selection and abnormal decision-making and handling method provided by an embodiment of the present invention.

[0017] Figure 3 It is a schematic diagram of a vision-based sample tube selection and abnormal decision-making and handling system provided by an embodiment of the present invention.

[0018] Figure 4 It is a schematic diagram of a vision-based sample tube selection and abnormal decision-making and handling system provided by an embodiment of the present invention.

[0019] Figure 5It is a schematic diagram of a vision-based sample tube selection and abnormal decision-making and handling system provided by an embodiment of the present invention.

[0020] In the figure: 1-robotic arm tube picking system, 11-robotic arm, 12-tube picking gripper, 2-visual recognition component, 3-storage mechanism, 31-freezing box, 32-sample tube. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a method and system for selecting sample tubes and handling abnormal decisions based on vision, and the method includes the following steps: Step S1: before the tube picking gripper extracts the sample tube from the cryopreservation box, after the common abnormality detection process detects the abnormality of tube picking, the visual recognition component takes a picture of the arrangement of the sample tubes in the cryopreservation box to capture the scene image; Step S2: the image is passed into the multimodal model for recognition, and the image content is analyzed with the help of prompt text to determine different types of grabbing scenarios; Step S3: Determine the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes according to different types of grasping scenarios.

[0023] In this embodiment of the present invention, step S2 includes the following steps: Step S21: passing the image into the multimodal model for recognition; Step S22: Analyzing the image content with the aid of a prompt text, the prompt text explaining the background information of the image and providing an example of the format of the output result, including the output of the number, posture and spatial relationship fields of the abnormal sample tubes; Step S23: Determine different types of grabbing scenarios based on the number, arrangement posture and spatial relationship of the returned abnormal sample tubes.

[0024] like Figure 2 As shown, in the embodiment of the present invention, step S3 determines the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes according to different types of grabbing scenarios, including: According to the number of abnormal sample tubes, determine whether the tube picking system needs to adopt the abnormal processing mode. If the number of abnormal sample tubes is 0, the tube picking system will be positioned normally. If the number of abnormal sample tubes is ≥ 1, the abnormal processing mode needs to be adopted. The multimodal model includes but is not limited to GPT-4, Gemini 1.5 and other models. In the embodiment of the present invention, GPT-4 is preferred.

[0025] In the embodiment of the present invention, if the number of abnormal sample tubes is 2, it is determined whether the abnormal sample tubes are in contact with each other. If the abnormal sample tubes are in contact with each other, the spatial relationship between the abnormal sample tubes is further determined. If the abnormal sample tubes are crossed, the robotic arm tube picking system pushes the upper sample tube away and grabs the tube. If the abnormal sample tubes are parallel, the robotic arm tube picking system pushes any sample tube away from the front of the tube cover and grabs the tube. If the abnormal sample tubes are not in contact with each other, the robotic arm tube picking system will directly grab any sample tube; If the number of abnormal sample tubes is 1, determine the posture of the abnormal sample tubes. If the abnormal sample tubes are in a horizontal position, the robotic arm tube picking system grabs the middle of the sample tube. If the abnormal sample tubes are in a reclining position, grab them from above the sample tube cover.

[0026] In an embodiment of the present invention, according to the returned disposal decision, the corresponding robot arm action function is called to process the abnormal sample tube, otherwise the sample tube is picked according to the normal working process. The method provided by the patent of the present invention can work independently or be an improvement of the existing abnormal detection process. The abnormal disposal decision process can avoid manual disposal, shorten the response time, reduce the exposure time of the sample in the high temperature environment, and protect the activity of the sample inside.

[0027] The above method can be directly used in a device and method for high-precision intelligent positioning and guiding of a storage mechanism access channel disclosed in Chinese patent CN117262658A.

[0028] like Figure 3-Figure 5 As shown, the embodiment of the present invention also provides a vision-based sample tube selection and abnormal decision-making processing system, including: A storage mechanism 3, wherein a freezing box 31 is disposed on the top of the storage mechanism 3, and a plurality of sample tubes 32 are disposed in the freezing box 31; The visual recognition component 2 includes a visual recognition unit, a USB data cable and a computer. The USB data cable is used to transmit visual images between the visual recognition unit and the computer. The visual recognition component 2 is located above the pipe picking area, specifically, installed on the robotic arm 11.

[0029] The visual recognition unit includes but is not limited to a 3D camera, a binocular camera, a trinocular camera, and a common automatic zoom camera, etc. In a low-light or no-light environment, it can be used in conjunction with a lighting system to assist in recognition. In this embodiment, a 3D camera is preferred.

[0030] A robotic arm tube picking system 1, the robotic arm tube picking system 1 comprising a robotic arm 11 and a tube picking clamp 12, the tube picking clamp 12 being fixed at the end of the robotic arm 11, and the tube picking clamp 12 being used to take out a sample tube from a storage mechanism 3; In this embodiment, the robot arm 11 is preferably a six-degree-of-freedom robot arm, which increases the degree of freedom of executing actions. It can not only increase the pipe picking action under normal conditions, but also realize the pipe picking action under abnormal conditions, thereby improving the adaptability and intelligence of the overall system. In addition, the robot arm can also be any robot arm in the prior art, such as a three-axis truss, four-axis or five-axis robot arm.

[0031] In addition, the robotic arm can be in a single-arm form, a dual-arm form, or a multi-arm collaborative form, and the collaborative control method can be based on a large model (for example, a dual-arm robot controlled by RDT, the full name of RDT is Robotics Diffusion Transformer, which is the world's largest diffusion-based model for dual-arm robot operation tasks, built by the TSAIL team of the Tsinghua University Artificial Intelligence Research Institute).

[0032] Also includes multimodal model software.

[0033] In this embodiment, before the tube picking gripper 12 extracts the sample tube 32 from the freezing box 31, after the common abnormality detection process detects the abnormality of the tube picking, the visual recognition component 2 takes a picture of the arrangement of the sample tubes 32 in the freezing box 31 to capture the scene image; Step S2: the image is passed into the multimodal model for recognition, and the image content is analyzed with the help of prompt text to determine different types of grabbing scenarios; Step S3: Determine the action strategy that the robot arm tube picking system 1 should take to handle abnormal sample tubes according to different types of grasping scenarios.

[0034] In this embodiment of the present invention, step S2 includes the following steps: Step S21: passing the image into the multimodal model for recognition; Step S22: Analyzing the image content with the aid of a prompt text, the prompt text explaining the background information of the image and providing an example of the format of the output result, including the output of the number, posture and spatial relationship fields of the abnormal sample tubes; Step S23: With the help of prompts, such as text or voice, etc., taking text as an example here, different types of grasping scenarios are determined according to the number, arrangement posture and spatial relationship of the returned abnormal sample tubes.

[0035] like Figure 2 As shown, in the embodiment of the present invention, step S3 determines the action strategy that the robot arm tube picking system 1 should take to handle abnormal sample tubes according to different types of grabbing scenarios, including: According to the number of abnormal sample tubes, determine whether the tube picking system needs to adopt the abnormal processing mode. If the number of abnormal sample tubes is 0, the tube picking system will be positioned normally. If the number of abnormal sample tubes is ≥ 1, the abnormal processing mode needs to be adopted. The multimodal model includes but is not limited to GPT-4, Gemini 1.5 and other models. In the embodiment of the present invention, GPT-4 is preferred.

[0036] In the embodiment of the present invention, if the number of abnormal sample tubes is 2, it is determined whether the abnormal sample tubes are in contact with each other. If the abnormal sample tubes are in contact with each other, the spatial relationship between the abnormal sample tubes is further determined. If the abnormal sample tubes are crossed, the robot arm tube picking system 1 pushes the upper sample tube away and grabs it. If the abnormal sample tubes are parallel, the robot arm tube picking system 1 pushes any sample tube away from the front of the tube cover and grabs it. If the abnormal sample tubes are not in contact with each other, the robot arm tube picking system 1 directly grabs any sample tube; If the number of abnormal sample tubes is 1, the posture of the abnormal sample tubes is determined. If the abnormal sample tube is in a horizontal position, the robotic arm tube picking system 1 grabs the middle of the sample tube. If the abnormal sample tube is in a reclining position, it is grabbed from above the sample tube cover.

[0037] It should be noted that the system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0038] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0039] The embodiment of the present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0040] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0041] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0042] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0043] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0044] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0045] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0046] In addition, it should be pointed out that in the apparatus and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. Moreover, the steps of performing the above series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it can be understood that all or any steps or components of the method and apparatus of the present invention can be implemented in hardware, firmware, software or a combination thereof in any computing device (including processors, storage media, etc.) or a network of computing devices, which can be achieved by those of ordinary skill in the art using basic programming skills after reading the description of the present invention.

[0047] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing device. The computing device can be a well-known general device. Therefore, the purpose of the present invention can also be achieved by simply providing a program product containing a program code that implements the method or device. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the device and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order. Some steps can be performed in parallel or independently of each other.

[0048] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for selecting sample tubes and handling abnormal decisions based on vision, characterized in that: The steps include: Step S1: before the tube picking gripper extracts the sample tube from the cryopreservation box, after the common abnormality detection process detects the abnormality of tube picking, the visual recognition component takes a picture of the arrangement of the sample tubes in the cryopreservation box to capture the scene image; Step S2: the image is passed into the multimodal model for recognition, and the image content is analyzed with the help of prompt text to determine different types of grabbing scenarios; Step S3: Determine the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes according to different types of grasping scenarios.

2. The method for selecting sample tubes and handling abnormal decisions based on vision according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: passing the image into the multimodal model for recognition; Step S22: Analyzing the image content with the aid of a prompt text, the prompt text explaining the background information of the image and providing an example of the format of the output result, including the output of the number, posture and spatial relationship fields of the abnormal sample tubes; Step S23: Determine different types of grabbing scenarios based on the number, arrangement posture and spatial relationship of the returned abnormal sample tubes.

3. The method for selecting sample tubes and handling abnormal decisions based on vision according to claim 2 is characterized in that: Step S3 determines the action strategy that the robot arm tube picking system should take to handle abnormal sample tubes according to different types of grabbing scenarios, including: According to the number of abnormal sample tubes, determine whether the tube picking system needs to adopt the abnormal processing mode. If the number of abnormal sample tubes is 0, the tube picking system will be positioned normally. If the number of abnormal sample tubes is ≥ 1, the abnormal processing mode needs to be adopted. If the number of abnormal sample tubes is greater than 1, determine whether the abnormal sample tubes are in contact with each other. If the abnormal sample tubes are in contact with each other, further determine the spatial relationship between the abnormal sample tubes. If the abnormal sample tubes are crossed, the robotic arm tube picking system pushes the upper sample tube away and grabs the sample tube. If the abnormal sample tubes are parallel, the robotic arm tube picking system pushes any sample tube away from the front of the tube cover and grabs the sample tube. If the abnormal sample tubes are not in contact with each other, the robotic arm tube picking system will directly grab any sample tube; If the number of abnormal sample tubes is 1, determine the posture of the abnormal sample tubes. If the abnormal sample tubes are in a horizontal position, the robotic arm tube picking system grabs the middle of the sample tube. If the abnormal sample tubes are in a reclining position, grab them from above the sample tube cover.

4. The method for selecting sample tubes and handling abnormal decisions based on vision according to claim 3 is characterized in that: The multimodal model is GPT-4 or Gemini 1.

5.

5. A vision-based sample tube selection and abnormal decision-making method system, characterized in that: include: A storage mechanism, wherein a freezing box is disposed on the top of the storage mechanism, and a plurality of sample tubes are disposed in the freezing box; A visual recognition component, the visual recognition component includes a visual recognition unit, a USB data cable and a computer, the USB data cable is used to transmit visual images between the visual recognition unit and the computer, and the visual recognition component is located above the tube selection area; A robotic arm tube picking system, the robotic arm tube picking system comprising a robotic arm and a tube picking claw, the tube picking claw being fixed on the robotic arm, and the tube picking claw being used to take out a sample tube from a storage mechanism; Also includes multimodal model software.

6. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which implements the method according to any one of claims 1 to 3 when executed by a processor.

Images