Unmanned laboratory target detection method and system

Abstract

The invention relates to the technical field of artificial intelligence and computer vision, discloses an unmanned laboratory target detection method and system, and aims to solve the problems of insufficient stability, poor adaptability, relatively high algorithm reasoning delay and the like in an automatic identification and positioning system of an existing unmanned laboratory. According to the method, the YOLO-World target detection framework is combined with the CLIP multi-mode semantic model, efficient target recognition without specific data set labeling is achieved, and high-precision three-dimensional coordinate extraction is achieved in combination with the depth point cloud and the SAM segmentation model. The method can be widely applied to automatic identification, positioning and operation tasks of the unmanned laboratory.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and more specifically, to a method and system for unmanned laboratory target detection based on zero-shot recognition and rapid localization, applicable to target recognition, localization and operation tasks in intelligent experimental environments. Background Technology

[0002] With the rapid development of artificial intelligence technology, scientific research experiments are gradually shifting from traditional manual operation to unmanned and intelligent methods. Unmanned laboratories, as an important direction for intelligent scientific research, significantly improve experimental efficiency, reduce labor costs, and enhance experimental repeatability by combining automated experimental equipment with intelligent algorithms.

[0003] However, in practical applications, the automatic identification and positioning system of unmanned laboratories still faces the following main problems: 1. The laboratory environment is complex and variable, and different lighting conditions and reflective surfaces lead to insufficient stability of the visual inspection system; 2. The experimental subjects are diverse in type and complex in form. Traditional supervised learning methods rely on a large amount of manually labeled data, which makes it difficult to adapt to the rapid changes in new categories. 3. Traditional detection algorithms have high inference latency, which cannot meet the real-time and high-throughput experimental requirements of automatic identification and positioning in unmanned laboratories.

[0004] Therefore, there is an urgent need for a detection method that can achieve high-precision target recognition and three-dimensional positioning under zero-sample conditions, so as to improve the automation and intelligence level of unmanned laboratories. Summary of the Invention

[0005] In view of the above problems, the purpose of this invention is to provide a detection method and system that can achieve high-precision target recognition and three-dimensional positioning under zero-sample conditions, so as to improve the automation and intelligence level of unmanned laboratories.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] In a first aspect, the present invention provides a method for target detection in an unmanned laboratory, comprising: Obtain a dataset corresponding to a preset task in the experimental scenario; wherein the dataset includes at least candidate regions, the number of candidate targets, and candidate targets; The candidate region is subjected to zero-shot semantic recognition through a preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate region. By using a preset SAM segmentation model and a deep point cloud fusion strategy, the candidate target is re-localized to determine the centroid coordinates, 3D bounding box, and principal axis direction of the candidate target.

[0008] In embodiments of the present invention, by combining the YOLO-World object detection framework with the CLIP multimodal semantic model, efficient object recognition without the need for specific dataset labeling is achieved, and high-precision 3D coordinate extraction is achieved by combining deep point cloud and SAM segmentation model.

[0009] In addition, an optional approach is that obtaining the dataset corresponding to the preset task of the experimental scenario includes: If the preset task of the experimental scenario does not have a corresponding dataset, then based on the preset visual acquisition module and depth information acquisition module, the color image data and depth data of the experimental environment are acquired respectively. The color image data is initially detected using a preset YOLO-World model to generate a set of candidate target boxes, and the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the set of candidate target boxes are obtained. Write the two-dimensional location coordinates, confidence scores, and depth data of each candidate target in the candidate target box set into the dataset.

[0010] Alternatively, the step of performing zero-shot semantic recognition on the candidate region using a preset CLIP multimodal semantic matching mechanism includes: The text description of the predefined experimental object is embedded into the semantic space using the CLIP text encoder to determine the semantic vector of the predefined experimental object. The visual features of the candidate targets in the candidate region are determined by an image encoder, so as to determine the semantic vector of the visual features; The semantic vectors of the predefined experimental objects and the semantic vectors of the visual features are calculated using cosine similarity. The target category of the candidate target is determined based on the principle of maximizing similarity.

[0011] In addition, an alternative approach is to further include, after determining the target category of each candidate target in the candidate region, writing the target category into the category label of the dataset.

[0012] Alternatively, an optional approach is to perform secondary localization of the candidate targets using a pre-defined SAM segmentation model and a deep point cloud fusion strategy, including: The candidate targets are segmented using a pre-defined SAM model to generate a target segmentation mask; The target point set of the candidate target is determined by projecting the target segmentation mask onto the aligned depth point cloud; Calculate the geometric centroid, enclosing volume, and principal axis direction of the target in the target point set.

[0013] In addition, an optional approach is to further analyze the geometric centroid, enclosing volume, and principal axis direction of the target using principal component analysis to obtain the principal direction vector of the target, thereby achieving three-dimensional attitude estimation of the candidate target.

[0014] In addition, an optional approach is to, after calculating the geometric centroid, bounding volume, and principal axis direction of the targets in the target point set, further include: writing the geometric centroid, bounding volume, and principal axis direction of the targets in the target point set into the dataset.

[0015] Furthermore, an optional approach is to include a capture and execution step and a closed-loop verification step after the target is located a second time; wherein, The grasping execution steps include: performing a grasping operation by the robotic arm based on a grasping trajectory generated by the determined centroid coordinates and spindle direction; The closed-loop verification step includes: if the robotic arm's grasping operation fails, it automatically returns to the dataset to reselect the target and repeat the verification, thus forming a closed-loop detection and operation process.

[0016] Secondly, the present invention provides an unmanned laboratory target detection system, comprising: A dataset acquisition unit is used to acquire a dataset corresponding to a preset task in the experimental scenario; wherein, the dataset includes at least a candidate region, a number of candidate targets, and candidate targets; The semantic recognition unit is used to perform zero-sample semantic recognition on the candidate region through a preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate region; The secondary localization unit is used to perform secondary localization on the candidate target by using a preset SAM segmentation model and a deep point cloud fusion strategy to determine the centroid coordinates, three-dimensional bounding box, and principal axis direction of the candidate target.

[0017] In addition, an optional embodiment is that the dataset acquisition unit further includes: The source data acquisition unit is used to acquire color image data and depth data of the experimental environment based on the preset visual acquisition module and depth information acquisition module when there is no corresponding dataset for the preset task in the experimental scenario. The source data preprocessing unit is used to perform preliminary detection on the color image data using a preset YOLO-World model to generate a set of candidate target boxes, and to obtain the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the set of candidate target boxes; The two-dimensional location coordinates, confidence scores, and depth data of each candidate target in the candidate target box set are written into the dataset.

[0018] In addition, an optional embodiment is that the semantic recognition unit further includes: A text embedding unit is used to embed the text description of a predefined experimental object into a semantic space using a CLIP text encoder, so as to determine the semantic vector of the predefined experimental object. An image encoding unit is used to determine the visual features of the candidate target in the candidate region through an image encoder, so as to determine the semantic vector of the visual features; The category determination unit is used to calculate the cosine similarity between the semantic vector of the predefined experimental object and the semantic vector of the visual feature, and determine the target category of the candidate target according to the principle of maximum similarity.

[0019] In addition, alternative solutions include: The gripping execution unit is used to perform gripping operations by the robotic arm based on the gripping trajectory generated by the determined center of mass coordinates and the spindle direction; The closed-loop verification unit is used to automatically return to the dataset to reselect the target and repeat the verification if the robotic arm's grasping operation fails, so as to form a closed-loop detection and operation process.

[0020] Thirdly, the present invention provides an electronic device, the electronic device including a memory, a processor, and an unmanned laboratory target detection program stored in the memory and executable on the processor, wherein the unmanned laboratory target detection program, when executed by the processor, implements the unmanned laboratory target detection method as described above.

[0021] As can be seen from the above technical solution, the unmanned laboratory target detection method and system provided by the present invention have the following advantages compared with the existing ones: 1) No labeled data required: This invention achieves target detection under zero-sample conditions by combining the YOLO-World detection framework with the CLIP multimodal semantic model, which significantly reduces the cost of manual data labeling and improves the adaptability of the system; 2) Efficient recognition and fast reasoning: This invention can achieve efficient target recognition and real-time reasoning in complex experimental environments, meeting the needs of unmanned laboratories for high-throughput and low-latency processing; 3) High-precision 3D positioning: By introducing the SAM segmentation model and point cloud fusion strategy, high-precision target 3D centroid and attitude estimation are obtained to ensure the stability and accuracy of the robotic arm when performing grasping and manipulation tasks; 4) Strong system robustness: Under complex conditions such as changes in lighting, occlusion interference, and reflection, the present invention can still maintain stable recognition and positioning performance and adapt to diverse experimental scenarios; 5) Excellent scalability: The method of this invention is applicable to different types of experimental platforms, experimental instruments and equipment, and can support multi-task collaboration and intelligent operation in unmanned laboratories, with good versatility and expansion potential.

[0022] To achieve the foregoing and related objectives, one or more aspects of the invention include the features that will be described in detail below. The following description and accompanying drawings illustrate certain exemplary aspects of the invention. However, these aspects indicate only a few of the various ways in which the principles of the invention can be used. Furthermore, the invention is intended to encompass all such aspects and their equivalents. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the unmanned laboratory target detection method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the experimental platform structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the experimental process framework of an embodiment of the present invention; Figure 4 This is a schematic diagram of the global perception generation process in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the zero-sample semantic recognition process according to an embodiment of the present invention. Figure 6 This is a flowchart illustrating the secondary precise positioning process according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the capture, execution, and closed-loop verification process according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the framework structure of an unmanned laboratory target detection system according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the electronic device structure framework according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0026] All patent and non-patent literature publications cited in this invention are incorporated herein by reference.

[0027] The terms “comprising,” “including,” “containing,” “covering,” “having,” “with,” or any other variations thereof, as used in this invention, are intended to cover non-exclusive inclusion. For example, a process, method, article of manufacture, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to the process, method, article of manufacture, or apparatus. Furthermore, unless expressly stated otherwise, “or” means inclusive “or” rather than exclusive “or.” For example, condition A or B satisfies any of the following: A is real (or exists) and B is fictitious (or does not exist); A is fictitious (or does not exist) and B is real (or exists); and both A and B are real (or exist). The phrase “one or more” is intended to cover non-exclusive inclusion. For example, one or more A, B, and C means any of the following: A alone, B alone, C alone, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C.

[0028] Additionally, the terms "an" or "a" are used to describe the elements and components described herein. This is done solely for convenience and to provide a general meaning regarding the scope of the invention. This description should be understood to include one or at least one, one or at least one, and the singular includes the plural unless explicitly stated otherwise.

[0029] To address the shortcomings of existing automatic identification and localization schemes in unmanned laboratories, such as insufficient stability, poor versatility and scalability, and inadequate real-time performance, this invention provides a target detection method and system for unmanned laboratories. By combining the YOLO-World target detection framework with the CLIP multimodal semantic model, it achieves efficient target recognition without the need for specific dataset annotation, and further combines deep point cloud and SAM segmentation models to achieve high-precision 3D coordinate extraction. This invention significantly outperforms existing technologies in both recognition accuracy and processing speed, and can be widely applied to automatic identification, localization, and operation tasks in unmanned laboratories.

[0030] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.

[0031] CLIP (contrastive language-image pre-training) is a multimodal vision and language model that overcomes the problem of a fixed number of object categories when training computer vision models by creating strategies for visual and language representations. It can be used to create highly specific and high-performance classifiers without any training data.

[0032] The YOLO-World model is an advanced real-time open-vocabulary detection model based on the YOLOv8 framework. Combining visual language modeling and pre-training on large-scale datasets, it can efficiently identify various objects in zero-shot scenes. It achieves high performance with low computational requirements and is suitable for a variety of visual tasks. YOLO-World addresses the dependence on predefined categories by introducing open-vocabulary detection. It supports real-time detection using a "hint-then-detect" strategy, improving efficiency through an offline vocabulary. The YOLO-World model outperforms the MTETR and GLIP series in both speed and performance, especially excelling on a single NVIDIA V100 GPU.

[0033] SAM (Segment Anything Model) is a computer vision model for image segmentation. This model can detect, segment, and track objects in images and videos using text, examples, and visual cues. SAM supports open-vocabulary phrase input, possesses powerful cross-modal interaction capabilities, and can correct segmentation results in real time.

[0034] The technical solution of the present invention will be further described in detail below with reference to several preferred embodiments and accompanying drawings. This embodiment is implemented on the premise of the technical solution of the invention, and provides detailed implementation methods and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.

[0035] Figure 1 The flowchart of an unmanned laboratory target detection method according to an embodiment of the present invention is shown. Figure 2 An experimental platform structure according to an embodiment of the present invention is shown. Figure 3 An experimental procedure framework according to an embodiment of the present invention is shown. Figures 1 to 3 As shown in the figure, the unmanned laboratory target detection method provided by the present invention is based on zero-shot recognition and rapid localization, and mainly includes the following steps: S100: Obtain the dataset corresponding to the preset task of the experimental scenario; wherein the dataset includes at least candidate regions, the number of candidate targets, and candidate targets.

[0036] S200: Zero-shot semantic recognition is performed on the candidate region through a preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate region; S300: By using a preset SAM segmentation model and a deep point cloud fusion strategy, the candidate target is repositioned to determine the centroid coordinates, 3D bounding box, and principal axis direction of the candidate target.

[0037] Specifically, in step S100, if a dataset corresponding to the preset task of the experimental scenario exists in the database, the required data can be extracted based on the dataset for subsequent zero-sample semantic recognition (step S200) and secondary precise localization (step S300). However, if a dataset corresponding to the preset task of the experimental scenario does not exist in the database, or if the dataset corresponding to the preset task of the experimental scenario lacks corresponding data, then data collection and processing need to be performed first through global perception generation to obtain the corresponding dataset.

[0038] Specifically, as an example, if the preset task of the experimental scenario does not have a corresponding dataset, step S100 further includes: S110: Based on the preset visual acquisition module and depth information acquisition module, acquire color image data and depth data of the experimental environment respectively; S120: Use the preset YOLO-World model to perform preliminary detection on the color image data to generate a set of candidate target boxes, and obtain the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the set of candidate target boxes; S130: Write the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the candidate target box set into the dataset.

[0039] More specifically, Figure 4 The process of generating global awareness according to an embodiment of the present invention is illustrated. For example... Figure 4 As shown, in this global perception generation process, firstly, global visual and depth information is acquired for the experimental scene task number n, and then candidate target boxes b are generated using the YOLO-World model. i Obtain the two-dimensional position (X) of each target j. j Y j ), confidence level P(b) j ) and depth data Z nj .

[0040] Figure 4 The global awareness generation process shown includes the following steps: (1.1) Determine the current task number n and the corresponding data table datasheet-n for the experimental scenario. For example, if the current task number n = 0103, then the data table corresponding to this task number is datasheet-0103. Read the completed target num from the current datasheet-n; in the following implementation descriptions, task number n = 0103 will be used as an example. Here, the completed target num refers to the number of objects that have completed target detection stored in the current data table. The completed target is the target with all of the following fields: "Task Number", "Number of Candidate Targets", "Candidate Target Number", "Candidate Region", "Acquisition Time", "2D Position Coordinates", "Depth Data", "Confidence Score", "Category Label", "Geometric Centroid Coordinates", "Bounding Box Volume", "Main Axis Direction", and "Grabbing Result".

[0041] (1.1.1) If datasheet-0103 does not exist, create datasheet-0103 and set the target num=0; datasheet-0103 shall contain at least the following fields: "Task Number", "Candidate Area", "Number of Candidate Targets", and "Candidate Targets"; (1.1.2) If datasheet-0103 exists, read the current completed target num and set the current candidate target j = num + 1; for example, if the completed target num = 5 in datasheet-0103, then the current candidate target j; j = num + 1 = 6.

[0042] (1.2) Establish a vision acquisition module and a depth information acquisition module, start the image acquisition thread Thread_img, and acquire color image data I of the experimental environment respectively. n With depth data Z n ; (1.3) Start the YOLO-World detection model and process the input image I n Perform preliminary detection to obtain the number of candidate targets i, and generate a set of candidate target boxes B={b1,b2,…,b i For example, when candidate i=4 is obtained, the generated candidate bounding box set B={b1,b2,…,b4}. Segmenting depth data Z 0103 Calculate the two-dimensional position coordinates (X6, Y6), confidence score P(b6), and depth data Z for each candidate target j=6. 01036 In a specific embodiment of the present invention, the calculated two-dimensional position coordinates (X6, Y6) = (352.11, 178.32), confidence score P(b6) = 0.86, and depth data Z are compared with the calculated two-dimensional position coordinates (X6, Y6) = (352.11, 178.32), confidence score P(b6) = 0.86, and depth data Z. 01036 =496.72.

[0043] (1.4) The two-dimensional position coordinates (X6, Y6) and confidence score P(b6) of the candidate region b6 of the candidate target 6 are compared with the depth data Z by the control system. 01036 Write to datasheet-0103 and stop the image acquisition thread Thread_img.

[0044] After obtaining the dataset corresponding to the preset task of the experimental scenario, step S200 can be entered, and zero-shot semantic recognition of the candidate regions in the dataset can be performed through the preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate regions.

[0045] Figure 5 The flowchart of zero-sample semantic recognition according to an embodiment of the present invention is shown.

[0046] like Figure 5 As shown, in the process of zero-shot semantic recognition, the CLIP multimodal semantic matching mechanism is introduced for candidate regions. As an example, text descriptions of 30 experimental objects can be predefined, and the text can be embedded into the semantic space by the CLIP text encoder. Cosine similarity is calculated between the text and the visual features of the image region. Then, the target category is determined according to the principle of maximum similarity, thus achieving zero-shot recognition without dataset labeling.

[0047] Figure 5 The zero-shot semantic recognition process shown includes the following steps: (2.1) The workstation reads candidate region b6 from datasheet-0103 and establishes the CLIP multimodal semantic recognition module. The system predefines the semantic description text set T={t1,t2,…,t...} for the experimental objects. 30 The categories include "sample blocks", "beakers", "heating plates", "metal blocks", "measuring cylinders", "pipettes", "sintering cups", "ball mills", "conical flasks", "separating funnels", "glass rods", "volume flasks", and "centrifuge tubes". (2.2) Start CLIP text encoder and image encoder to generate semantic vector E for candidate target 6 respectively. 6t With E 6l Calculate the cosine similarity between the two, and construct a similarity matrix S(4,6)=cos(E 6t E 6l ); (2.3) The control system determines the category label R = "beaker" of the candidate region based on the principle of maximum similarity, writes the category label R = "beaker" into datasheet-0103, and closes the CLIP multimodal semantic recognition module.

[0048] After determining the target category of each candidate target in the candidate region, the determined target category can be written into the category label of the dataset.

[0049] The aforementioned zero-sample semantic recognition enables target detection under zero-sample conditions, significantly reducing the cost of manual data annotation and improving the system's adaptability. Furthermore, it enables efficient target recognition and real-time reasoning in complex experimental environments, meeting the needs of unmanned laboratories for high-throughput and low-latency processing.

[0050] Since the above zero-shot semantic recognition is based on two-dimensional recognition of image information, its results lack geometric depth information. Therefore, after completing the zero-shot semantic recognition, we can proceed to step S300 for secondary localization. By introducing the SAM segmentation model and the deep point cloud fusion strategy, we first use the SAM model to generate a high-precision target segmentation mask; then we project the mask onto the aligned depth point cloud to extract the target point set; then we calculate the geometric centroid, bounding volume, and principal axis direction of all targets in the target point set; finally, we use the principal component analysis method to obtain the principal direction vector and realize the three-dimensional pose estimation of the target.

[0051] Specifically, as an example, Figure 6 The process of secondary positioning according to an embodiment of the present invention is illustrated. For example... Figure 6 As shown, the specific process of secondary positioning includes the following steps: (3.1) Establish an independent three-dimensional positioning thread Thread_loc; (3.2) Start the SAM segmentation model, generate a target mask M6 for candidate target 6, and compare the mask M6 with the depth point cloud Z. 01036 Alignment and fusion, extract the target point set P6={p1,p2,…,p 3865}; (3.3) Calculate the geometric centroid coordinates C6, bounding box volume V6 and principal axis direction U6 based on the point set P6, and obtain the three-dimensional attitude parameters of the target through principal component analysis; (3.4) The control system writes the extracted C6, V6, and U6 positioning information into datasheet-0103 for generating the robotic arm's grasping trajectory and stops the three-dimensional positioning thread Thread_loc.

[0052] In the secondary positioning process, this invention introduces the SAM segmentation model and point cloud fusion strategy to obtain high-precision target three-dimensional centroid and attitude estimation, ensuring the stability and accuracy of the robotic arm when performing grasping and manipulation tasks.

[0053] After secondary positioning of the target, target detection in the unmanned laboratory is achieved, and then the robotic arm can be controlled to perform subsequent automatic grasping operations according to the generated grasping trajectory.

[0054] Specifically, the robotic arm executes the grasping task based on the grasping trajectory generated by the extracted target centroid and the main axis direction, and performs automatic grasping operations.

[0055] In a preferred embodiment of the present invention, if the grasping task fails, the system can automatically return to the candidate dataset to reselect the target and repeat the verification to form a closed-loop detection and operation process, thereby improving the automatic control capability of the unmanned laboratory.

[0056] Therefore, in a specific embodiment of the present invention, after the secondary positioning is completed, the method further includes a capture execution step and a closed-loop verification step. Figure 7 The flow of crawling execution and closed-loop verification according to an embodiment of the present invention is illustrated. Figure 7 As shown, the steps for crawling execution and closed-loop verification provided in this embodiment include: (4.1) The control system reads the centroid coordinates C6 and spindle direction U6 of candidate target 6 in datasheet-0103 to plan the grasping trajectory and sends it to the robotic arm for execution.

[0057] (4.2) Start the grasping verification submodule to determine whether the robotic arm grasped successfully; (4.2.1) If candidate target 6 is successfully captured, the success record is stored in the capture result of datasheet-0103 and the process jumps to (4.3); otherwise, the failure record is stored in the capture result of datasheet-0103, candidate target j=j+1=7, the optimal target is re-selected and the process returns to step (4.1). (4.3) The verification of the control system is completed.

[0058] Corresponding to the above-described unmanned laboratory target detection method, the present invention also provides an unmanned laboratory target detection system, which utilizes the unmanned laboratory target detection method described above to perform target detection in unmanned laboratories.

[0059] Figure 8 The logical framework structure of an unmanned laboratory target detection system 800 according to an embodiment of the present invention is shown. Figure 8 As shown, based on the functions implemented, the unmanned laboratory target detection system 800 provided in this embodiment mainly includes a dataset acquisition unit 810, a semantic recognition unit 820, and a secondary positioning unit 830.

[0060] The unit of this invention can also be called a module, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0061] Furthermore, in one possible implementation of the present invention, the functions of each module / unit are as follows: The dataset acquisition unit 810 is used to acquire a dataset corresponding to a preset task in the experimental scenario; wherein the dataset includes at least a candidate region, a number of candidate targets, and candidate targets.

[0062] In a specific embodiment of the present invention, the dataset acquisition unit 810 further includes: The source data acquisition unit 811 is used to acquire color image data and depth data of the experimental environment based on the preset visual acquisition module and depth information acquisition module when there is no corresponding dataset for the preset task in the experimental scenario. The source data preprocessing unit 812 is used to perform preliminary detection on the color image data using a preset YOLO-World model to generate a set of candidate target boxes, and to obtain the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the set of candidate target boxes; The two-dimensional location coordinates, confidence scores, and depth data of each candidate target in the candidate target box set are written into the dataset.

[0063] The semantic recognition unit 820 is used to perform zero-sample semantic recognition on the candidate region through a preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate region.

[0064] In one possible implementation of the present invention, the semantic recognition unit 820 further includes: The text embedding unit 821 is used to embed the text description of the predefined experimental object into the semantic space through the CLIP text encoder to determine the semantic vector of the predefined experimental object. Image encoding unit 822 is used to determine the visual features of the candidate target in the candidate region by an image encoder, so as to determine the semantic vector of the visual features; The category determination unit 823 is used to perform cosine similarity calculation on the semantic vector of the predefined experimental object and the semantic vector of the visual feature, and determine the target category of the candidate target according to the principle of maximum similarity.

[0065] The secondary localization unit 830 is used to perform secondary localization on the candidate target by using a preset SAM segmentation model and a deep point cloud fusion strategy to determine the centroid coordinates, three-dimensional bounding box, and principal axis direction of the candidate target.

[0066] In one possible implementation of the present invention, the unmanned laboratory target detection system 800 may further include a grasping execution unit 840 and a closed-loop verification unit 850.

[0067] The gripping execution unit 840 is used to perform gripping operations by the robotic arm based on the gripping trajectory generated by the determined center of mass coordinates and the main axis direction. The closed-loop verification unit 850 is used to automatically return to the dataset to reselect the target and repeat the verification if the robotic arm's grasping operation fails, so as to form a closed-loop detection and operation process.

[0068] The aforementioned unmanned laboratory target detection system is an implementation device corresponding to the aforementioned unmanned laboratory target detection method. Its specific execution steps can be referred to in the specific embodiments of the aforementioned unmanned laboratory target detection method, and will not be described in detail here.

[0069] like Figure 9 As shown, the present invention also provides an electronic device 1 for implementing a target detection method in an unmanned laboratory.

[0070] The electronic device 1 may include a processor 10, a memory 11, and a bus. It may also include a computer program, such as an unmanned laboratory target detection program 12, stored in the memory 11 and executable on the processor 10. The memory 11 may include both internal storage units of the aforementioned unmanned laboratory target detection system and external storage devices. The memory 11 can be used not only to store application software and various types of data, such as the code of the unmanned laboratory target detection program, but also to temporarily store data that has been output or will be output.

[0071] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of an unmanned laboratory target detection program, but also to temporarily store data that has been output or will be output.

[0072] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., unmanned laboratory target detection programs) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0073] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0074] Figure 9 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 9 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0075] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management system, thereby enabling functions such as charging management, discharging management, and power consumption management through the power management system. The power supply may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0076] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0077] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0078] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0079] The unmanned laboratory target detection program 12 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can realize the unmanned laboratory target detection method as described above.

[0080] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here. Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0081] This invention also provides a computer-readable storage medium, which may be non-volatile or volatile, and stores a computer program that, when executed by a processor, implements the unmanned laboratory target detection method described above.

[0082] Specifically, the specific implementation method of the computer program when executed by the processor can be referred to the description of the relevant steps in the unmanned laboratory target detection method of the embodiment, and will not be repeated here.

[0083] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0084] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0086] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim may also be implemented by a single unit or system through software or hardware.

[0087] The unmanned laboratory target detection method and system according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that the above are merely preferred embodiments of the present invention and do not limit the scope of protection of the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any changes, modifications, substitutions, integrations, and parameter changes to these embodiments within the spirit and principles of the present invention, achieved through conventional substitutions or by achieving the same function without departing from the principles and spirit of the present invention, fall within the scope of protection of the present invention.

Claims

1. A method for target detection in an unmanned laboratory, characterized in that, include: Obtain a dataset corresponding to a preset task in the experimental scenario; wherein the dataset includes at least candidate regions, the number of candidate targets, and candidate targets; The candidate region is subjected to zero-shot semantic recognition through a preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate region. By using a preset SAM segmentation model and a deep point cloud fusion strategy, the candidate target is re-localized to determine the centroid coordinates, 3D bounding box, and principal axis direction of the candidate target.

2. The unmanned laboratory target detection method according to claim 1, characterized in that, The acquisition of the dataset corresponding to the preset task of the experimental scenario includes: If the preset task of the experimental scenario does not have a corresponding dataset, then based on the preset visual acquisition module and depth information acquisition module, the color image data and depth data of the experimental environment are acquired respectively. The color image data is initially detected using a preset YOLO-World model to generate a set of candidate target boxes, and the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the set of candidate target boxes are obtained. Write the two-dimensional location coordinates, confidence scores, and depth data of each candidate target in the candidate target box set into the dataset.

3. The unmanned laboratory target detection method according to claim 1 or 2, characterized in that, The step of performing zero-shot semantic recognition on the candidate region using a preset CLIP multimodal semantic matching mechanism includes: The text description of the predefined experimental object is embedded into the semantic space using the CLIP text encoder to determine the semantic vector of the predefined experimental object. The visual features of the candidate targets in the candidate region are determined by an image encoder, so as to determine the semantic vector of the visual features; The semantic vectors of the predefined experimental objects and the semantic vectors of the visual features are calculated using cosine similarity. The target category of the candidate target is determined based on the principle of maximizing similarity.

4. The unmanned laboratory target detection method according to claim 3, characterized in that, After determining the target category of each candidate target in the candidate region, the process further includes: Write the target category into the category label of the dataset.

5. The unmanned laboratory target detection method according to claim 3, characterized in that, The candidate targets are then re-localized using a pre-defined SAM segmentation model and a deep point cloud fusion strategy, including: The candidate targets are segmented using a pre-defined SAM model to generate a target segmentation mask; The target point set of the candidate target is determined by projecting the target segmentation mask onto the aligned depth point cloud; Calculate the geometric centroid, enclosing volume, and principal axis direction of the target in the target point set.

6. The unmanned laboratory target detection method according to claim 5, characterized in that, Also includes: Principal component analysis is used to analyze the geometric centroid, bounding volume, and principal axis direction of the target to obtain the principal direction vector of the target, thereby realizing the three-dimensional attitude estimation of the candidate target.

7. The unmanned laboratory target detection method according to claim 5, characterized in that, After calculating the geometric centroid, bounding volume, and principal axis direction of the targets in the target point set, the method further includes: writing the geometric centroid, bounding volume, and principal axis direction of the targets in the target point set into the dataset.

8. The unmanned laboratory target detection method according to claim 1, characterized in that, After secondary localization of the target, the process also includes a capture and execution step and a closed-loop verification step; wherein, The grasping execution steps include: performing a grasping operation by the robotic arm based on a grasping trajectory generated by the determined centroid coordinates and spindle direction; The closed-loop verification step includes: if the robotic arm's grasping operation fails, it automatically returns to the dataset to reselect the target and repeat the verification, thus forming a closed-loop detection and operation process.

9. An unmanned laboratory target detection system, characterized in that, include: The dataset acquisition unit is used to acquire a dataset corresponding to a preset task in the experimental scenario; wherein, the dataset includes at least the subsequent target, the number of candidate targets, and the candidate region; The semantic recognition unit is used to perform zero-sample semantic recognition on the candidate region through a preset CLIP multimodal semantic matching mechanism to determine the target category of each candidate target in the candidate region; The secondary localization unit is used to perform secondary localization on the candidate target by using a preset SAM segmentation model and a deep point cloud fusion strategy to determine the centroid coordinates, three-dimensional bounding box, and principal axis direction of the candidate target.

10. The unmanned laboratory target detection system according to claim 9, characterized in that, The dataset acquisition unit further includes: The source data acquisition unit is used to acquire color image data and depth data of the experimental environment based on the preset visual acquisition module and depth information acquisition module when there is no corresponding dataset for the preset task in the experimental scenario. The source data preprocessing unit is used to perform preliminary detection on the color image data using a preset YOLO-World model to generate a set of candidate target boxes, and to obtain the two-dimensional position coordinates, confidence scores and depth data of each candidate target in the set of candidate target boxes; The two-dimensional location coordinates, confidence scores, and depth data of each candidate target in the candidate target box set are written into the dataset.

11. The unmanned laboratory target detection system according to claim 10, characterized in that, The semantic recognition unit further includes: A text embedding unit is used to embed the text description of a predefined experimental object into a semantic space using a CLIP text encoder, so as to determine the semantic vector of the predefined experimental object. An image encoding unit is used to determine the visual features of the candidate target in the candidate region through an image encoder, so as to determine the semantic vector of the visual features; The category determination unit is used to calculate the cosine similarity between the semantic vector of the predefined experimental object and the semantic vector of the visual feature, and determine the target category of the candidate target according to the principle of maximum similarity.

12. The unmanned laboratory target detection system according to claim 10, characterized in that, Also includes: The gripping execution unit is used to perform gripping operations by the robotic arm based on the gripping trajectory generated by the determined center of mass coordinates and the spindle direction; The closed-loop verification unit is used to automatically return to the dataset to reselect the target and repeat the verification if the robotic arm's grasping operation fails, so as to form a closed-loop detection and operation process.

13. An electronic device, characterized in that, The electronic device includes a memory, a processor, and an unmanned laboratory target detection program stored in the memory and executable on the processor. When executed by the processor, the unmanned laboratory target detection program implements the unmanned laboratory target detection method as described in claims 1 to 8.

Images