A method, apparatus and device for layout detection of a sample disc

Abstract

The application provides a sample disc layout detection method, device and equipment, the method comprising: determining sample bottle positions and idle hole position positions based on an acquired sample disc image; determining a matrix template corresponding to the sample disc image, the matrix template comprising M*N position points; clustering column coordinates of the sample bottle positions and the idle hole position positions to obtain M column center coordinates, clustering row coordinates of the sample bottle positions and the idle hole position positions to obtain N row center coordinates, the M column center coordinates and the N row center coordinates comprising the M*N position points; a predicted category of a position point matched with a sample bottle position is an occupied category, and a predicted category of a position point matched with an idle hole position position is an idle category; determining a sample disc layout based on predicted categories of the position points in the matrix template, the sample disc layout comprising hole types corresponding to M*N holes of the sample disc respectively. Through the technical solution of the application, accurate judgment of the sample disc layout can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, in particular to a sample disc layout detection method, device and equipment. BACKGROUND

[0002] A liquid chromatograph is an instrument for separating and analyzing components in a mixture. Its working principle is: the sample to be tested is separated in the chromatographic column by the liquid mobile phase, and then the components in the mixture are analyzed by the detector. It is widely used in the fields of medical detection, food detection, etc. The liquid chromatograph can include a sample injector, which includes a sample disc, a sample holder, a sample bottle, an idle hole position, an occupied hole position, a sample needle, etc.

[0003] The sample injector is a key component of the liquid chromatograph, which is used to accurately introduce the sample to be tested into the chromatographic column for separation. The sample disc is one of the core components of the sample injector of the liquid chromatograph, which is used to store the sample to be tested. A plurality of (such as tens to hundreds) sample bottles can be placed on the sample disc to store the sample to be tested. The sample holder is one of the core components of the sample injector of the liquid chromatograph, which is used to place the sample disc and drive the sample disc to rotate or move by the motor. The sample bottle is a container for placing the sample to be tested, which can be placed in the hole position of the sample disc. The idle hole position refers to the hole position of the sample disc which does not place the sample bottle, and the occupied hole position refers to the hole position of the sample disc which has placed the sample bottle. The sample needle is one of the core components of the sample injector of the liquid chromatograph, which is responsible for accurately extracting or injecting the sample to be tested (such as liquid sample) from the sample bottle.

[0004] During the working process of the liquid chromatograph, the sample disc layout needs to be obtained, which includes the hole position type of each hole position of the sample disc, indicating whether the hole position is an occupied hole position or an idle hole position, and then subsequent processing is performed based on the sample disc layout. For example, the sample to be tested can be extracted from the sample bottle of the occupied hole position by the sample needle, and the sample to be tested does not need to be extracted from the idle hole position by the sample needle.

[0005] However, how to obtain the sample disc layout has not been effectively implemented in the related art, that is, it is not possible to accurately know which hole positions of the sample disc are occupied hole positions and which hole positions are idle hole positions. SUMMARY

[0006] The present application provides a sample disc layout detection method, which comprises:

[0007] determining the sample bottle position and the idle hole position based on the obtained sample disc image; wherein the sample injector comprises a sample disc, the sample disc comprises a plurality of hole positions, the sample bottle position is the position of the sample bottle placed in the occupied hole position, and the idle hole position is the position of the idle hole position which does not place the sample bottle;

[0008] determine a matrix template corresponding to the sample disc image, the matrix template comprising M*N position points, M representing a number of transverse well positions of the sample disc, and N representing a number of longitudinal well positions of the sample disc; wherein column coordinates of the sample bottle positions and the idle well position are clustered to obtain M column center coordinates, and row coordinates of the sample bottle positions and the idle well position are clustered to obtain N row center coordinates, the M column center coordinates and the N row center coordinates forming the M*N position points; wherein a predicted category of a position point matched with the sample bottle position is an occupied category, and a predicted category of a position point matched with the idle well position is an idle category;

[0009] determine a sample disc layout based on the predicted categories of the position points in the matrix template, the sample disc layout comprising well types corresponding to the M*N well positions of the sample disc.

[0010] The application provides a layout detection device of a sample disc, the device comprising:

[0011] a determination module configured to determine sample bottle positions and idle well positions based on the obtained sample disc image; wherein the sample disc comprises a plurality of well positions, the sample bottle position is a position of a sample bottle placed in an occupied well position, and the idle well position is a position of an idle well position without a sample bottle placed therein;

[0012] a processing module configured to determine a matrix template corresponding to the sample disc image, the matrix template comprising M*N position points, M representing a number of transverse well positions of the sample disc, and N representing a number of longitudinal well positions of the sample disc; wherein column coordinates of the sample bottle positions and the idle well position are clustered to obtain M column center coordinates, and row coordinates of the sample bottle positions and the idle well position are clustered to obtain N row center coordinates, the M column center coordinates and the N row center coordinates forming the M*N position points; wherein a predicted category of a position point matched with the sample bottle position is an occupied category, and a predicted category of a position point matched with the idle well position is an idle category;

[0013] a detection module configured to determine a sample disc layout based on the predicted categories of the position points in the matrix template, the sample disc layout comprising well types corresponding to the M*N well positions of the sample disc.

[0014] The application provides an electronic device, comprising a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by the processor; the processor is configured to execute the machine executable instructions to implement the layout detection method of the sample disc of the above examples of the application.

[0015] The application provides a computer program product, the computer program product comprising a computer program, the computer program being executed by a processor to implement the layout detection method of the sample disc of the above examples of the application.

[0016] The application provides a machine readable storage medium storing machine executable instructions capable of being executed by a processor; wherein the processor is configured to execute the machine executable instructions to implement the sample disc layout detection method of the above examples of the application.

[0017] As can be seen from the above technical solutions, in the embodiments of the application, the sample bottle positions and the free well position positions can be determined based on the sample disc image, the matrix template can be determined based on the sample bottle positions and the free well position positions, the matrix template comprises a plurality of position points, the predicted category of the position point matched with the sample bottle position is the occupied category, and the predicted category of the position point matched with the free well position position is the free category. In this way, the sample disc layout can be determined based on the predicted categories of the position points in the matrix template, so that it is known which wells of the sample disc are occupied wells and which wells are free wells, and accurate judgment of the sample disc layout can be realized. The automatic detection of the sample disc layout in the label-free scene can be realized without relying on additional physical sensors or pre-arranged labels, and accurate judgment of the sample disc layout can be realized only by analyzing the collected sample disc image.

[0018] By determining the matrix template, the matrix templates of different perspectives are matched and information is fused, which can effectively reduce the missed detection rate and effectively overcome the difference in missed detection caused by single-perspective occlusion, light changes and the like. Even if there is missed detection in some perspectives, it can be completed through the information of other perspectives, which significantly improves the accuracy of identifying each well type. The system robustness can be enhanced, the matrix template is not sensitive to a small amount of missing detection, as long as the overall row and column structure is not damaged, the matching process can be stably performed, so that the single detection fluctuation has strong fault tolerance capability, and the overall stability is high. The processing efficiency can be improved, the matching process of the matrix template reduces the algorithm complexity of multi-perspective correlation, and can quickly and accurately integrate massive perspective information. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the sample disc layout detection method in an embodiment of the application;

[0020] Figure 2 is a structural schematic diagram of a liquid chromatograph in an embodiment of the application;

[0021] Figure 3A is a layout schematic diagram of a sample disc and a sample rack in an embodiment of the application;

[0022] Figure 3B is a layout schematic diagram of a sample disc in an embodiment of the application;

[0023] Figure 4is a schematic diagram of a layout detection method of a sample disc in an embodiment of the present application;

[0024] Figure 5A is a distribution diagram of the bottle mouth and the idle hole position in the horizontal space in the correct detection in the present application;

[0025] Figure 5B is a distribution diagram of the bottle mouth and the idle hole position in the horizontal space in the error detection in the present application;

[0026] Figure 6 is a schematic diagram of a matrix template of the bottle mouth and the idle hole position in an embodiment of the present application;

[0027] Figure 7 is a structural schematic diagram of a layout detection device of a sample disc in an embodiment of the present application;

[0028] Figure 8 is a hardware structure diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] A layout detection method of a sample disc is provided in the embodiments of the present application, which can be applied to an electronic device, as shown in Figure 1 The method can include the following steps:

[0030] In step 101, the positions of sample bottles and the positions of idle hole positions are determined based on the acquired sample disc image; wherein the sample disc can include a plurality of hole positions, the position of a sample bottle is the position of a sample bottle placed in a hole position, and the position of an idle hole position is the position of an idle hole position without a sample bottle.

[0031] In step 102, a matrix template corresponding to the sample disc image is determined, the matrix template includes M*N position points, M represents the number of horizontal hole positions of the sample disc, and N represents the number of vertical hole positions of the sample disc; wherein the column coordinates of the positions of the sample bottles and the positions of the idle hole positions are clustered to obtain M column center coordinates, and the row coordinates of the positions of the sample bottles and the positions of the idle hole positions are clustered to obtain N row center coordinates, the M column center coordinates and the N row center coordinates form the M*N position points; wherein the predicted category of the position point matched with the position of the sample bottle is the occupied category, and the predicted category of the position point matched with the position of the idle hole position is the idle category.

[0032] For the position point of the matrix template, if the position point is not matched with the position of the sample bottle, and the position point is not matched with the position of the idle hole position, the predicted category of the position point is the missing category, indicating that the predicted category of the position point is not detected, that is, the predicted category of the position point of the matrix template is allowed to be the missing category.

[0033] In step 103, the sample disc layout is determined based on the predicted categories of the position points in the matrix template. The sample disc layout can include the types of the M*N wells of the sample disc.

[0034] For example, the sample disc image can include K sample disc images under K perspectives, and the matrix template can include K matrix templates under K perspectives, where K is a positive integer. Based on this, determining the sample disc layout based on the predicted categories of the position points in the matrix template can include, but is not limited to: for any well of the sample disc, determining the predicted categories of the corresponding position points in the K matrix templates, and counting the first number of occupied categories and the second number of free categories. The type of the well is determined based on the first number and the second number. If the ratio of the first number to the total number is greater than a preset confidence threshold, the type of the well is the occupied type. If the ratio of the second number to the total number is greater than the preset confidence threshold, the type of the well is the free type. Based on the types of the M*N wells of the sample disc, the sample disc layout is determined.

[0035] For example, determining the sample bottle position and the free well position based on the obtained sample disc image can include, but is not limited to: inputting the sample disc image into a detection model to obtain a first detection box of the sample bottle and a second detection box of the free well, determining a first center point of the first detection box and a second center point of the second detection box; mapping the first center point to the sample bottle position in a reference coordinate system based on a first height between the sample bottle and the sample rack; and mapping the second center point to the free well position in the reference coordinate system based on a second height between the free well and the sample rack. The first height and the second height can be determined based on the type of the sample disc. The sample rack can be included in the sample injector, and the sample disc is placed on the sample rack. The origin of the reference coordinate system can be the rotation center of the sample rack. The X-axis and the Y-axis of the reference coordinate system can be located on the horizontal plane of the sample rack and orthogonal to each other. The Z-axis of the reference coordinate system can be perpendicular to the horizontal plane of the sample rack.

[0036] In one possible implementation, the sample bottle position or the free well position can be determined by using the following formula: ; wherein, represents the first center point, represents the sample bottle position, represents the first height; or, represents the second center point, represents the free well position, represents the second height; wherein, represents the rotation component in the camera extrinsic matrix; represents the camera intrinsic matrix; represents the component of the direction vector of the ray from the camera optical center in the Z-axis of the reference coordinate system; denotes a coordinate of a camera optical center in a reference coordinate system, , denotes a translation component in a camera extrinsic matrix; denotes a rotation component in a camera extrinsic matrix. denotes a component of a Z-axis of a reference coordinate system.

[0037] Exemplarily, before determining the matrix template corresponding to the sample disc image, for any sample bottle position, the sample bottle position is reversely rotated based on a reference rotation angle to obtain a rotated sample bottle position, and the rotated sample bottle position is used to determine the matrix template; for any idle hole position, the idle hole position is reversely rotated based on the reference rotation angle to obtain a rotated idle hole position, and the rotated idle hole position is used to determine the matrix template; wherein the sample disc is placed on the sample holder, the sample disc image is obtained by rotating the sample holder, and the reference rotation angle represents the current rotation angle of the sample holder.

[0038] Exemplarily, before determining the matrix template corresponding to the sample disc image, for any first sample bottle position, based on a first interval between the first sample bottle position and a second sample bottle position, if a difference between the first interval and a reference interval is less than a target interval threshold, the first sample bottle position and the second sample bottle position can be filtered; based on a second interval between the first sample bottle position and a first idle hole position, if a difference between the second interval and the reference interval is less than the target interval threshold, the first sample bottle position and the first idle hole position can be filtered; and / or, for any second idle hole position, based on a third interval between the second idle hole position and a third sample bottle position, if a difference between the third interval and the reference interval is less than the target interval threshold, the second idle hole position and the third sample bottle position can be filtered; based on a fourth interval between the second idle hole position and a third idle hole position, if a difference between the fourth interval and the reference interval is less than the target interval threshold, the second idle hole position and the third idle hole position can be filtered. Wherein the reference interval can be determined based on the type of the sample disc; the reference interval can represent an interval between two adjacent hole positions of the sample disc. On this basis, the sample bottle positions and the idle hole positions remaining after filtering are used to determine the matrix template.

[0039] Exemplarily, the target interval threshold is determined based on at least one of a physical diameter of the sample bottle, a camera resolution, and a bounding box positioning error; the target interval threshold is greater than the physical diameter of the sample bottle; the target interval threshold is negatively correlated with the camera resolution; and the target interval threshold is positively correlated with the bounding box positioning error.

[0040] As can be seen from the above technical solutions, in this embodiment, the positions of sample bottles and available well positions can be determined based on the sample tray image. A matrix template is then determined based on these sample bottle and available well positions. The matrix template includes multiple position points; the predicted category of the position point matching the sample bottle position is the occupied category, and the predicted category of the position point matching the available well position is the available category. Thus, the sample tray layout can be determined based on the predicted categories of each position point in the matrix template, accurately identifying which well positions are occupied and which are available, enabling precise judgment of the sample tray layout. This allows for automatic detection of the sample tray layout in label-free scenarios, without relying on additional physical sensors or pre-placed labels; precise judgment of the sample tray layout can be achieved simply by analyzing the acquired sample tray images.

[0041] By defining a matrix template and matching and fusing information from different viewpoints, the false negative rate can be effectively reduced. This effectively overcomes the differential false negatives caused by single-viewpoint occlusion, lighting variations, etc. Even if some viewpoints have false negatives, they can be supplemented by information from other viewpoints, significantly improving the accuracy of identifying each hole type. It enhances system robustness; the matrix template is insensitive to a small number of detection omissions. As long as the overall row and column structure is not disrupted, the matching process can proceed stably, providing strong fault tolerance to fluctuations in single detections and resulting in high overall stability. It also improves processing efficiency; the matrix template matching process reduces the algorithmic complexity of multi-viewpoint association, enabling rapid and accurate integration of massive amounts of viewpoint information.

[0042] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0043] A liquid chromatograph is an instrument used to separate and analyze the components in a mixture. See also: Figure 2 The diagram shows the structure of a liquid chromatograph (LC). An LC may include a mobile phase (such as a liquid mobile phase), a pump, an injector, a column oven, a chromatographic column, and a detector. The working principle of an LC is as follows: a terminal device (such as a PC) sends a command to the pump, causing the pump to deliver the liquid mobile phase into the injector. The terminal device then sends a command to the injector, causing the injector's needle to accurately extract the sample from the sample vial. This sample is then carried by the liquid mobile phase into the chromatographic column within the column oven. The sample is then separated within the column, and the detector analyzes the components in the mixture, sending the signals back to the terminal device.

[0044] from Figure 2As can be seen, a liquid chromatograph may include an injector, which may include a sample tray, sample holder, sample vials, empty well positions, occupied well positions, and a syringe. The sample tray is used to store the sample to be tested; multiple sample vials can be placed on the tray to store the sample. The sample holder is used to hold the sample tray and is driven by a motor to rotate or move the tray. Sample vials can be placed in the well positions of the sample tray. Empty well positions are those in the sample tray where no sample vials are placed, while occupied well positions are those where sample vials are placed. The syringe is responsible for accurately extracting or injecting the sample to be tested from the sample vials.

[0045] During the operation of a liquid chromatograph, it is necessary to acquire the sample tray layout, which includes the well type of each well on the sample tray. This well type indicates whether a well is occupied or vacant. Based on this, this embodiment enables automatic detection of the sample tray layout in label-free scenarios. It does not rely on additional physical sensors or pre-placed labels; it can accurately determine the sample tray layout simply by analyzing the acquired sample tray images, precisely identifying which wells are occupied and which are vacant. The sample tray layout refers specifically to the layout of the sample tray within the liquid chromatograph injector.

[0046] For example, see Figure 3A The diagram shows the layout of the sample tray and sample holder. Both the sample tray and sample holder are within the camera's field of view, allowing the camera to capture images of both. For example, the sample tray is placed on the sample holder, which can rotate around its center point (i.e., ...). Figure 3A The circular area in the sample holder is rotated so that the sample tray rotates with the sample holder.

[0047] Initially, the sample holder rotates at 0 degrees, and image 1 can be captured at this rotation angle by the camera. Then, the sample holder rotates 20 degrees around its center point (the rotation angle can be configured according to actual needs), resulting in image 2 at this rotation angle. Next, the sample holder rotates 20 degrees around its center point, resulting in image 3 at this rotation angle, and so on, obtaining images at multiple rotation angles. Each image at a rotation angle can be considered as an image from a single viewpoint, thus allowing for the acquisition of images from multiple viewpoints.

[0048] As can be seen from the above, the sample rack is provided with at least one sample disc (herein, one sample disc is taken as an example), and the camera is fixedly arranged above the sample rack. By rotating the sample rack, the camera can observe and shoot the sample disc from multiple perspectives, so as to obtain images under multiple perspectives and accurately record the rotation angles corresponding to each image, for example, image 1 corresponds to 0 degrees, image 2 corresponds to 20 degrees, and so on. For example, the sample rack can be provided with an encoder, and the encoder can be used to realize zero-point positioning and record the rotation angle.

[0049] For example, referring to FIG. 1, which is a schematic diagram of a sample disc layout, the sample disc layout can be determined by using a sample disc layout detection method. Figure 3B As can be seen from the above, the sample rack is provided with at least one sample disc (herein, one sample disc is taken as an example), and the camera is fixedly arranged above the sample rack. By rotating the sample rack, the camera can observe and shoot the sample disc from multiple perspectives, so as to obtain images under multiple perspectives and accurately record the rotation angles corresponding to each image, for example, image 1 corresponds to 0 degrees, image 2 corresponds to 20 degrees, and so on. For example, the sample rack can be provided with an encoder, and the encoder can be used to realize zero-point positioning and record the rotation angle.

[0050] In this embodiment, the sample disc layout needs to be determined, and the sample disc layout includes the types of the hole positions, so as to accurately know which hole positions of the sample disc are occupied hole positions and which hole positions are idle hole positions.

[0051] For example, referring to FIG. 1, which is a schematic diagram of a sample disc layout, the sample disc layout can be determined by using a sample disc layout detection method. Figure 4 As can be seen from the above, the sample rack is provided with at least one sample disc (herein, one sample disc is taken as an example), and the camera is fixedly arranged above the sample rack. By rotating the sample rack, the camera can observe and shoot the sample disc from multiple perspectives, so as to obtain images under multiple perspectives and accurately record the rotation angles corresponding to each image, for example, image 1 corresponds to 0 degrees, image 2 corresponds to 20 degrees, and so on. For example, the sample rack can be provided with an encoder, and the encoder can be used to realize zero-point positioning and record the rotation angle.

[0052] In the multi-target detection and verification process, the sample disc detection, the bottle mouth detection and the idle hole position detection can be performed by using a detection model, so as to obtain multi-perspective multi-target detection results. The sample disc detection, the bottle mouth detection and the idle hole position detection are performed based on the corrected image under the perspective 1, so as to obtain the multi-target detection results under the perspective 1. Similarly, the sample disc detection, the bottle mouth detection and the idle hole position detection are performed based on the corrected image under the perspective K, so as to obtain the multi-target detection results under the perspective K. The multi-target detection results include the sample disc detection results, the bottle mouth detection results and the idle hole position detection results. The sample disc detection results are used to represent the type of the sample disc, the bottle mouth detection results are used to represent the position of the sample bottle (for example, the center position of the bottle mouth of the sample bottle), and the idle hole position detection results are used to represent the position of the idle hole position (for example, the center position of the upper surface of the idle hole position).

[0053] Based on the sample disc detection result (i.e. the type of the sample disc), the spatial constraint relationship of the current scene can be determined, and then the multi-view bottle mouth detection result and the idle hole site detection result are checked mutually exclusive based on the spatial constraint relationship, and the bottle mouth detection result and the idle hole site detection result that do not conform to the spatial constraint relationship are filtered.

[0054] In the multi-view fusion process, the bottle mouth detection result and the idle hole site detection result can be template matched based on the sample disc detection result to obtain a multi-view matrix template. For example, the bottle mouth detection result and the idle hole site detection result of view 1 are template matched to obtain a matrix template of view 1, and so on. The bottle mouth detection result and the idle hole site detection result of view K are template matched to obtain a matrix template of view K. Then, the multi-view matrix templates are fused to obtain the final sample disc layout.

[0055] First, the image preprocessing process. Image preprocessing is used to eliminate interference introduced by camera hardware and other factors in the imaging process, improve image quality, and provide reliable and consistent input data for subsequent multi-target detection and multi-view fusion. For example, the image preprocessing process can include the following steps:

[0056] Step S11, acquiring a multi-view original image through a camera.

[0057] For example, in the initial state, the rotation angle of the sample holder is 0 degrees, and the original image 1 under the rotation angle (0 degrees) is acquired through the camera. Then, the sample holder is rotated 20 degrees around the center point, and the original image 2 under the rotation angle (e.g. 20 degrees) is acquired through the camera. In this way, K original images under K rotation angles can be obtained, K being a positive integer, i.e. K original images under K views can be obtained.

[0058] For example, the sample disc layout detection method can be applied to an electronic device, which can be a liquid chromatograph or a management device of a liquid chromatograph, and the type of the electronic device is not limited. If the sample disc layout detection method is applied to a liquid chromatograph, the liquid chromatograph can execute the sample disc layout detection method based on the K original images under K views after obtaining the K original images under K views. Alternatively, if the sample disc layout detection method is applied to a management device (such as a PC, etc.) of a liquid chromatograph, the liquid chromatograph can send the K original images under K views to the management device after obtaining the K original images under K views, and the management device can execute the sample disc layout detection method based on the K original images under K views.

[0059] Step S12, image pre-processing is performed on the multi-view original images to obtain multi-view corrected images. For example, image pre-processing is performed on the original image of view 1 to obtain the corrected image of view 1, and so on, image pre-processing is performed on the original image of view K to obtain the corrected image of view K.

[0060] For example, the image pre-processing includes but is not limited to de-distortion and the like. For example, due to lens manufacturing and assembly errors, the original image often has radial distortion and tangential distortion, which causes the imaging point to deviate from its ideal position. In order to correct such geometric distortion, a distortion correction model based on camera intrinsic parameters can be used.

[0061] Assuming that the normalized coordinates in the non-distorted image coordinate system are , the actual observed distorted normalized coordinates are , the relationship between the non-distorted normalized coordinates and the distorted normalized coordinates can be seen from formula (1):

[0062] Formula (1)

[0063] In formula (1), can represent the normalized coordinates in the original image, i.e. the normalized coordinates before image pre-processing, can represent the normalized coordinates in the corrected image, i.e. the normalized coordinates after image pre-processing. In addition, , can be a radial distortion coefficient, can be a tangential distortion coefficient. The radial distortion coefficient and the tangential distortion coefficient can be camera intrinsic parameters, and the above radial distortion coefficient and tangential distortion coefficient can be obtained by pre-calibration. In addition, the original image can be corrected pixel by pixel using the inverse mapping interpolation method to restore the non-distorted corrected image.

[0064] Step S13, a mapping relationship between the multi-view corrected images and the sample holder rotation angles is established.

[0065] For example, the corrected image of view 1 (the original image of view 1) has a mapping relationship with the rotation angle 1 (such as 0 degrees), the corrected image of view 2 has a mapping relationship with the rotation angle 2 (such as 20 degrees), the corrected image of view 3 has a mapping relationship with the rotation angle 3 (such as 40 degrees), and so on. In this way, the mapping relationship between the multi-view corrected images and the sample holder rotation angles can be established, forming a structured data set, so as to provide accurate and traceable angle image corresponding information for subsequent multi-view fusion algorithms.

[0066] Second, for the multi-target detection and verification process. In the multi-target detection process, sample disc detection, bottle mouth detection and idle hole site detection can be performed through the detection model to obtain multi-view multi-target detection results. In the multi-target verification process, the spatial constraint relationship of the current scene can be determined based on the sample disc detection result, and then the bottle mouth detection result and the idle hole site detection result of the multi-view are mutually exclusive verified based on the spatial constraint relationship, and the bottle mouth detection result and the idle hole site detection result that do not conform to the spatial constraint relationship are filtered.

[0067] For example, the multi-target detection and verification process can include the following steps:

[0068] Step S21, for any sample disc image, input the sample disc image into the detection model to obtain the first detection box of the sample bottle, the second detection box of the idle hole site and the type of the sample disc.

[0069] For example, after obtaining the multi-view corrected image, the multi-view corrected image can be referred to as a sample disc image, that is, a multi-view sample disc image is obtained. For example, input the sample disc image of view 1 into the detection model to obtain the first detection box of the sample bottle (the first detection box is the bottle mouth detection result), the second detection box of the idle hole site (the second detection box is the idle hole site detection result) and the type of the sample disc (that is, the type of the sample disc is the sample disc detection result), and so on. Input the sample disc image of view K into the detection model to obtain the first detection box of the sample bottle, the second detection box of the idle hole site and the type of the sample disc.

[0070] For convenience of description, a sample disc image is taken as an example for subsequent description, which can be input into three detection models. The first detection model outputs the first detection box of the sample bottle based on the sample disc image, the second detection model outputs the second detection box of the idle hole site based on the sample disc image, and the third detection model outputs the type of the sample disc based on the sample disc image. Or, input the sample disc image into the same detection model, which outputs the first detection box of the sample bottle, the second detection box of the idle hole site and the type of the sample disc based on the sample disc image. For example, the detection model can be a deep learning model or a neural network model, and the network structure of the detection model is not limited. For example, when three independent detection models are used, the detection accuracy can be improved. When a single detection model is used to output three types of detection results at the same time, the calculation efficiency can be improved, and the prediction process of the three types of detection results can share features.

[0071] For example, the first detection box of the sample bottle can be the minimum circumscribed rectangle of the sample bottle opening or the minimum circumscribed rectangle of other positions of the sample bottle, and no limitation is made. When the detection model outputs the first detection box of the sample bottle, the detection model can output the category corresponding to the first detection box (i.e., the sample bottle category, indicating that the hole position at the corresponding position is occupied by the sample bottle) and the coordinates of the first detection box.

[0072] For the second detection box of the idle hole position, the second detection box can be the minimum circumscribed rectangle of the upper surface of the idle hole position, or the minimum circumscribed rectangle of other positions of the idle hole position, and no limitation is made. When the detection model outputs the second detection box of the idle hole position, the detection model can output the category corresponding to the second detection box (i.e., the idle hole position category, indicating that the hole position at the corresponding position is not occupied) and the coordinates of the second detection box.

[0073] For the type of sample disc, assuming that the sample rack can place a type 1 sample disc, a type 2 sample disc and a type 3 sample disc, then the detection model can output type 1, type 2 and type 3 when outputting the type of sample disc. These types of sample disc can constrain some parameters, see the subsequent embodiments.

[0074] For example, if the detection model can achieve a false detection rate of 0.1%, then for a sample disc with dozens of (such as 48) hole positions, the overall layout correct probability is (99.9%)^48≈95.3%, that is, there is about 4.7% of the error rate, it is difficult to meet the requirements of high reliability application, that is, the accuracy of the detection model is not enough to ensure the overall accuracy of the layout recognition. For example, see Figure 5A As shown in FIG. 5, it is a schematic diagram of the distribution of the bottle opening and the idle hole position in the horizontal space correctly detected by the detection model. As shown in FIG. 6, it is a schematic diagram of the distribution of the bottle opening and the idle hole position in the horizontal space incorrectly detected by the detection model. Obviously, the detection model has false detection and missed detection in the actual scene. Figure 5B

[0075] In view of the above finding, in the embodiments of the present application, the spatial constraint relationship of the current scene can also be determined based on the type of the sample disc, and the first detection box of the sample bottle and the second detection box of the idle hole position are mutually checked based on the spatial constraint relationship, and the first detection box and the second detection box that do not conform to the spatial constraint relationship are filtered. In this way, the inherent spatial constraint relationship of the bottle opening, the idle hole position and the sample disc in the current scene can be fully utilized to mutually check the preliminary detection result, and the following will explain the mutual checking process.

[0076] Step S22, determining a first center point of the first detection box and a second center point of the second detection box.

[0077] ​After obtaining the first detection frame of the sample bottle, a first center point of the first detection frame, i.e., a center position point of the first detection frame, is determined. The first center point can be a center point of a minimum circumscribed rectangle of the mouth of the sample bottle. After obtaining the second detection frame of the idle hole site, a second center point of the second detection frame, i.e., a center position point of the second detection frame, is determined. The second center point can be a center point of a minimum circumscribed rectangle of the upper surface of the idle hole site.

[0078] In step S23, based on the first height between the sample bottle and the sample rack, the first center point is mapped to a sample bottle position in a reference coordinate system (such as a world coordinate system); and based on the second height between the idle hole site and the sample rack, the second center point is mapped to an idle hole site position in the reference coordinate system. Thus, the sample bottle position (at least one sample bottle position) in the reference coordinate system and the idle hole site position (at least one idle hole site position) in the reference coordinate system are obtained. For example, the liquid chromatograph can include a sample injector, the sample injector can include a sample disc, the sample disc can include a plurality of hole sites, the sample bottle position is the position of the sample bottle placed in the hole site, and the idle hole site position is the position of the idle hole site without the sample bottle.

[0079] For example, the first center point is a pixel coordinate in the image coordinate system , and the sample bottle position is the mapped position of the first center point in the reference coordinate system, i.e., the first center point is mapped to the reference coordinate system to obtain the distribution position of the sample bottle in the reference coordinate system. The second center point is a pixel coordinate in the image coordinate system , and the idle hole site position is the mapped position of the second center point in the reference coordinate system, i.e., the second center point is mapped to the reference coordinate system to obtain the distribution position of the idle hole site in the reference coordinate system.

[0080] For the image coordinate system , the image coordinate system takes the top-left corner point of the image as the origin, takes the horizontal right direction of the image as the U-axis, takes the vertical downward direction of the image as the V-axis, and takes the pixel as the unit. For the reference coordinate system , the reference coordinate system can be a world coordinate system, the origin of the reference coordinate system can be the rotation center (or other position) of the sample rack, the X-axis and the Y-axis of the reference coordinate system can be located in the horizontal plane of the sample rack and orthogonal, and the Z-axis of the reference coordinate system can be perpendicular to the horizontal plane of the sample rack, i.e., the Z-axis of the reference coordinate system is orthogonal to the X-axis and the Y-axis of the reference coordinate system and located in the vertical direction.

[0081] For example, the first height and the second height can be determined based on the type of the sample disc. For example, the size parameters of various types of sample discs can be pre-configured, such as the size parameters of a type 1 sample disc, the size parameters of a type 2 sample disc, and the like. Based on this, after the type of the sample disc is obtained, the size parameters of the sample disc can be queried, and the size parameters of the sample disc can include the first height and the second height.

[0082] For example, the first height is the height between the sample bottle and the sample rack, indicating the height distance between the sample bottle opening and the sample rack horizontal plane. This first height is an attribute of the sample disc, and after the sample disc placed on the sample rack is determined, this first height is a known value, which can be read from the size parameters of the sample disc.

[0083] The second height is the height between the idle hole and the sample rack, indicating the height distance between the upper surface of the idle hole and the sample rack horizontal plane. This second height is an attribute of the sample disc, and after the sample disc placed on the sample rack is determined, this second height is a known value, which can be read from the size parameters of the sample disc.

[0084] For example, based on the first height between the sample bottle and the sample rack, the first center point can be mapped to the sample bottle position in the reference coordinate system, such as can be determined by using the following formula (2); based on the second height between the idle hole and the sample rack, the second center point can be mapped to the idle hole position in the reference coordinate system, such as can be determined by using the following formula (2).

[0085] Formula (2)

[0086] In formula (2), if represents the first center point, represents the first height, then represents the sample bottle position, that is, the first center point is mapped to the sample bottle position in the reference coordinate system based on the first height. If represents the second center point, represents the second height, then represents the idle hole position, that is, the second center point is mapped to the idle hole position in the reference coordinate system based on the second height.

[0087] In formula (2), represents the rotation component in the camera extrinsic matrix (the camera extrinsic matrix is the extrinsic matrix of the camera relative to the coordinate system with the rotation center of the sample rack as the origin); represents the camera intrinsic matrix; represents the component of the ray direction vector from the camera optical center in the Z axis (Z) of the reference coordinate system; ​This represents the coordinates of the camera's optical center in the reference coordinate system. , This represents the translation component in the camera extrinsic matrix; express On the Z-axis of the reference coordinate system ( The amount of ).

[0088] For example, regarding the rotation component in the camera extrinsic matrix Camera intrinsic parameter matrix Translation components in the camera extrinsic matrix Coordinates of the camera's optical center Components of the ray direction vector along the Z-axis All parameters can be obtained through camera calibration, and there are no restrictions on the method of obtaining them. Thus, for objects at different heights from the horizontal plane of the sample holder, the parameters in formula (2) will be... All parameters are the same, only There are some differences. Based on this, for the bottle neck target, the first center point can be mapped to the sample bottle position in the reference coordinate system based on the first height between the sample bottle and the sample holder. For the vacant hole target, the second center point can be mapped to the vacant hole position in the reference coordinate system based on the second height between the vacant hole and the sample holder. Then, the distribution of the bottle neck and vacant holes on the plane can be analyzed.

[0089] Step S24: For any sample bottle position, rotate the sample bottle position in the opposite direction based on the reference rotation angle to obtain the rotated sample bottle position. The rotated sample bottle position is used to determine the matrix template. For any empty well position, rotate the empty well position in the opposite direction based on the reference rotation angle to obtain the rotated empty well position. The rotated empty well position is used to determine the matrix template.

[0090] For example, when obtaining a sample tray image by rotating the sample holder, the reference rotation angle represents the current rotation angle of the sample holder. For instance, when obtaining the sample bottle position and the position of the available orifice based on the sample tray image from viewpoint 1, the reference rotation angle is the rotation angle corresponding to the sample tray image from viewpoint 1, i.e., the reference rotation angle is 0 degrees. When obtaining the sample bottle position and the position of the available orifice based on the sample tray image from viewpoint 2, the reference rotation angle is the rotation angle corresponding to the sample tray image from viewpoint 2, i.e., the reference rotation angle is 20 degrees.

[0091] For example, taking the reference rotation angle as 20 degrees, for the first straight line segment between the rotation center of the sample holder and the sample bottle position, the rotation center is kept unchanged, the first straight line segment is rotated in the opposite direction of the rotation direction (i.e., the sample bottle position is reversely rotated), and the rotation angle is the reference rotation angle, i.e., 20 degrees. In this way, the end point of the rotated straight line segment (the end point other than the rotation center) is the rotated sample bottle position. In addition, for the second straight line segment between the rotation center of the sample holder and the idle hole position, the rotation center is kept unchanged, the second straight line segment can be rotated in the opposite direction of the rotation direction (i.e., the idle hole position is reversely rotated), and the rotation angle is the reference rotation angle, i.e., 20 degrees. In this way, the end point of the rotated straight line segment (the end point other than the rotation center) is the rotated idle hole position.

[0092] Step S25, filter the sample bottle position and / or the idle hole position that does not conform to the spatial constraint relationship, and the subsequent sample bottle position and idle hole position are rotated sample bottle positions and idle hole positions.

[0093] For example, referring to FIG. 6A, in the correct detection case, the sample bottle positions and the idle hole positions are regularly and uniformly distributed in the horizontal projection, the distances between the two types of targets (such as sample bottle positions, sample bottle positions, sample bottle positions, and idle hole positions) are basically consistent, and the physical layout characteristics of the sample disc are met. However, referring to FIG. 6B, in the incorrect detection case, the two types of targets (such as sample bottle positions, sample bottle positions, sample bottle positions, and idle hole positions) can be abnormally close (such as the distance being less than a set threshold), or the distribution distance of the two types of targets is obviously uneven. Figure 5A Figure 5B For example, referring to FIG. 6A, in the correct detection case, the sample bottle positions and the idle hole positions are regularly and uniformly distributed in the horizontal projection, the distances between the two types of targets (such as sample bottle positions, sample bottle positions, sample bottle positions, and idle hole positions) are basically consistent, and the physical layout characteristics of the sample disc are met. However, referring to FIG. 6B, in the incorrect detection case, the two types of targets (such as sample bottle positions, sample bottle positions, sample bottle positions, and idle hole positions) can be abnormally close (such as the distance being less than a set threshold), or the distribution distance of the two types of targets is obviously uneven.

[0094] ​Based on the above principle, in the embodiment, for any first sample bottle position (i.e. any sample bottle position as the first sample bottle position), based on the first interval (i.e. the distance between the first sample bottle position and the second sample bottle position) between the first sample bottle position and the second sample bottle position (any sample bottle position other than the first sample bottle position), if the difference between the first interval and the reference interval is less than the target interval threshold, it indicates that the two types of targets are abnormally close, and it is determined as conflict detection, therefore, the first sample bottle position and the second sample bottle position can be filtered out, i.e. deleted under the current perspective. Based on the second interval between the first sample bottle position and the first idle hole position (such as any idle hole position), if the difference between the second interval and the reference interval is less than the target interval threshold, it indicates that the two types of targets are abnormally close, and it is determined as conflict detection, therefore, the first sample bottle position and the first idle hole position are filtered out.

[0095] For any second idle hole position (i.e. any idle hole position), based on the third interval between the second idle hole position and the third sample bottle position (i.e. any sample bottle position), if the difference between the third interval and the reference interval is less than the target interval threshold, it indicates that the two types of targets are abnormally close, and it is determined as conflict detection, therefore, the second idle hole position and the third sample bottle position can be filtered out. Based on the fourth interval between the second idle hole position and the third idle hole position (any idle hole position other than the second idle hole position), if the difference between the fourth interval and the reference interval is less than the target interval threshold, it indicates that the two types of targets are abnormally close, and it is determined as conflict detection, therefore, the second idle hole position and the third idle hole position are filtered out.

[0096] In the above process, the reference interval can represent the interval between two adjacent hole positions of the sample disc, and the reference interval can be determined based on the type of the sample disc. For example, the size parameters of each type of sample disc can be pre-configured, so that after the type of the sample disc is obtained, the size parameters of the sample disc can be queried, and the size parameters of the sample disc can include the interval between two adjacent hole positions. Since the two types of targets (such as sample bottle positions, sample bottle positions, sample bottle positions, and idle hole positions) present regular and uniform interval distribution in horizontal projection, i.e. the interval between two adjacent hole positions is the same, therefore, the reference interval can be an attribute of the sample disc, and after the sample disc placed on the sample rack is determined, this reference interval is a known value, which can be read from the size parameters of the sample disc.

[0097] In the above process, the target interval threshold value can be configured according to actual needs, and the target interval threshold value can be determined based on at least one of the physical diameter of the sample bottle, the camera resolution, and the detection frame positioning error, without limitation. For example, when the target interval threshold value is determined based on the physical diameter of the sample bottle, the target interval threshold value can be greater than the physical diameter of the sample bottle. For example, when the target interval threshold value is determined based on the camera resolution, the target interval threshold value is negatively correlated with the camera resolution. For example, when the target interval threshold value is determined based on the detection frame positioning error, the target interval threshold value is positively correlated with the detection frame positioning error.

[0098] For the physical diameter of the sample bottle, the physical diameter of the sample bottle is determined based on the type of the sample disc. For example, after obtaining the type of the sample disc, the size parameters of the sample disc are queried, and the size parameters of the sample disc can include the physical diameter of the sample bottle, i.e., the physical diameter of the sample bottle is an attribute of the sample disc, the physical diameter of the sample bottle is a known value, and is read from the size parameters of the sample disc. The physical diameter of the sample bottle is approximately equal to the hole diameter of the sample disc. On this basis, the target interval threshold value can be greater than the physical diameter of the sample bottle.

[0099] For the camera resolution, the higher the camera resolution, the more accurate the position measurement, and therefore, as the camera resolution decreases, the target interval threshold value should be appropriately increased to accommodate the position measurement error caused by the decrease in camera resolution, thereby avoiding the correct detection being judged as an error detection. On this basis, the target interval threshold value is negatively correlated with the camera resolution, i.e., the lower the camera resolution, the greater the target interval threshold value.

[0100] For the detection frame positioning error, the greater the detection frame positioning error, the greater the position error of the bottle mouth and the idle hole position projected to the reference coordinate system (such as the position error of the sample bottle position and / or the idle hole position), and therefore, as the detection frame positioning error increases, the target interval threshold value should be appropriately increased to accommodate the position projection error caused by the detection frame positioning error, thereby avoiding the correct detection being judged as an error detection. On this basis, the target interval threshold value can be positively correlated with the detection frame positioning error.

[0101] In summary, in the present embodiment, by exploiting the spatial relationship between multiple targets in the liquid chromatograph sample injector scene, the contradictory detection results can be effectively identified and removed without significantly increasing the computational complexity, and the system false detection rate is significantly reduced by the mutual exclusivity of the bottle mouth and the idle hole position in the horizontal space.

[0102] Third, for the multi-view fusion process. In the multi-view fusion process, template matching can be performed on the filtered remaining sample bottle positions and the idle well position positions to obtain a matrix template, i.e., the filtered remaining sample bottle positions and the idle well position positions are used to determine the matrix template. On this basis, the matrix templates of multiple views can be fused to obtain the final sample disc layout, and the sample disc layout is output.

[0103] For example, through the multi-view fusion process, the missed detection caused by occlusion, illumination change or detection model limitation under single view can be solved, and the overall accuracy of sample disc layout detection can be improved. In the multi-view fusion process, it is necessary to establish a correlation relationship for the same detection target (such as an idle well or a bottle mouth) under different views. However, due to the large number of detection targets under single view, when there are multiple sample discs on the sample rack, the number of detection targets further increases, resulting in an increase in correlation complexity. In addition, the missed detection of bottle mouths and idle wells under different views is inconsistent, which also leads to an increase in correlation complexity. To solve the above problems, in the multi-view fusion process in this embodiment, a correlation relationship can be established for the same detection target under different views based on the matrix template, so that the same detection target under different views can be found even if there is a small amount of missed detection.

[0104] On this basis, the multi-view fusion process can include the following steps:

[0105] Step S31, for any sample disc image, determine the matrix template corresponding to the sample disc image.

[0106] In the above process, the sample bottle positions and the idle well position positions have been inversely rotated based on the reference rotation angle (i.e., the rotation angle corresponding to the current view) to obtain the rotated sample bottle positions and the rotated idle well position positions, so as to normalize the sample bottle positions and the idle well position positions to a 0-degree rotation space (a space in which the sample rack is in an initial unrotated state). In this space, the ideal sample bottle positions and the idle well position positions should present a regular distribution of horizontal flatness and verticality. Based on the space constraint that the sample bottle positions and the idle well position positions present a regular distribution of horizontal flatness and verticality, the matrix template corresponding to the sample disc image can be determined.

[0107] For example, the number of horizontal well positions M of the sample disc and the number of vertical well positions N of the sample disc, i.e., the number of rows and the number of columns, are determined. The number of horizontal well positions M represents that there are M well positions horizontally on the sample disc, and the number of vertical well positions N represents that there are N well positions vertically on the sample disc. In this way, there are M*N well positions on the sample disc. The matrix template can include M*N position points, and the M*N position points correspond one-to-one to the M*N well positions of the sample disc.

[0108] For example, the number of transverse hole positions M and the number of longitudinal hole positions N can be determined based on the type of the sample disc. For example, the size parameters of each type of sample disc can be pre-configured, so that after the type of the sample disc is obtained, the size parameters of the sample disc can be queried, and the size parameters of the sample disc can include the number of transverse hole positions M and the number of longitudinal hole positions N. For example, the number of transverse hole positions M and the number of longitudinal hole positions N are attributes of the sample disc, which are known values and can be read from the size parameters of the sample disc.

[0109] For example, the column coordinates of the sample bottle positions and the free hole positions can be clustered to obtain M column center coordinates. For example, based on all sample bottle positions and all free hole positions, the column coordinates of these positions can be obtained, and then the column coordinates of these positions are clustered in the column direction, and the number of clusters is set to the number of columns of the hole matrix, that is, the number of transverse hole positions M, so that M cluster clusters can be obtained, and the center of the cluster cluster is the column center coordinate, thereby obtaining M column center coordinates corresponding to the M cluster clusters.

[0110] When the column coordinates of these positions are clustered in the column direction, the K-means clustering algorithm or the grouping method based on coordinate sorting can be used, and the clustering method is not limited. Since the column coordinates of these positions are approximately uniformly distributed in a grid, the clustering process of the column coordinates is simple and converges quickly.

[0111] For example, the row coordinates of the sample bottle positions and the free hole positions can be clustered to obtain N row center coordinates. For example, based on all sample bottle positions and all free hole positions, the row coordinates of these positions can be obtained, and then the row coordinates of these positions are clustered in the row direction, and the number of clusters is set to the number of rows of the hole matrix, that is, the number of longitudinal hole positions N, so that N cluster clusters can be obtained, and the center of the cluster cluster is the row center coordinate, thereby obtaining N row center coordinates corresponding to the N cluster clusters.

[0112] For example, after the M column center coordinates and the N row center coordinates are obtained, the M column center coordinates and the N row center coordinates form M*N position points, that is, by combining the column center coordinates and the row center coordinates obtained by clustering, a matrix template can be constructed, and the matrix template can include M*N position points, and the intersection of the rows and columns of the matrix template is a template point position for associating the bottle mouth and the free hole.

[0113] For example, referring to Figure 6The matrix template can include M*N position points, and for each position point, the column coordinate of the position point is determined based on the column center coordinate, and the row coordinate of the position point is determined based on the row center coordinate. In this way, through coordinate matching, conversion from discrete multi-target detection results to an ordered template structure is completed, providing a data basis for multi-view fusion. Obviously, through the template matching technology, an ordered matrix template (i.e., a matrix structure) can be generated, the number of matrix templates corresponds to the number of sample plates on the sample rack (taking one matrix template as an example), and the number of position points of the matrix template corresponds to the number of well positions on the sample plate, such as M*N position points.

[0114] In the matrix template construction process, since the completeness of the bottle mouth and the idle well position relative to the well position matrix is fully utilized, the probability of failure of the matrix template construction is extremely low under the premise that the missed detection rate is controlled at a reasonable level. The missed detection rate being controlled at a reasonable level means that under a single view, the combined missed detection rate of the bottle mouth and the idle well position should not result in all targets on a row or a column being missing, thereby destroying the row and column structure of the matrix template, and the detection model can meet this demand. After the matrix template is constructed, there is only one or a few (corresponding to the number of sample plates) matrix templates under each view, and the matrix templates under different views have significant spatial coordinate differences, thereby making the correlation matching between the matrix templates of multiple views simple and robust. Under this strong constraint, accurate multi-view information fusion can be performed for each specific well position on the sample plate. In this embodiment, taking only one matrix template under each view as an example for description.

[0115] Step S32, determining the predicted category of each position point of the matrix template.

[0116] For any sample bottle position, the distance between the sample bottle position and each position point of the matrix template can be calculated, and the position point corresponding to the minimum distance is taken as the position point matched with the sample bottle position, so that the predicted category of the position point matched with the sample bottle position can be the occupied category, indicating that the well position is occupied.

[0117] For any idle well position, the distance between the idle well position and each position point of the matrix template can be calculated, and the position point corresponding to the minimum distance is taken as the position point matched with the idle well position, and the predicted category of the position point matched with the idle well position is the idle category, indicating that the well position is not occupied.

[0118] After traversing all the sample bottle positions and all the free well position positions, for any position point of the matrix template, if the position point is not matched with the sample bottle position and the position point is not matched with the free well position position, the predicted category of the position point is the missing category, indicating that the predicted category of the position point is not detected, i.e., the predicted category of the position point is empty. For any position point of the matrix template, if the position point is matched with the sample bottle position and the position point is matched with the free well position position, the predicted category of the position point is updated to the missing category, i.e., the predicted category of the position point is empty. For any position point of the matrix template, if the position point is matched with the sample bottle position, the predicted category of the position point is the occupied category. For any position point of the matrix template, if the position point is matched with the free well position position, the predicted category of the position point is the free category.

[0119] In step S33, the sample disc layout is determined based on the predicted categories of the position points in the matrix template. The sample disc layout can include the well types corresponding to the M*N wells of the sample disc.

[0120] For example, considering that the sample disc image includes K sample disc images under K perspectives, the matrix template can include K matrix templates under K perspectives, such as a matrix template under perspective 1,..., a matrix template under perspective K. On this basis, the matrix templates under multiple perspectives can be fused to obtain the sample disc layout.

[0121] For example, the sample disc can include M*N wells, and the matrix template can include M*N position points, i.e., the M*N position points of the matrix template correspond one-to-one to the M*N wells of the sample disc. For any well of the sample disc, the predicted categories of the corresponding position points in the K matrix templates are determined, which can be the occupied category, the free category, or the missing category. On this basis, the first number of the occupied categories and the second number of the free categories can be counted. For example, the first number can be the total number of the predicted categories of the corresponding position points in the K matrix templates being the occupied category, and the second number can be the total number of the predicted categories of the corresponding position points in the K matrix templates being the free category.

[0122] After obtaining the first quantity and the second quantity, the hole type corresponding to the hole site can be determined based on the first quantity and the second quantity. For example, if the proportion of the first quantity to the total quantity (e.g., the total quantity is the number K of the matrix template, or the total quantity is the sum of the first quantity and the second quantity) is greater than a pre-set confidence threshold (which can be configured according to experience, such as 70%, 80%, etc.), the hole type is the occupied type, indicating that the hole site is occupied by a sample bottle. Alternatively, if the proportion of the second quantity to the total quantity is greater than the pre-set confidence threshold, the hole type is the idle type, indicating that the hole site is not occupied by a sample bottle. Alternatively, if the proportion of the first quantity to the total quantity is not greater than the pre-set confidence threshold, and the proportion of the second quantity to the total quantity is not greater than the pre-set confidence threshold, it indicates that the detection result for the hole site is abnormal, and an alarm information can be output.

[0123] After the above operation is performed for each hole site of the sample disc, the hole types corresponding to the M*N hole sites of the sample disc can be obtained, and then the sample disc layout is determined based on the hole types corresponding to the M*N hole sites of the sample disc, that is, the sample disc layout includes the hole types corresponding to the M*N hole sites of the sample disc.

[0124] For example, the multi-view information fusion process for a single hole site is as follows: for any hole site on the sample disc, the detection information of the hole site at all views is integrated, and a voting principle based on confidence is used for determination. First, the number of votes (quantity) of the hole site identified as the occupied category and the idle category under different views is counted, and the category with higher votes is determined. Then, if the number of votes of the category exceeds a pre-set confidence threshold, the hole site is determined to be the hole type corresponding to the category, and if the number of votes of the category does not reach the pre-set confidence threshold, the hole type is determined to be unknown and a corresponding level of low confidence alarm is triggered.

[0125] For the pre-set confidence threshold, it can be configured according to actual needs, or it can be flexibly set according to different tolerances of false detection and missed detection, so as to improve the robustness and configurability of the overall determination.

[0126] As can be seen from the above technical solutions, this embodiment does not rely on improving the accuracy of a single detection model. Instead, it utilizes the mutual exclusion of bottle openings and empty well positions in the physical space within the liquid chromatograph injector scenario to perform spatial logic verification on the detection results of multiple targets (bottle openings, empty well positions) output by the detection model, thereby reducing the false detection rate. This significantly reduces the false detection rate: it effectively identifies and eliminates detection results with spatial position conflicts caused by misjudgments by the detection model, thus significantly reducing the overall false detection rate at the system level. It also improves the reliability of the results: by introducing prior knowledge of the scenario for logical verification, it enhances the interpretability and reliability of the detection results. Furthermore, it ensures real-time performance: compared to methods that improve model accuracy, this verification mechanism is based on geometric transformation and distance calculation, has low computational complexity, and does not affect the system's real-time performance.

[0127] This approach abandons the complex correlation of numerous discrete targets from multiple perspectives. Instead, it leverages the complete structure of the detection results set of bottle openings and empty holes relative to a known hole matrix template. Through template matching technology, discrete detection results from a single perspective are rapidly and robustly organized into an ordered matrix template. Matching and information fusion are then performed between matrix templates from different perspectives. This effectively reduces the false negative rate: transforming the complex "target-target" correlation problem into a "template-template" matching problem, it effectively overcomes the differential false negatives caused by single-view occlusion, lighting variations, etc. Even if some perspectives have false negatives, information from other perspectives can be used to complete the identification, significantly improving the accuracy of each hole type identification. It also enhances system robustness: the template structure is insensitive to a small number of detection omissions. As long as the overall row and column structure is not disrupted, the matching process can proceed stably, giving the system strong fault tolerance to fluctuations in single detections and high overall stability. Finally, it improves processing efficiency: template matching greatly reduces the algorithmic complexity of multi-view correlation, avoids high-dimensional optimization problems, and enables the system to quickly and accurately integrate massive amounts of perspective information.

[0128] In the above process, spatial verification ensures detection accuracy, and template fusion guarantees detection precision. Together, these two methods enable highly reliable, highly accurate, and highly recall fully automated sample tray layout detection in industrial applications, avoiding false positives and false negatives in layout detection. Visual algorithms directly analyze the sample tray layout and bottle neck status without any manual pre-labeling, significantly improving efficiency and reducing operating costs. This enables more autonomous and structured sample tray layout recognition and intelligent sample loading.

[0129] Based on the same concept as the method described above, this application proposes a sample tray layout detection device, see [link to relevant documentation]. Figure 7 The diagram shown is a structural schematic of the device, which may include:

[0130] The determining module 71 is configured to determine sample bottle positions and idle well position based on the acquired sample disc image; wherein the sample disc includes a plurality of wells, the sample bottle position is a position of a sample bottle placed in the well, and the idle well position is a position of an idle well in which no sample bottle is placed.

[0131] The processing module 72 is configured to determine a matrix template corresponding to the sample disc image, the matrix template including M*N position points, M representing a number of horizontal wells of the sample disc, and N representing a number of vertical wells of the sample disc; wherein column coordinates of the sample bottle positions and the idle well positions are clustered to obtain M column center coordinates, and row coordinates of the sample bottle positions and the idle well positions are clustered to obtain N row center coordinates, the M column center coordinates and the N row center coordinates forming the M*N position points; a predicted category of a position point matched with the sample bottle position is an occupied category, and a predicted category of a position point matched with the idle well position is an idle category.

[0132] The detecting module 73 is configured to determine a sample disc layout based on the predicted categories of the position points in the matrix template, the sample disc layout including well types corresponding to the M*N wells of the sample disc respectively.

[0133] For example, the sample disc image includes K sample disc images under K perspectives, and the matrix template includes K matrix templates under the K perspectives; when the detecting module 73 determines the sample disc layout based on the predicted categories of the position points in the matrix template, the detecting module 73 is specifically configured to: for any well of the sample disc, determine predicted categories of position points corresponding to the well in the K matrix templates, and count a first number of occupied categories and a second number of idle categories; determine a well type corresponding to the well based on the first number and the second number; wherein if a proportion of the first number to a total number is greater than a preset confidence threshold, the well type is an occupied type, and if a proportion of the second number to the total number is greater than the preset confidence threshold, the well type is an idle type; and determine the sample disc layout based on the well types corresponding to the M*N wells of the sample disc respectively.

[0134] Illustratively, the determining module 71 is specifically configured to determine the sample bottle position and the free well position based on the acquired sample disc image by: inputting the sample disc image into a detection model to obtain a first detection frame of a sample bottle and a second detection frame of a free well, determining a first center point of the first detection frame and a second center point of the second detection frame; mapping the first center point to the sample bottle position in a reference coordinate system based on a first height between the sample bottle and a sample holder; mapping the second center point to the free well position in the reference coordinate system based on a second height between the free well and the sample holder; wherein the first height and the second height are determined based on the type of the sample disc; wherein the sample disc is placed on the sample holder, the origin of the reference coordinate system is the rotation center of the sample holder, the X-axis and the Y-axis of the reference coordinate system are located on the horizontal plane of the sample holder and are orthogonal, and the Z-axis of the reference coordinate system is perpendicular to the horizontal plane of the sample holder.

[0135] Illustratively, the determining module 71 is specifically configured to determine the sample bottle position or the free well position by using the following formula: ; denotes the first center point, denotes the sample bottle position, denotes the first height; or denotes the second center point, denotes the free well position, denotes the second height; denotes the rotation component in the camera extrinsic matrix; denotes the camera intrinsic matrix; denotes the component of the direction vector of the ray from the camera optical center in the Z-axis of the reference coordinate system; denotes the coordinates of the camera optical center in the reference coordinate system, , denotes the translation component in the camera extrinsic matrix; denotes the component in the Z-axis of the reference coordinate system.

[0136] Illustratively, the processing module 72 is further configured to, for any sample bottle position, perform inverse rotation on the sample bottle position based on a reference rotation angle to obtain a rotated sample bottle position, and the rotated sample bottle position is used to determine the matrix template; for any free well position, perform inverse rotation on the free well position based on the reference rotation angle to obtain a rotated free well position, and the rotated free well position is used to determine the matrix template; wherein the sample disc is placed on a sample holder, the sample disc image is obtained by rotating the sample holder, and the reference rotation angle represents the current rotation angle of the sample holder.

[0137] For any first sample bottle position, the processing module 72 is further configured to filter the first sample bottle position and a second sample bottle position based on a first interval between the first sample bottle position and the second sample bottle position if a difference between the first interval and a reference interval is less than a target interval threshold; filter the first sample bottle position and a first idle well position based on a second interval between the first sample bottle position and the first idle well position if a difference between the second interval and the reference interval is less than the target interval threshold; and / or for any second idle well position, filter the second idle well position and a third sample bottle position based on a third interval between the second idle well position and the third sample bottle position if a difference between the third interval and the reference interval is less than the target interval threshold; filter the second idle well position and a third idle well position based on a fourth interval between the second idle well position and the third idle well position if a difference between the fourth interval and the reference interval is less than the target interval threshold; wherein the reference interval is determined based on a type of the sample disc; and the reference interval represents an interval between two adjacent well positions of the sample disc.

[0138] For example, the target interval threshold is determined based on at least one of a physical diameter of the sample bottle, a camera resolution, and a bounding box positioning error; the target interval threshold is greater than the physical diameter of the sample bottle; the target interval threshold is negatively correlated with the camera resolution; and the target interval threshold is positively correlated with the bounding box positioning error.

[0139] Based on the same application concept as the above method, an electronic device is provided in the embodiments of the present application, as shown in Figure 8 The electronic device includes a processor 81 and a machine readable storage medium 82, the machine readable storage medium 82 stores machine executable instructions that can be executed by the processor 81; the processor 81 is configured to execute the machine executable instructions to implement the sample disc layout detection method disclosed in the above examples of the present application.

[0140] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by a processor to implement the sample disc layout detection method disclosed in the above examples of the present application.

[0141] The machine-readable storage medium described above can be any electronic, magnetic, optical, or other physical storage device that contains or stores information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be a Random Access Memory (RAM), a volatile memory, a non-volatile memory, a flash memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., a floppy disk, a DVD, etc.), or any suitable storage medium, or a combination thereof.

[0142] Based on the same application concept as the method described above, the embodiments of the present application further provide a computer program product, which comprises a computer program. When the computer program is executed by a processor, the layout detection method of the sample disc disclosed in the examples of the present application is implemented.

[0143] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, a disk storage, a CD-ROM, an optical storage, etc.) containing computer-usable program code.

[0144] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for detecting the layout of a sample tray, characterized in that, The method includes: The sample bottle position and the free well position are determined based on the acquired sample tray image; wherein, the injector includes a sample tray, the sample tray includes multiple well positions, the sample bottle position is the position of the sample bottle placed in the well position, and the free well position is the position of the free well position where no sample bottle is placed. A matrix template corresponding to the sample tray image is determined. The matrix template includes M*N position points, where M represents the number of horizontal wells in the sample tray and N represents the number of vertical wells in the sample tray. M column center coordinates are obtained by clustering the column coordinates of the sample bottle position and the vacant well position, and N row center coordinates are obtained by clustering the row coordinates of the sample bottle position and the vacant well position. The M column center coordinates and N row center coordinates form the M*N position points. The predicted category of the position point matching the sample bottle position is the occupied category, and the predicted category of the position point matching the vacant well position is the vacant category. The sample disk layout is determined based on the predicted category of each position point in the matrix template. The sample disk layout includes the hole type corresponding to each of the M*N holes in the sample disk. The step of determining the sample bottle position and the position of the free hole based on the acquired sample plate image includes: inputting the sample plate image into the detection model to obtain a first detection frame of the sample bottle and a second detection frame of the free hole, and determining a first center point of the first detection frame and a second center point of the second detection frame; Based on the first height between the sample vial and the sample holder, the first center point is mapped to the position of the sample vial in the reference coordinate system; based on the second height between the vacant hole and the sample holder, the second center point is mapped to the position of the vacant hole in the reference coordinate system. The first height and the second height are determined based on the type of the sample plate; The sample injector includes a sample holder, the sample tray is placed on the sample holder, the origin of the reference coordinate system is the rotation center of the sample holder, the X-axis and Y-axis of the reference coordinate system are located on the horizontal plane of the sample holder and are orthogonal, and the Z-axis of the reference coordinate system is perpendicular to the horizontal plane of the sample holder.

2. The method according to claim 1, characterized in that, The sample disk image includes sample disk images from K perspectives, and the matrix template includes matrix templates from K perspectives, where K is a positive integer; The step of determining the sample disk layout based on the predicted category of each position point in the matrix template includes: For any hole in the sample disk, determine the predicted category of the corresponding position point within K matrix templates, and count the first number of occupied categories and the second number of idle categories; The hole type corresponding to the hole is determined based on the first quantity and the second quantity; wherein, if the ratio of the first quantity to the total quantity is greater than the preset confidence threshold, the hole type is occupied; if the ratio of the second quantity to the total quantity is greater than the preset confidence threshold, the hole type is idle. The layout of the sample disk is determined based on the hole types corresponding to the M*N holes of the sample disk.

3. The method according to claim 1, characterized in that, The position of the sample vial or the position of the vacant well is determined using the following formula: ； in, Indicates the first center point, Indicates the position of the sample bottle. Indicates the first height; or, Indicates the second center point, This indicates the location of the available hole. Indicates the second height; wherein, This represents the rotation component in the camera extrinsic matrix; Represents the camera intrinsic parameter matrix; This represents the component of the ray direction vector originating from the camera's optical center along the Z-axis in the reference coordinate system. This represents the coordinates of the camera's optical center in the reference coordinate system. , This represents the translation component in the camera extrinsic matrix; express The Z-axis component in the reference coordinate system.

4. The method according to any one of claims 1-3, characterized in that, Before determining the matrix template corresponding to the sample disk image, the method further includes: For any sample bottle position, the sample bottle position is rotated in the opposite direction based on the reference rotation angle to obtain the rotated sample bottle position. The rotated sample bottle position is used to determine the matrix template. For any available hole position, the available hole position is rotated in the opposite direction based on the reference rotation angle to obtain the rotated available hole position. The rotated available hole position is used to determine the matrix template. The sample tray is placed on a sample holder, and an image of the sample tray is obtained by rotating the sample holder. The reference rotation angle represents the current rotation angle of the sample holder.

5. The method according to any one of claims 1-3, characterized in that, Before determining the matrix template corresponding to the sample disk image, the method further includes: For any first sample vial position, based on a first interval between the first sample vial position and the second sample vial position, if the difference between the first interval and a reference interval is less than a target interval threshold, then both the first sample vial position and the second sample vial position are filtered; based on a second interval between the first sample vial position and the first available well position, if the difference between the second interval and a reference interval is less than a target interval threshold, then both the first sample vial position and the first available well position are filtered; and / or, For any second vacant well position, based on the third interval between the second vacant well position and the third sample vial position, if the difference between the third interval and the reference interval is less than the target interval threshold, then the second vacant well position and the third sample vial position are filtered; based on the fourth interval between the second vacant well position and the third vacant well position, if the difference between the fourth interval and the reference interval is less than the target interval threshold, then the second vacant well position and the third vacant well position are filtered. The reference interval is determined based on the type of the sample disk; The reference interval represents the interval between two adjacent wells of the sample disk.

6. The method according to claim 5, characterized in that, The target interval threshold is determined based on at least one of the physical diameter of the sample bottle, camera resolution, and detection frame positioning error. Wherein, the target interval threshold is greater than the physical diameter of the sample bottle; The target interval threshold is negatively correlated with the camera resolution. The target interval threshold is positively correlated with the detection box positioning error.

7. A sample tray layout detection device, characterized in that, The device includes: The determination module is used to determine the sample bottle position and the position of the empty well position based on the acquired sample tray image; wherein, the sample injector includes a sample tray, the sample tray includes multiple well positions, the sample bottle position is the position of the sample bottle placed in the well position, and the empty well position is the position of the empty well position where no sample bottle is placed. The processing module is used to determine a matrix template corresponding to the sample tray image. The matrix template includes M*N position points, where M represents the number of horizontal wells in the sample tray and N represents the number of vertical wells in the sample tray. Specifically, the column coordinates of the sample bottle position and the vacant well position are clustered to obtain M column center coordinates, and the row coordinates of the sample bottle position and the vacant well position are clustered to obtain N row center coordinates. The M column center coordinates and N row center coordinates constitute the M*N position points. The predicted category of the position point matching the sample bottle position is the occupied category, and the predicted category of the position point matching the vacant well position is the vacant category. The detection module is used to determine the sample tray layout based on the predicted category of each position point in the matrix template. The sample tray layout includes the hole type corresponding to each of the M*N holes in the sample tray. Specifically, when the determining module determines the sample bottle position and the position of the available aperture based on the acquired sample tray image, it is used to: input the sample tray image into the detection model to obtain a first detection frame for the sample bottle and a second detection frame for the available aperture; determine a first center point for the first detection frame and a second center point for the second detection frame; map the first center point to the sample bottle position in the reference coordinate system based on a first height between the sample bottle and the sample holder; map the second center point to the available aperture position in the reference coordinate system based on a second height between the available aperture and the sample holder; wherein the first height and the second height are determined based on the type of the sample tray; wherein the sample feeder includes a sample holder, the sample tray is placed on the sample holder, the origin of the reference coordinate system is the rotation center of the sample holder, the X-axis and Y-axis of the reference coordinate system are located on the horizontal plane of the sample holder and are orthogonal, and the Z-axis of the reference coordinate system is perpendicular to the horizontal plane of the sample holder.

8. The apparatus according to claim 7, characterized in that, The sample tray image includes sample tray images from K perspectives, and the matrix template includes matrix templates from K perspectives, where K is a positive integer. When the detection module determines the sample tray layout based on the predicted categories of each position point in the matrix template, it specifically performs the following: for any well position in the sample tray, it determines the predicted category of the corresponding position point within the K matrix templates, and counts the first number of occupied categories and the second number of idle categories; it determines the well position type corresponding to the well position based on the first and second numbers; wherein, if the ratio of the first number to the total number is greater than a preset confidence threshold, the well position type is occupied; if the ratio of the second number to the total number is greater than the preset confidence threshold, the well position type is idle; and it determines the sample tray layout based on the well position types corresponding to the M*N well positions in the sample tray. The determining module uses the following formula to determine the position of the sample bottle or the position of the available well: ;in, Indicates the first center point, Indicates the position of the sample bottle. Indicates the first height; or, Indicates the second center point, This indicates the location of the available hole. Indicates the second height; This represents the rotation component in the camera extrinsic matrix; Represents the camera intrinsic parameter matrix; This represents the component of the ray direction vector originating from the camera's optical center along the Z-axis in the reference coordinate system. This represents the coordinates of the camera's optical center in the reference coordinate system. , This represents the translation component in the camera extrinsic matrix; express The Z-axis component in the reference coordinate system; The processing module is further configured to, for any sample bottle position, rotate the sample bottle position in reverse based on a reference rotation angle to obtain a rotated sample bottle position, the rotated sample bottle position being used to determine the matrix template; and for any empty hole position, rotate the empty hole position in reverse based on a reference rotation angle to obtain a rotated empty hole position, the rotated empty hole position being used to determine the matrix template; wherein, the sample tray is placed on the sample holder, and the sample holder is rotated to obtain an image of the sample tray, the reference rotation angle representing the current rotation angle of the sample holder; The processing module is further configured to, for any first sample vial position, filter the first sample vial position and the second sample vial position based on a first interval between the first sample vial position and the second sample vial position, if the difference between the first interval and the reference interval is less than a target interval threshold; filter the first sample vial position and the first idle vial position based on a second interval between the first sample vial position and the first idle vial position, if the difference between the second interval and the reference interval is less than a target interval threshold; and / or, for any second idle vial position, filter the second idle vial position and the third sample vial position based on a third interval between the second idle vial position and the third sample vial position, if the difference between the third interval and the reference interval is less than a target interval threshold; filter the second idle vial position and the third idle vial position based on a fourth interval between the second idle vial position and the third idle vial position, if the difference between the fourth interval and the reference interval is less than a target interval threshold; wherein, the reference interval is determined based on the type of the sample tray; the reference interval represents the interval between two adjacent vials of the sample tray; The target interval threshold is determined based on at least one of the physical diameter of the sample bottle, camera resolution, and detection frame positioning error; the target interval threshold is greater than the physical diameter of the sample bottle; the target interval threshold is negatively correlated with the camera resolution; and the target interval threshold is positively correlated with the detection frame positioning error.

9. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-6.

Images