Material sorting method and material sorting device

Abstract

The material sorting method specifically comprises the following steps: acquiring an X-ray image and a three-dimensional image of a to-be-sorted material; based on at least one of the X-ray image and the three-dimensional image, the material state of each material is identified, the material state comprises a non-adhesion state and an adhesion state, and the adhesion state comprises a state in contact with at least one of the rest materials; for the X-ray image and the three-dimensional image, if the material state recognition is executed, image segmentation is carried out based on the material state of each material, and if the material state recognition is not executed, image segmentation is directly carried out on each material; on the basis of the material state of each material, matching each material in the X-ray image subjected to image segmentation with a corresponding material in the three-dimensional image subjected to image segmentation; and carrying out material sorting according to a material matching result between the X-ray image and the three-dimensional image. The invention further provides a material sorting device.

Description

Technical Field

[0001] The present disclosure relates to the field of material sorting, and more specifically, to a material sorting method and a material sorting device. Background Art

[0002] Material sorting refers to the process of classifying and selecting materials according to their specific properties or standards during production, logistics, warehousing, etc. For example, the materials to be sorted are evenly delivered to the sorting area through a conveyor belt; then the materials to be sorted are scanned using X-ray imaging technology and the X-ray images are identified; the computer analyzes these data, determines the injection strategy, and passes the instructions to the injection execution system; finally, the injection execution system controls the opening and closing of the injection valve and the injection time according to the instructions to spray the materials to the target sorting position.

[0003] In the material sorting process, it is crucial to ensure high accuracy of material identification, which usually depends on whether the images of each material can be accurately segmented. Some materials are very similar in atomic number and color, and the difference between them may not be clearly distinguished through X-ray images. When similar materials with very similar atomic number and color stick to each other during transportation, existing X-ray image segmentation algorithms may not be able to handle it accurately. For example, it is difficult to accurately segment the images of sticky materials, or long sticky materials may be over-segmented into multiple independent parts. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a material sorting method and a material sorting device.

[0005] According to a first aspect of the present disclosure, a material sorting method is provided, comprising: acquiring an X-ray image and a three-dimensional image of a material to be sorted; identifying a material state of each material based on the X-ray image and at least one of the three-dimensional images, wherein the material state includes a non-adhesion state and an adhesion state, and the adhesion state includes a state of contact with at least one other material; for the X-ray image and the three-dimensional image, if material state identification has been performed, image segmentation is performed based on the material state of each material, and if material state identification has not been performed, image segmentation is directly performed on each material; based on the material state of each material, each material in the segmented X-ray image is matched with a corresponding material in the segmented three-dimensional image; and material sorting is performed according to the material matching result between the X-ray image and the three-dimensional image.

[0006] According to an embodiment of the present disclosure, material sorting according to the material matching results between the X-ray image and the three-dimensional image includes at least one of the following: if the material matching result shows that at least one of any two successfully matched materials is in an adhesion state, and at least one has not been processed by image segmentation, a first sorting strategy is executed; if any material in the X-ray image or the three-dimensional image fails to match, the first sorting strategy is executed for the material that failed to match; if the material matching result shows that at least one of any two successfully matched materials is in an adhesion state and has been processed by image segmentation, a second sorting strategy is executed; if the material matching result shows that any two successfully matched materials are both in a non-adhesion state, the second sorting strategy is executed.

[0007] According to an embodiment of the present disclosure, based on the material status of each material, each material in the image segmented X-ray image is matched with the corresponding material in the image segmented three-dimensional image, including: determining the correspondence between the X-ray image and the materials in the three-dimensional image based on the X-ray data of the X-ray image and the point cloud data of the three-dimensional image; for any two materials having a correspondence between the X-ray image and the three-dimensional image, if at least one of the two materials is in a state of adhesion, executing a first matching strategy; if the two materials are not in a state of adhesion, executing a second matching strategy.

[0008] According to an embodiment of the present disclosure, executing the first matching strategy includes: calculating the overlap degree of the two materials, wherein the overlap degree is determined based on at least one of the two-dimensional projection shape and the two-dimensional projection area of ​​the materials; when the overlap degree is greater than or equal to a first value, determining that the two materials with a corresponding relationship are matched successfully.

[0009] According to an embodiment of the present disclosure, when the degree of overlap is less than the first value, at least one of the two materials is determined as a first target material; the first target material is detected for a merging degree with the remaining N materials that are adjacent to each other in the same image, where N is an integer greater than or equal to 1; the first target material is merged with at least one specific material to obtain a merged material, wherein the degree of merging between the specific material and the first target material is greater than or equal to a second value, and the merging process includes marking multiple materials as the same material for rematching.

[0010] According to an embodiment of the present disclosure, when the first target material is a material in the three-dimensional image, performing a merging degree detection on the first target material and the remaining N materials that are adjacent to each other in the same image includes: performing a height continuity detection on the first target material and the N materials respectively, and the height continuity is positively correlated with the merging degree.

[0011] According to an embodiment of the present disclosure, determining at least one of the two materials as the first target material includes: comparing the two-dimensional projection area of ​​the material in the X-ray image and the two-dimensional projection area of ​​the material in the three-dimensional image; and selecting the material with the smaller two-dimensional projection area as the first target material.

[0012] According to an embodiment of the present disclosure, executing the second matching strategy includes: when a single material in the X-ray image forms a one-to-one correspondence with a single material in the three-dimensional image, confirming that the matching is successful. If a single material in one of the X-ray image and the three-dimensional image corresponds to multiple materials in the other image, merging the multiple materials into a single material to form a one-to-one correspondence, and confirming that the matching is successful.

[0013] According to an embodiment of the present disclosure, for the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material; if material state recognition has not been performed, image segmentation is directly performed on each material, including: aligning the X-ray image with a two-dimensional depth image, wherein the two-dimensional depth image is obtained based on the three-dimensional image; based on the material state of each material, segmenting the two-dimensional material image of each material from the aligned X-ray image to obtain a third material set; segmenting the depth material image of each material from the aligned depth image to obtain a fourth material set; wherein, based on the material state of each material, matching each material in the segmented X-ray image with the corresponding material in the segmented three-dimensional image, respectively, includes: matching the material images in the fourth material set with the material images in the third material set as a reference.

[0014] According to an embodiment of the present disclosure, executing the second sorting strategy includes: for any two materials that are successfully matched, obtaining the center of mass and boundary information of the materials in the three-dimensional image; and performing blowing based on the center of mass and boundary information of the materials in the three-dimensional image.

[0015] According to an embodiment of the present disclosure, obtaining the centroid of the material in the three-dimensional image includes: calculating the sum of the heights of all pixel points corresponding to the material in the three-dimensional image; taking a weighted sum of the width coordinates of all the pixel points, and then dividing it by the sum of the heights to obtain the centroid width coordinate; taking a weighted sum of the length coordinates of all the pixel points, and then dividing it by the sum of the heights to obtain the centroid length coordinate.

[0016] According to an embodiment of the present disclosure, the weighted summation includes: taking the height of each pixel point as a weight and multiplying its coordinate value to obtain a weighted result, wherein the coordinate value includes a width coordinate value or a length coordinate value; and summing the weighted results of all the pixel points.

[0017] According to an embodiment of the present disclosure, executing the second sorting strategy also includes: using the X-ray image and the three-dimensional image to identify the category of the second target material, wherein the second target material is a material to be identified that complies with the second sorting strategy; wherein, blowing based on the center of mass and boundary information of the material in the three-dimensional image includes: determining blowing parameters based on the category of the second target material; and blowing according to the blowing parameters based on the center of mass and boundary information of the material in the three-dimensional image.

[0018] According to an embodiment of the present disclosure, identifying the category of the second target material based on the X-ray image and the three-dimensional image includes: identifying the X-ray image to obtain a first confidence score of the second target material, wherein the first confidence score represents the score that the second target material is of the first category; identifying the three-dimensional image to obtain a second confidence score of the second target material, wherein the second confidence score represents the score that the second target material is of the first category; performing a weighted calculation based on the first confidence score and its first weight, and the second confidence score and its second weight to obtain a third confidence score, and when the third confidence score is greater than or equal to a third value, determining that the second target material is of the first category, otherwise it is of the second category.

[0019] According to an embodiment of the present disclosure, identifying the category of the second target material based on the X-ray image and the three-dimensional image includes: inputting the X-ray image into a first network branch of an image recognition model to extract a first feature; inputting the three-dimensional image into a second network branch of the image recognition model to extract a second feature, wherein the image recognition model is constructed based on a deep learning network; splicing the first feature and the second feature, and inputting the spliced ​​features into a fully connected layer of the image recognition model; and allowing the classification layer of the image recognition model to process the output result of the fully connected layer to identify the category of the second target material.

[0020] Another aspect of the disclosed embodiment provides a material sorting device, characterized in that it includes: a conveying module for conveying materials to be sorted; an X-ray image acquisition module for capturing X-ray images of the materials to be sorted; a three-dimensional image acquisition module for capturing three-dimensional images of the materials to be sorted; a control module for acquiring the X-ray image and the three-dimensional image, and executing the material sorting method as described in any one of the above items to generate a sorting instruction; and a blowing module for blowing the material leaving the conveying module in response to the sorting instruction of the control module.

[0021] One or more of the above embodiments have the following beneficial effects: in view of the defect that a single X-ray image segmentation algorithm cannot accurately implement image segmentation, X-ray images and three-dimensional images are comprehensively acquired, and the material state of each material is considered, and X-ray image segmentation and three-dimensional image segmentation are performed respectively, and then the material in the X-ray image is matched with the corresponding material in the three-dimensional image, and then the material is sorted based on the material matching result. Therefore, the continuity of the material in the three-dimensional space can be more accurately captured by using three-dimensional information, and the material image segmentation and material sorting can be performed in combination with X-ray information. For example, for materials with similar atomic numbers and unclear color characteristics, but different material shapes and surface convexity and concavity, or different three-dimensional shapes, the sorting accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 The structure diagram of the material sorting device according to the embodiment of the present application is schematically shown;

[0024] Figure 2 A flow chart of a material sorting method according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 3 A flow chart of material matching according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 4 A flow chart of material matching according to another embodiment of the present disclosure is schematically shown;

[0027] Figure 5 Schematically shows a flow chart of executing a second sorting strategy according to an embodiment of the present disclosure;

[0028] Figure 6 A flowchart of obtaining the centroid of a material in a three-dimensional image according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 7 A flowchart of performing material matching and calculating nozzle opening and closing time based on point cloud data according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 8 A flowchart schematically shows material matching based on a depth image and calculation of nozzle opening and closing time according to an embodiment of the present disclosure;

[0031] Fig. 9 A flowchart for identifying the category of a second target material according to an embodiment of the present disclosure is schematically shown;

[0032] Fig.10 A flowchart for identifying the category of a second target material according to another embodiment of the present disclosure is schematically shown;

[0033] Fig.11 A block diagram of an electronic device suitable for implementing a material sorting method according to an embodiment of the present disclosure is schematically shown.

[0034] The reference numerals related to the above drawings are as follows:

[0035] 100, conveying module; 210, three-dimensional image acquisition module; 220, X-ray image acquisition module; 300, blowing module; 400, sorting bin; 500, control module.

[0036] It should be noted that, for the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the sizes of the overall / local structures or the overall / local areas may be enlarged or reduced, that is, these drawings are not drawn according to the actual scale. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0038] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0039] Figure 1 The structural diagram of a material sorting device according to an embodiment of the present application is schematically shown.

[0040] like Figure 1As shown, the material sorting device includes a conveying module 100, a three-dimensional image acquisition module 210, an X-ray image acquisition module 220, a blowing module 300, a sorting bin 400 and a control module 500. The conveying module 100 is used to convey the material to be sorted; the X-ray image acquisition module 220 is used to capture the X-ray image of the material to be sorted; the three-dimensional image acquisition module 210 is used to capture the three-dimensional image of the material to be sorted; the control module 500 is used to obtain the X-ray image and the three-dimensional image, and execute the material sorting method provided by any embodiment of the present disclosure to generate a sorting instruction; the blowing module 300 is used to respond to the sorting instruction of the control module 500 and blow the material leaving the conveying module 100.

[0041] Exemplarily, the conveying module 100 can be a combination of one or more of a horizontally arranged conveyor belt, an inclined conveyor belt, and an angled inclined slide. In the following, unless otherwise specified, the conveying module 100 is a horizontally arranged conveyor belt. Materials of different types and particle sizes are distributed along the length direction (i.e., the conveying direction) and the width direction on the conveying module 100. Since the conveyor belt has a certain running speed, different materials are scattered on the conveyor belt, and when the conveyor belt is transported to the end position at the end, the materials are thrown out from the conveyor belt for horizontal throwing motion. Referring to the coordinate system in the accompanying drawings, hereinafter, the length direction of the conveying module 100 is set as the first direction (y direction), the width direction is set as the second direction (x direction), and the height direction is set as the third direction (z direction).

[0042] Exemplarily, the X-ray image acquisition module 220 may include an X-ray source and an X-ray detector. For example, the X-ray source may be disposed above the material to be sorted, and the X-ray detector may be disposed below the material to be sorted, for example, the X-ray detector is installed below the conveyor belt of the conveying module 100. The X-ray detector receives the X-rays that penetrate the material and converts them into an X-ray image.

[0043] For example, in the coal and gangue sorting scenario, the X-ray source can transmit the coal and gangue through dual-energy X-rays, and the dual-energy detector obtains analog electrical signals that can reflect the intensity of the transmitted X-rays. By converting the analog-to-digital signals, the coal and gangue images are obtained. Then, according to the different attenuation intensities of the transmitted X-rays, the sorting parameters are calculated from the images, and the coal and gangue are identified and sorted.

[0044] The dual-energy detector can detect the high-energy part and the low-energy part of the transmitted X-ray at the same time. The control module 500 may include a display device (not shown in the figure), which is configured to display the transmission intensity data generated by the high-energy part and the low-energy part of the X-ray. That is, in some optional embodiments of the present application, the transmission intensity data can be determined based solely on the high-energy part or the low-energy part of the X-ray transmitted through the coal and gangue, and the corresponding transmission image can be generated.

[0045] Exemplarily, the three-dimensional image acquisition module 210 may include a three-dimensional structured light camera, which uses a specific pattern of light (such as stripes or grid-shaped light) to project onto the surface of the material, and calculates the three-dimensional shape of the material to obtain a three-dimensional image by analyzing the deformation of the light pattern reflected from the surface of the material. Exemplarily, the three-dimensional image acquisition module 210 may include a line laser binocular stereo camera, which uses a linear laser as a light source to project a straight line-shaped laser beam, and is equipped with two cameras to capture the projection of the laser line on the surface of the material from different angles, and calculates the three-dimensional information of the material to obtain a three-dimensional image by analyzing the difference in the position of the laser line in the images captured by the two cameras.

[0046] Exemplarily, the spray module 300 includes one or more nozzles, for example, a plurality of nozzles arranged in an array. The spray module 300 may also include a solenoid valve and a gas supply device, each nozzle may be connected to a solenoid valve, and the type of the solenoid valve may be a high-frequency solenoid valve. The solenoid valve is connected to a gas supply device (not shown). The gas supply device is equipped with compressed gas, which is used to provide a gas source for the nozzle to spray different types of materials.

[0047] The blowing module 300 in the embodiment of the present application is located at the lower side of the material movement trajectory. The material movement trajectory in the present application refers to the movement trajectory of the material after it leaves the end of the conveying module 100. In this embodiment, by setting the blowing module 300 at the lower side of the movement trajectory, a blowing force can be applied to the material, reducing energy consumption while improving the influence of different blowing positions on the sorting accuracy. For example, when the material reaches the blowing position, the blowing module 300 can instantly spray out a high-pressure airflow of a specific duration, and change the movement trajectory of the material through the airflow, so that the material falls into the corresponding sorting bin 400.

[0048] In some embodiments, the spraying pressure can also be controlled by adjusting the number of nozzles. For example, when the area of ​​the material is large, a large number of nozzles can be used to spray the material surface, and when the area of ​​the material is small, a small number of nozzles can be used to spray the material surface, so that the spraying distance (horizontal movement distance) of materials of different volumes and weights is maintained within a certain range.

[0049] In some embodiments, during the sorting process, the control module 500 can adjust the blowing strategy in real time, such as increasing or decreasing the pressure of the nozzle to change the intensity of the airflow, thereby affecting the flight distance and direction of the material; adjusting the spray angle of the nozzle so that the airflow direction is perpendicular to the tangent at a certain point on the flight trajectory of the material; adjusting the start and stop time of the blowing to match the speed and position of the material through the nozzle.

[0050] The following will be based on Figure 1 The material sorting device described by Figure 2~Figure 10 The material sorting method of the embodiment of the present disclosure is described in detail.

[0051] Figure 2 The flowchart of the material sorting method according to the embodiment of the present disclosure is schematically shown.

[0052] like Figure 2 As shown, this embodiment includes:

[0053] In operation S210, an X-ray image and a three-dimensional image of the material to be sorted are acquired.

[0054] Reference Figure 1 In the embodiment of the present disclosure, the installation height of the X-ray image acquisition module 220 and the 3D image acquisition module 210 can be adjusted according to the width of the conveyor belt, the viewing angle range, etc. For example, the fan-shaped X-ray perpendicular to the carrying surface of the conveyor belt covers the entire width of the conveyor belt. Figure 1 The left side (feeding port) is the front. Figure 1 The right side (discharging port) of the conveyor belt is the rear. A three-dimensional structured light camera is added in front of the X-ray source, and the imaging range of the three-dimensional structured light camera covers the entire width of the conveyor belt. Thus, the X-ray image acquisition module 220 is used to capture the X-ray image of the material on the conveyor belt, and the three-dimensional image acquisition module 210 is used to capture the three-dimensional image of the material on the conveyor belt. It can be understood that the X-ray image acquisition module 220 and the three-dimensional image acquisition module 210 can respectively capture and image a part or all of the area on the conveyor belt.

[0055] In operation S220, a material state of each material is identified based on at least one of the X-ray image and the three-dimensional image, wherein the material state includes a non-adhesive state and an adhering state, and the adhering state includes a state of contact with at least one other material.

[0056] For example, it can be manually preset to identify an X-ray image or a three-dimensional image, or to identify both an X-ray image and a three-dimensional image. For another example, the control module 500 can perform image quality judgment after receiving an X-ray image and a three-dimensional image, respectively, to determine whether the image quality meets the conditions for material state recognition. Determine whether to identify an X-ray image or a three-dimensional image, or to identify both an X-ray image and a three-dimensional image based on the image quality judgment result. For example, one or more of the image quality characteristics such as clarity, contrast, and signal-to-noise ratio are used to pre-set a quality threshold. Select an image whose image quality is greater than or equal to the quality threshold for recognition. If the quality of both the X-ray image and the three-dimensional image is lower than the quality threshold, the image with higher image quality is selected for recognition.

[0057] For X-ray images and three-dimensional images that do not perform material state recognition, the contours of each material in the image are directly detected, and then image segmentation is performed on each contour.

[0058] Exemplarily, the sticking state means that adjacent materials are in contact with each other on the conveyor belt, for example, one material is located above another material. The non-sticking state means that a single material has a certain gap with the surrounding materials, for example, the projections on the carrying surface of the conveyor belt do not intersect.

[0059] In some embodiments, for example, a pre-trained deep learning model can be used to process X-ray images and / or three-dimensional images to effectively identify the material state of each material. For example, the deep learning model includes an edge detection module and an adhesion state recognition module. The edge detection module can run an edge detection algorithm (such as the Canny edge detection method) to identify the edge contour of each material in the input image (such as represented by a set of pixel points). Then, the edge contour of each identified material output by the edge detection module is input to the adhesion state recognition module. The adhesion state recognition module can be constructed based on a convolutional neural network, and can predict whether there is adhesion based on the edge contour of each material. The input image includes an X-ray image and / or a three-dimensional image.

[0060] For example, the material state of each material can also be identified using graphic recognition methods such as edge detection, contour analysis, and convex hull analysis. For example, convex defects in X-ray images and / or three-dimensional image materials can be detected to effectively identify the adhesion state of each material. Among them, convex defects in image processing and computer vision usually refer to a part on the outline or surface of an object that is more prominent or extended than the surrounding area. When two materials are adhered, convex defects may appear as bulges or irregular protrusions on the surface of the material formed by stacking on each other. Effectively identifying convex defects helps to evaluate the adhesion state of each material. In addition to detecting convex defects, adhesion can also be identified by detecting boundary overlap, gray value differences, or shape features.

[0061] In operation S230, for the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material, and if material state recognition has not been performed, image segmentation is directly performed on each material.

[0062] In some embodiments, since the light beam emitted by the X-ray source will inevitably pass through the conveyor belt during the scanning process, the conveyor belt is composed of a whole circle of uniform material, but its joints are bonded by a vulcanization process. The imaging of the vulcanized part in the low-energy X-ray image is slightly different from the imaging of thin and small materials, but the difference is larger in the high-energy X-ray image. Therefore, in order to reduce the interference of the vulcanized part on the image segmentation, the X-ray images used for image segmentation below are preferably high-energy X-ray images.

[0063] For example, whether it is an X-ray image or a three-dimensional image, if the material state of each material in the image is not identified, the contour of each material is directly detected, and the image is segmented based on the material contour; if the material state of each material in the image is identified, different image segmentation operations are performed according to whether it is in an adhesion state. Specifically, the present disclosure performs image segmentation on materials in an adhesion state by performing multiple segmentation operations, and the overall contour containing multiple materials can be first segmented to obtain the overall material image, and then the segmentation operation is continued on the overall material image to obtain the image of each material. The present disclosure performs image segmentation on materials in a non-adhesion state by performing a single segmentation operation, that is, directly segmenting the image of the non-adhesion material.

[0064] For example, X-ray images have been used to identify the state of materials. During the processing of X-ray images, the X-ray image processing algorithm (such as Otsu algorithm, adaptive threshold algorithm, etc.) can be used to detect the material contour first, and then the depth of the convex defects of the contour can be detected to distinguish and process the materials in the non-adhesive and adhering states. For materials in the non-adhesive state, the independent contour of each material can be clearly identified. Once its contour is detected, it is considered that the segmentation condition is met, and the region segmentation is performed directly according to these contours. The materials in the adhering state are regarded as an independent contour of a whole, which usually contains multiple materials. For materials in the adhering state, for example, multiple materials in the adhering state are first segmented from the X-ray image as a whole, and on the basis of obtaining the overall material image, the materials are further segmented into separate individuals. This can be achieved through the concave point detection and matching algorithm, which first identifies the connection points between the materials, that is, the concave points, and then uses certain criteria to determine the optimal segmentation path based on the characteristics of the concave points, and then continues to segment each material in the adhering material. The segmentation condition can be flexibly set according to the parameters of the concave point detection and concave point matching algorithm. For example, if the best segmentation path is not determined, the segmentation condition is not met and each material in the adhesion material is no longer segmented.

[0065] For example, the three-dimensional image has been used to identify the material state. During the processing of the three-dimensional image, the three-dimensional image processing algorithm (such as the three-dimensional Canny boundary detector) can be used to detect the three-dimensional contours of each material. The materials in the non-adhesive state can detect independent three-dimensional contours, while the multiple materials in the adhesive state are regarded as independent three-dimensional contours as a whole. For multiple materials in the adhesive state, the multiple materials in the adhesive state are first separated from the three-dimensional image as a whole, and then the whole material image is input into the three-dimensional convolutional neural network for voxel-level image segmentation, and the materials are further segmented into separate individuals. For example, a discriminator built based on a deep learning algorithm can also be provided to evaluate whether the material image output by the three-dimensional convolutional neural network is accurately segmented. If the evaluation result shows that the segmentation is inaccurate, the segmentation condition is not met, and the original state is restored and no longer segmented. The three-dimensional convolutional neural network is pre-trained using the material image samples in the adhesive state, so that it can learn the complex boundaries between the material adhesions and provide high-precision segmentation results. The discriminator can be trained synchronously with the three-dimensional convolutional neural network. For materials in the non-adhesive state, once their contours are detected, it is considered that the segmentation condition is met, and the region segmentation is directly performed based on these contours.

[0066] For the same material, since the positions and shapes displayed by the X-ray image and the three-dimensional image may be different, the two images of the same material are matched and then used for further sorting.

[0067] In operation S240, each material in the image segmented X-ray image is matched with a corresponding material in the image segmented three-dimensional image based on the material state of each material.

[0068] In some embodiments, the three-dimensional structured light camera is adjusted according to the pixel size of the X-ray image acquisition module 220 so that its spatial resolution in the second direction of the conveyor belt is consistent with the X-ray pixel size, or is an integer multiple or a fractional multiple of the pixel size. At the same time, the sampling frequency of the three-dimensional structured light camera is set to be consistent with the sampling frequency of the X-ray image acquisition module 220. After the sampling frequency and other equipment parameters are fixed, during the transportation of the conveying module 100, the material first passes through the acquisition area of ​​the three-dimensional structured light camera, and a three-dimensional image is acquired at time t1, and then passes through the acquisition area of ​​the X-ray image acquisition module 220 after a fixed distance, and a dual-energy X-ray image is acquired at time t2. Since the acquisition time difference (the difference between time t1 and time t2) between the three-dimensional structured light camera and the X-ray image acquisition module 220 is fixed, the control module 500 can correct the offset of the three-dimensional image and the X-ray image in the time sequence according to the fixed time difference, that is, the three-dimensional image acquired at time t1 and the X-ray image acquired at time t2 are determined as images with the same material set. In operation S240, the three-dimensional image and the X-ray image that have been corrected in time sequence are matched.

[0069] For example, for any material, the three-dimensional image and the X-ray image are preprocessed by translating, rotating, scaling and other operations, and then the position similarity, shape similarity and whether the two-dimensional projection area is greater than the corresponding threshold can be used for matching, where the position similarity reflects the proximity of the center of the three-dimensional image and the X-ray image, and the shape similarity reflects the overlap of the boundaries of the three-dimensional image and the X-ray image. The two-dimensional projection area reflects the difference in the two-dimensional area of ​​the material in the three-dimensional image and the X-ray image. Among them, the X-ray image is a two-dimensional material image obtained from top to bottom, and the size of the area occupied by the material is regarded as the two-dimensional projection area in the X-ray image.

[0070] In some embodiments, operation S240 may include determining the correspondence between the X-ray image and the materials in the three-dimensional image based on the X-ray data of the X-ray image and the point cloud data of the three-dimensional image; for any two materials having a correspondence between the X-ray image and the three-dimensional image, if at least one of the two materials is in a state of adhesion, executing a first matching strategy; if the two materials are not in a state of adhesion, executing a second matching strategy.

[0071] For example, only one of the X-ray image and the three-dimensional image performs material status identification. In this case, at least one of the two materials is in an adhesion state, including that the material in one of the images is marked as an adhesion state, while the corresponding material in the other image has no status mark; in this case, the two materials are not in an adhesion state, including that the material in one of the images is marked as a non-adhesion state, while the corresponding material in the other image has no status mark.

[0072] For another example, both the X-ray image and the three-dimensional image perform material state identification. In this case, at least one of the two materials is in an adhesion state, including that the material in one of the images is marked as an adhesion state, and the corresponding material in the other image is marked as an adhesion state or a non-adhesion state; in this case, the two materials are not in an adhesion state, including that the materials in both images are marked as a non-adhesion state.

[0073] According to the embodiments of the present disclosure, by designing different matching strategies for different types of material states, the overall matching quality can be improved and possible errors that may occur in subsequent processing can be reduced.

[0074] In operation S250, materials are sorted according to the material matching results between the X-ray image and the three-dimensional image. For example, the material category can be identified according to the material matching results, and the blowing strategy can be determined, and then the sorting is completed by executing the blowing.

[0075] According to the embodiments of the present disclosure, three-dimensional information can be used to more accurately capture the continuity of materials in three-dimensional space, and combined with X-ray information to perform material image segmentation and material sorting. For example, for materials with similar atomic numbers and unclear color characteristics, but different material shapes and surface convexity or concavity, or different three-dimensional shapes, the sorting accuracy can be improved.

[0076] Figure 3 A flow chart of material matching according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 3 As shown, this embodiment is one of the embodiments of operation S240, including:

[0078] In operation S310, a correspondence between the X-ray image and a material in the three-dimensional image is determined based on X-ray data of the X-ray image and point cloud data of the three-dimensional image.

[0079] For example, the correspondence is determined based on the position information in the X-ray image and the three-dimensional image. Specifically, when the conveyor belt is running at a constant speed, the materials presented in the X-ray image and the three-dimensional image are the same, so the position of the same material relative to the conveyor belt in different images is consistent. After the two images are preprocessed and aligned with each other, the materials with the same relative position are determined from the X-ray data and the point cloud data, and feature points are found, such as the center, corner point, intersection point, texture point or edge point of the material. One or more feature points are matched using a feature matching algorithm, and the correspondence between the X-ray image and the three-dimensional image is established based on the matching results. In addition, the correspondence between the materials can also be established based on the gray value, color, geometric constraints or structural similarity of the materials. Among them, the correspondence can be characterized by a matching score. When the matching score is higher than a certain value, such as a matching score of 100 points, higher than 80 points, the same material identification is assigned to establish the correspondence.

[0080] Exemplarily, a two-dimensional material image of each material is extracted from the X-ray image to obtain a first material set, wherein, for materials in a sticky state, if the segmentation conditions are met, image segmentation is performed to obtain a two-dimensional material image; a point cloud material image of each material is extracted from the three-dimensional image to obtain a second material set, wherein, for materials in a sticky state, if the segmentation conditions are met, image segmentation is performed to obtain a point cloud material image; wherein, determining the correspondence between the X-ray image and the material in the three-dimensional image includes: determining the correspondence between the material images in the first material set and the second material set.

[0081] For any two materials having a corresponding relationship between the X-ray image and the three-dimensional image, if at least one of the two materials is in an adhesion state, the first matching strategy is executed, such as operation S320.

[0082] In operation S320, for any two materials having a corresponding relationship between the X-ray image and the three-dimensional image, the overlap of the two materials is calculated, and the overlap is determined based on at least one of the two-dimensional projection shape and the two-dimensional projection area of ​​the materials; the overlap refers to the measure of the overlapping area between the X-ray image and the three-dimensional image of two or more materials, which can be represented by a number between 0 and 1. For example, when the overlap is 1, it means that the two-dimensional projection shape and / or two-dimensional projection area of ​​the materials completely overlap. The X-ray image itself is a two-dimensional image, in which the two-dimensional projection shape and two-dimensional projection area of ​​the material are presented.

[0083] In operation S330, when the overlap degree is greater than or equal to the first value, it is determined that the two materials having the corresponding relationship are matched successfully.

[0084] Exemplarily, for the two-dimensional material image and the point cloud material image that have a corresponding relationship, the overlap of the materials displayed by the two is calculated; wherein, when the overlap is greater than or equal to a first value, it is determined that the two-dimensional material image and the point cloud material image that have a corresponding relationship are successfully matched. At this time, it can be considered that the two-dimensional material image and the point cloud material image of the two materials that are successfully matched are actually the two-dimensional material image and the point cloud material image respectively obtained based on the same material entity.

[0085] For example, only the shape similarity between the two materials may be calculated as the overlap degree, or only the two-dimensional projection area error between the two materials may be calculated as the overlap degree.

[0086] In some embodiments, the two-dimensional projection shape and two-dimensional projection area of ​​the material can also be used simultaneously to calculate the intersection of the two material projection areas to obtain the overlap. First, the three-dimensional material image is projected to two dimensions (such as the xoy plane, i.e., the bearing surface of the conveyor belt), the two-dimensional projection area of ​​the material under the three-dimensional image is calculated, and the coordinates of the xoy plane are used to characterize the boundary contour of the material. The two-dimensional plane displayed by the X-ray image is the xoy plane, so the area of ​​the material in the X-ray image and the boundary contour of the material are directly calculated. The overlapping area can be calculated by the coordinates of the two contours (for example, pixels with the same coordinates on the boundary contour are considered to overlap), and the intersection area is obtained. The overlap can be obtained by dividing the intersection area by the minimum two-dimensional projection area. The minimum two-dimensional projection area is the smaller two-dimensional projection area of ​​the two materials in the X-ray image and the three-dimensional image. The first value can be selected based on the type of material, sorting requirements or production experience, for example, it can be 0.8, 0.9, etc., which is not specifically limited here.

[0087] When the overlap degree is less than the first value, operations S340 to S360 are performed.

[0088] In operation S340, at least one of the two materials is determined as a first target material.

[0089] In some embodiments, the two-dimensional projection area of ​​the material in the X-ray image and the two-dimensional projection area of ​​the material in the three-dimensional image can be compared, and the one with the smaller two-dimensional projection area is selected as the first target material. The reason is that only when the two-dimensional projection area is smaller, it is necessary to merge to increase the area.

[0090] In operation S350, the first target material and the remaining N materials in the same image are subjected to a merging degree detection, where N is an integer greater than or equal to 1.

[0091] Exemplarily, the merging degree refers to the degree to which two materials meet specific merging criteria, such as whether the two materials are close in physical space or have similar properties. For example, the merging degree can be measured by at least one parameter such as atomic number, physical distance, three-dimensional properties and color similarity. The merging degree can be represented by a number between 0 and 1. For example, when the merging degree is 1, it means that the parameters such as atomic coefficients of the first target material and the other N materials adjacent to the same image are exactly the same. The merging degree detection is used to evaluate the similarity between the first target material and the surrounding materials in order to decide whether to merge.

[0092] In operation S360, the first target material is merged with at least one specific material to obtain a merged material, wherein the merging degree between the specific material and the first target material is greater than or equal to the second value, and the merging process includes marking multiple materials as the same material for re-matching, and the images of the multiple materials can be fused, for example, the boundary contour of the specific material in the X-ray image is fused with the boundary contour of the first target material, or the point cloud of the specific material in the three-dimensional image is fused with the point cloud of the first target material. The second value is a preset threshold value used to determine whether the two materials meet the merging standard. For example, the second value can be set to 0.7.

[0093] Combination Figure 3 To further elaborate, when the material is determined to be in a non-adhesive state, the image segmentation is accurate, and the two-dimensional projection area in the X-ray image and the two-dimensional projection area in the three-dimensional image are consistent, and the overlap is greater than or equal to the first value. When the material is in an adhering state, there may be multiple situations:

[0094] 1. A plurality of materials in an adhered state are correctly segmented twice in both the X-ray image and the three-dimensional image, and the overlap degree of each material during matching is greater than or equal to a first value.

[0095] 2. If the multiple materials in the adhesion state are not segmented again after the first segmentation operation is performed on the X-ray image to obtain the overall image, and are not segmented again after the first segmentation operation is performed on the three-dimensional image to obtain the overall image, then the overlap degree of the two overall images during matching is greater than or equal to the first value.

[0096] The first matching strategy may need to be merged in the following situations:

[0097] 3. If the multiple materials in the adhesion state are correctly segmented twice in the X-ray image but not segmented twice in the three-dimensional image, the overlap degree of the two images calculated based on the two-dimensional material image obtained by the correct secondary segmentation and the three-dimensional material image that is not segmented twice is less than a first value, and the multiple two-dimensional material images are further merged in the X-ray image, and then operation S310 and subsequent operations are executed in a loop until the multiple materials in the adhesion state are successfully matched as a whole between the two images.

[0098] 4. If the multiple materials in the adhesion state are not segmented twice in the X-ray image but are correctly segmented twice in the three-dimensional image, the overlap degree of the two images calculated based on the three-dimensional material image obtained by the correct secondary segmentation and the two-dimensional material image that is not segmented twice is less than a first value, the multiple three-dimensional material images are merged in the three-dimensional image, and then operation S310 and subsequent operations are executed in a loop until the multiple materials in the adhesion state are successfully matched as a whole between the two images.

[0099] 5. Multiple materials in an adhesion state are correctly segmented in the X-ray image, but the segmentation is wrong when the secondary segmentation is performed in the 3D image. Alternatively, multiple materials in an adhesion state are segmented in the X-ray image, but the segmentation is wrong, and the secondary segmentation is correct in the 3D image. Alternatively, multiple materials in an adhesion state are segmented in the X-ray image, but the segmentation is wrong, and the secondary segmentation is also wrong in the 3D image.

[0100] In the fifth case, for a certain material segmented in one image, it may be impossible to determine the corresponding material in another image, and the matching is considered to have failed. Alternatively, for a certain material segmented in one image, it is possible to determine the corresponding material in another image, but the calculated overlap is less than the first value, and after multiple (such as 3 times, just an example) merge processes and re-matching, the overlap is still less than the first value, and the matching is considered to have failed.

[0101] like Figure 3 As shown, after executing operation S360, the process returns to loop and executes operation S310 and subsequent operations to perform re-matching. The re-matching includes the situations of successful matching and failed matching within a loop of a predetermined number of times (such as 3 times, which is only an example).

[0102] In the case of a successful match, it is marked as "the material in the sticky state has not been processed by image segmentation". In the case of a failed match, it is marked as "matching failed". As a fallback strategy, merging processing can obtain other images that may be incorrectly segmented in the case of a successful match, thereby reducing the probability of incorrect matching. For example, an incorrect match includes two material images that are not the same material entity and still match successfully. For material A in a stuck state, the priority of merging processing can be set higher than the priority of successful matching, that is, if material A successfully matches with the material of another image, and it is determined that it can be merged with the adjacent material B that failed to match, then the merge processing is performed.

[0103] According to the embodiments of the present disclosure, multiple erroneously secondary-segmented material images can be fused through merging processing. For example, if a material in a sticky state is erroneously secondary-segmented, the overall material image before the secondary segmentation can be restored through merging processing. This helps to pre-screen out erroneously secondary-segmented material images and perform targeted processing later to avoid subsequent recognition errors and blowing errors, which in turn lead to sorting errors.

[0104] In other embodiments, not all successfully matched images are marked as "materials in an adhering state have not been processed by image segmentation". For example, when the merged material image is the whole material image before secondary segmentation, it is marked as "materials in an adhering state have not been processed by image segmentation"; when the successfully matched merged material image is not the whole material image before secondary segmentation, it can be marked as "matched successfully", for example, the material image of the same material entity in the X-ray image or the three-dimensional image is mistakenly divided into multiple images (the first target image is one of them), and after the merging process, the two-dimensional material image and the three-dimensional material image of the same material entity can be successfully matched.

[0105] In some embodiments, when the first target material is a material in a three-dimensional image, performing a merging degree detection on the first target material and the remaining N materials adjacent to each other in the same image includes: performing a height continuity detection on the first target material and the N materials respectively, and the height continuity is positively correlated with the merging degree.

[0106] Exemplarily, the material adjacent to the first target material is the material in a proximate position.

[0107] In some embodiments, performing height continuity detection on the first target material and the N materials respectively includes: calculating the height difference between the first target material and each of the N materials.

[0108] Exemplarily, height continuity refers to the degree of smoothness of the materials connecting with each other in height. For example, if the height difference between two materials is lower than a certain value, the degree of smoothness of the connection is high. For example, the height continuity index can be obtained based on the ratio of the absolute value of the height difference between the materials and a reference value (such as the average height of the materials). For example, when the height continuity index is less than 0.1, it is determined that the two materials are continuous. Assuming that the height of material A is 5 cm, the height of material B is 3 cm, the height difference is 2 cm, the average height is 4 cm, and the height continuity index is 0.5, the degree of smoothness of the connection between material A and material B is low, and the degree of merging is low.

[0109] According to the embodiments of the present disclosure, through high continuity detection, it is possible to more accurately determine whether materials can be merged, thereby improving recognition accuracy.

[0110] In some embodiments, when the first target material is a material in an X-ray image, the first target material is detected with the remaining N materials that are adjacent to it in the same image to determine the degree of merging, including: first determining the first target material and one or more materials adjacent to it, calculating the physical distance, relative atomic number similarity, and color similarity of the image information of each material, and assigning values ​​to obtain the degree of merging.

[0111] For example, first calculate the physical distance between the first target material and each adjacent material. The physical distance can be calculated by the shortest distance between the boundary contours of the materials, or by the distance between the center coordinates of the materials. When the first target material and the adjacent materials are obtained by secondary segmentation of the overall material image, the shortest distance between the boundary contours is 0, otherwise the Euclidean distance between the center coordinates is calculated. Then, calculate the relative atomic number similarity between the first target material and each adjacent material, and then calculate the color similarity between the first target material and each adjacent material. For example, the color vector of tissue A is (120, 130, 140), and the color vector of adjacent tissue B1 is (122, 128, 139), then the color similarity s(A, B1) = 0.98. Assign values ​​to the physical distance and color similarity respectively, and sum them to obtain the merging degree. The closer the physical distance, the higher the weight, the larger the relative atomic number, the higher the weight, and the greater the color similarity, the higher the weight.

[0112] In some embodiments, when the first target material is a material in an X-ray image, performing a merging degree detection on the first target material and the other N materials adjacent to each other in the same image further includes: reversely searching whether the material in the three-dimensional image still has a corresponding relationship with the other N materials, and when there is a one-to-many corresponding relationship, performing a high continuity detection on the material itself in the three-dimensional image. Wherein, merging the first target material with at least one specific material to obtain a merged material includes: if the high continuity detection result indicates that the merging degree is greater than or equal to a second value, then merging the first target material with at least one specific material to obtain a merged material; if the high continuity detection result indicates that the merging degree is less than the second value, then segmenting the materials in the three-dimensional image according to the segmentation result of the X-ray image, and then matching the segmented materials one by one.

[0113] In some embodiments, executing the second matching strategy includes: when a single material in the X-ray image forms a one-to-one correspondence with a single material in the three-dimensional image, confirming that the match is successful; if there is a correspondence between a single material in one of the X-ray image and the three-dimensional image and multiple materials in the other image, then when a one-to-one correspondence is formed after the multiple materials are merged into a single material, confirming that the match is successful.

[0114] For example, only the material state of the X-ray image is identified to determine whether the material is adhered, and then different matching strategies are executed according to the adhesion state judgment results. When executing the second matching strategy, if the material is determined to be in a non-adhesive state, the X-ray image segmentation algorithm will not further segment it into multiple materials. Then, it is identified whether the single material in the X-ray image and one or more materials in the three-dimensional image overlap partially or completely. For example, if the overlap is above 0.1, it is determined that there is a corresponding relationship, and all point cloud material images that intersect with the X-ray material image (such as the overlap is above 0.1) are identified. If only one point cloud material overlaps with it, it is considered that the segmentation of both parties is correct, and a one-to-one corresponding match is achieved. If the number of overlapping point cloud materials exceeds one, it indicates that the point cloud data is mis-segmented. At this time, these mis-segmented point cloud materials are merged to ensure that the merged single material forms a correct match with the single material based on the X-ray image.

[0115] Figure 4 A flow chart of material matching according to another embodiment of the present disclosure is schematically shown.

[0116] like Figure 4 As shown, in this embodiment, operations S410 to S420 are one embodiment of operation S230, and operation S430 is one embodiment of operation S240, which specifically includes:

[0117] In operation S410 , the X-ray image is registered with a two-dimensional depth image, where the two-dimensional depth image is obtained based on the three-dimensional image.

[0118] In operation S420, based on the material state of each material, the two-dimensional material image of each material is extracted from the registered X-ray image to obtain a third material set; the depth material image of each material is segmented from the registered two-dimensional depth image to obtain a fourth material set; wherein, for materials in an adhesion state, if the segmentation conditions are met, image segmentation is performed to obtain a depth material image.

[0119] In operation S430 , the material images in the fourth material set are matched with the material images in the third material set as a reference.

[0120] Exemplarily, the projection transformation matrix from the X-ray image to the depth image is obtained in advance. After receiving the depth image and the X-ray image, the control module 500 aligns the X-ray image with the depth image through the projection transformation matrix, and then performs image segmentation on the aligned X-ray image to cut the adhered materials, and then matches the materials in the depth image with the materials in the aligned X-ray image one by one.

[0121] Among them, taking the material images in the third material set as the reference means: assuming that all the material images in the third material set are segmented correctly, when the materials at the same position in the depth image and the X-ray image do not match, the materials in the depth image are re-segmented according to the material images segmented in the X-ray image. In this way, the X-ray image segmentation algorithm with higher accuracy can be used to provide a reference for the segmentation of the depth image, simplify the decision-making process, and reduce the time and resources consumed in the decision-making process during the matching process.

[0122] According to the embodiments of the present disclosure, materials in the X-ray image are matched with corresponding materials in the three-dimensional image based on the results of image segmentation, which can effectively overcome the limitations of X-ray segmentation, make full use of the three-dimensional characteristics of three-dimensional data (such as point cloud data), and achieve higher-precision material segmentation and matching.

[0123] In some embodiments, sorting materials according to the material matching results between the X-ray image and the three-dimensional image includes at least one of the following:

[0124] If the material matching result shows that at least one of any two successfully matched materials is in an adhesion state, and at least one of them has not been processed by image segmentation, the first sorting strategy is executed. If any material in the X-ray image or the three-dimensional image fails to match, the first sorting strategy is executed for the material that failed to match. Not processed by image segmentation means that the overall image obtained after the first segmentation operation has not been correctly segmented for the second time. As in the case of the combined processing in the first matching strategy above, for successful matches, it is marked as "materials in an adhesion state have not been processed by image segmentation." For failed matches, it is marked as "matching failed." For example, in the case where multiple materials in an adhesion state have not been correctly segmented for the second time in both the X-ray image and the three-dimensional image, it is marked as "materials in an adhesion state have not been processed by image segmentation."

[0125] If the material matching result shows that at least one of any two successfully matched materials is in an adhesion state and has been image segmented, the second sorting strategy is executed. If the material matching result shows that any two successfully matched materials are both in a non-adhesion state, the second sorting strategy is executed. Among them, any two successfully matched materials are regarded as the same target material entity, and the second sorting strategy is executed for the target material entity. The image segmentation processing here is to perform a correct secondary segmentation after obtaining the overall image after the first segmentation operation.

[0126] According to the embodiments of the present disclosure, a variety of sorting strategies are provided, which can take into account the object state, image segmentation process and material matching results, and apply targeted sorting strategies, thereby improving the material sorting effect.

[0127] Exemplarily, if at least one of any two materials that are successfully matched is in a state of adhesion, and at least one has not undergone secondary image segmentation processing, then it can be considered that at least one or two materials are matched in a state of adhesion. In this case, since it is difficult to determine the center of mass of each independent material, it may be difficult to determine the exact injection position, and it is also difficult to accurately identify the category. If any material in the X-ray image or the three-dimensional image fails to match, there may also be segmentation errors, which will also make it difficult to determine the center of mass, unable to inject, and difficult to accurately identify the category. Therefore, the first sorting strategy can be to place the target material that meets the strategy into a specific position, and then put it back on the conveyor belt, or perform manual sorting. For example, in the sorting scenario of concentrate and tailings, three sub-bins are set, corresponding to the concentrate sub-bin, the tailings sub-bin, and the sub-bin that meets the first sorting strategy. As Figure 1 As shown, the materials that meet the first sorting strategy fall directly into the corresponding sub-bin by gravity. The concentrate and tailings are sorted into the concentrate sub-bin and tailings sub-bin by blowing.

[0128] The content of the second sorting strategy is further explained below.

[0129] Figure 5 A flowchart for executing the second sorting strategy according to an embodiment of the present disclosure is schematically shown.

[0130] like Figure 5 As shown, this embodiment is one of the embodiments for executing the second sorting strategy. Any two materials that are successfully matched are regarded as the same target material entity:

[0131] In operation S510, the centroid and boundary information of the material in the three-dimensional image is obtained.

[0132] For example, when extracting point cloud data from a 3D image, a boundary detection algorithm (such as the Canny boundary detector, Sobel operator, Laplacian operator, etc.) can be used to identify the boundary of the material, and the boundary information can be represented by 3D coordinates. The center of mass can be determined by calculating the weighted average position of the material pixels, such as by weighted summing the coordinates of each voxel (3D pixel).

[0133] In operation S520, spraying is performed based on the centroid and boundary information of the material in the three-dimensional image. In this operation, the target material entity is sprayed.

[0134] In some embodiments, the control module 500 can more efficiently determine the opening and closing time and the spraying position of the control valve nozzle directly based on the center of mass and boundary information of the material in the three-dimensional image.

[0135] In other embodiments, the centroid and boundary information of the material in the three-dimensional image can be projected onto the same two-dimensional plane as the X-ray image, and correction can be performed based on the projected centroid and boundary information. The control module 500 can determine the opening and closing time and the spraying position of the control valve nozzle based on the corrected X-ray image.

[0136] For example, based on the boundary contour information of the three-dimensional image projected onto a two-dimensional plane (such as the Xoy plane), the boundary contour of the material in the X-ray image is adjusted through deformation correction (such as affine transformation, non-rigid transformation) to ensure the accuracy of the boundary. In addition, based on the centroid of the three-dimensional image projected onto the two-dimensional plane, the centroid of the material in the X-ray image is kept consistent.

[0137] In terms of identification, there are many types of ores, and their color, shape, density, composition and other characteristics vary significantly. Some concentrates and tailings are very close in atomic number and color, but there are large differences in the surface convexity or three-dimensional shape of the concentrate and tailings. The closer the atomic number of the concentrate and the tailings, the more difficult the identification result of dual-energy X-ray imaging is. The images collected by the visible light camera at high speed are difficult to accurately capture the detailed features of the concave and convex surface of the mineral due to inherent defects such as reflection and dispersion. X-ray imaging and visible light imaging are 2D imaging and cannot capture the three-dimensional shape of the ore. Therefore, the recognition ability of X-ray and visible light images for these ores is limited, which in turn affects the effective sorting process.

[0138] In terms of jet separation, X-ray imaging uses a fan-shaped beam. When the beam is obliquely projected onto the ore on the conveyor belt, the imaging position of the ore will be offset from the actual position. This offset is particularly obvious when the ore is thick. Similarly, the increase in the oblique penetration distance will further increase the deviation of the center of mass position. Since the valve nozzle needs to be accurately aligned according to the position of the ore in the image, this deviation directly affects the accuracy of the jet operation. In practical applications, visible light imaging technology is often affected by interference factors such as dirt and water stains on the conveyor belt surface. These interferences will cause the imaging boundary line between the ore boundary and the conveyor belt surface to become blurred, making it difficult to accurately determine the ore position. This ambiguity will also have an adverse effect on the accuracy of the jet operation. Therefore, three-dimensional imaging technology can be used as an effective supplement for material sorting based on X-ray imaging, and the defect of center of mass position deviation in X-ray images can be overcome through correction operations.

[0139] According to the embodiments of the present disclosure, the centroid offset of the X-ray image is accurately corrected using three-dimensional imaging technology, and the boundary of each piece of material in the three-dimensional image is used as a reference to correct the X-ray image, as the boundary of the valve opening, and then the accurate blowing instruction is calculated, so as to achieve precise control of the air valve and reduce the error rate of material sorting. In addition, when the material category is subsequently identified based on the X-ray image, the corrected centroid and material boundary can also be used as feature information for the material category, thereby providing more abundant information for identification, which is conducive to improving the accuracy of identification.

[0140] Figure 6 The flowchart of obtaining the centroid of a material in a three-dimensional image according to an embodiment of the present disclosure is schematically shown.

[0141] like Figure 6 As shown, this embodiment is one of the embodiments of operation S510, including:

[0142] In operation S610, the sum of the heights of all pixel points corresponding to the material in the three-dimensional image is calculated.

[0143] In operation S620, the width coordinates of all the pixels are weighted and summed, and then divided by the sum of the heights to obtain the centroid width coordinates; the length coordinates of all the pixels are weighted and summed, and then divided by the sum of the heights to obtain the centroid length coordinates.

[0144] In some embodiments, the weighted summation includes: taking the height of each pixel as a weight and multiplying its coordinate value to obtain a weighted result, where the coordinate value includes a width coordinate value or a length coordinate value; and summing the weighted results of all pixel points.

[0145] Specifically, assuming that the density of the material is uniform, the mass of a single pixel is equal to the density and the height of the pixel. Then the height of the pixel is proportional to the mass of the material in the pixel. The height information of the three-dimensional image is used as the mass information of the material, and the center of mass of the top view of the material can be obtained by the following formula:

[0146] The height of each pixel of each piece of material is denoted as mi. i refers to the i-th pixel, and i is an integer greater than or equal to 1.

[0147] Calculate the sum of the heights of all pixels of the material, denoted as M=∑mi.

[0148] Calculate the weighted average of the material width coordinates, using the height as the weight, and obtain the centroid width coordinate X, recorded as: .

[0149] Calculate the weighted average of the material's y coordinates, using the height as the weight, to get the centroid width coordinate Y, denoted as Y .

[0150] According to the embodiments of the present disclosure, the height information and boundary information of the three-dimensional image can be used to realize the centroid correction and boundary correction of the X-ray material. Thus, the accurate material centroid position and boundary information are obtained, and the precise positioning of the material is realized, so that a more accurate injection plan can be calculated.

[0151] In some embodiments, executing the second sorting strategy further includes: using the X-ray image and the three-dimensional image to identify the category of the second target material, wherein the second target material is a material to be identified that complies with the second sorting strategy.

[0152] Among them, the spraying based on the center of mass and boundary information of the material in the X-ray image includes:

[0153] The spraying parameters are determined based on the type of the second target material. For example, different materials may require different spraying pressures or modes, and the nozzle and spraying force (i.e., spraying parameters) suitable for the material on the spraying module can be intelligently selected according to the type of the material.

[0154] Based on the center of mass and boundary information of the material in the X-ray image, the spraying is performed according to the spraying parameters.

[0155] For example, first, the control module 500 uses the centroid position to calculate the motion trajectory of the material. Next, the control module 500 predicts the specific time when the material reaches the spraying position based on the motion trajectory, and further determines the time to start the spraying based on the boundary information of the material. When the material is about to reach the spraying position, the control module 500 sends a signal to start the spraying to the sorting module according to the predicted time.

[0156] The following Figure 7 and Figure 8 The process of executing the second sorting strategy is further described.

[0157] Figure 7 A flowchart for performing material matching based on point cloud data and calculating nozzle opening and closing time according to an embodiment of the present disclosure is schematically shown.

[0158] Reference Figure 3 and Figure 7 , and combined with reference Figure 2 , Figure 5 , Figure 6 After extracting point cloud data from the three-dimensional image, the PointNet++ algorithm is used to perform image segmentation (operation S220) to obtain a label image (i.e., a point cloud material image); the X-ray image is segmented (operation S220) to obtain a label image (i.e., a two-dimensional material image); the point cloud material image is matched with the two-dimensional material image one by one to achieve material matching between the two images (operation S230); based on the point cloud segmentation result of the PointNet++ algorithm, the center of mass position and boundary calculation of each material are performed (operation S510), and the center of mass position and boundary correction of the corresponding material in the X-ray image are performed (operation S520), thereby achieving accurate correspondence between the ore and the valve nozzle position, accurately calculating the nozzle opening and closing time, and accurately blowing the ore according to the blowing parameters.

[0159] Figure 8 The flowchart of performing material matching based on a depth image and calculating the nozzle opening and closing time according to an embodiment of the present disclosure is schematically shown.

[0160] Reference Figure 4 and Figure 8 , and combined with reference Figure 2 , Figure 5 , Figure 6, convert the point cloud data into a depth image to project the three-dimensional space information onto a two-dimensional plane, and then use the projection transformation matrix to register the depth image with the X-ray image (operation S410); perform image segmentation on the registered X-ray image and depth image respectively (operation S420); match the segmented two-dimensional material image and the depth material image one by one (operation S430); calculate the centroid position and boundary based on the depth information of each material (operation S510), and perform centroid position and boundary correction on the corresponding material in the X-ray image (operation S520), so that the nozzle opening and closing time can be accurately calculated based on the accurate centroid position and boundary. Subsequently, the spraying parameters can be determined based on the type of the material; based on the centroid and boundary information of the material, the spraying is performed according to the spraying parameters.

[0161] According to the embodiments of the present disclosure, accurate identification and classification of different types of target materials can be achieved, improving the accuracy and efficiency of sorting operations. Determining the injection parameters according to the type of target materials can better adapt to the conditions of different materials and improve the accuracy of sorting.

[0162] X-ray imaging (such as dual-energy imaging) can effectively distinguish materials with large differences in atomic numbers, but the closer the atomic numbers are, the lower the X-ray sorting effect will be. For example, some ores have similar colors, but the degree of concavity and convexity of the ore surface is quite different. Color cameras cannot present the concavity and convexity information of the ore surface well due to problems such as reflection and dispersion of the collected images. However, three-dimensional imaging technology can accurately measure the height information of the ore surface, thereby accurately reflecting the concavity and convexity information of the ore surface, making up for the defects of dual-energy X-ray imaging and color images.

[0163] In addition, the shapes of some concentrates and tailings are quite different. For example, most concentrates of phosphate ore are in the shape of cubes, while those with irregular shapes are mostly tailings. Since X-ray imaging is projection imaging, the imaged shape is a projection of a three-dimensional shape, and the height information is missing, so it is impossible to capture the three-dimensional shape characteristics.

[0164] For minerals with similar atomic numbers and unobvious color features, but with different particle shapes and surface convexity or three-dimensional shapes, the height information in the three-dimensional image effectively makes up for the shortcomings of X-ray imaging and color camera imaging in this regard. For ores with large texture convexity features or three-dimensional shapes, the introduction of three-dimensional images will effectively improve the recognition accuracy. Fig. 9 and Fig.10 An embodiment of identifying the category of a material from two different dimensions by using an X-ray image and a three-dimensional image is described.

[0165] Fig. 9 The flowchart for identifying the category of the second target material according to an embodiment of the present disclosure is schematically shown.

[0166] like Fig. 9 As shown, this embodiment includes:

[0167] In operation S910, the X-ray image is identified to obtain a first confidence score of the second target material, wherein the first confidence score represents the score of the second target material being of the first category. For example, a two-dimensional radiographic image of the second target material segmented from the X-ray image is identified.

[0168] In operation S920, the three-dimensional image is identified to obtain a second confidence score of the second target material, wherein the second confidence score represents the score of the second target material being of the first category. For example, the three-dimensional point cloud image of the second target material segmented from the three-dimensional image is identified.

[0169] In operation S930, a weighted calculation is performed based on the first confidence score and its first weight, and the second confidence score and its second weight to obtain a third confidence score.

[0170] In operation S940 , when the third confidence score is greater than or equal to the third value, the second target material is determined to be of the first category, otherwise it is of the second category.

[0171] For example, the material is ore, the first category is concentrate, and the second category is tailings.

[0172] The grade of the ore is continuous. The ore is identified as a concentrate with a confidence score_Xray by an X-ray image recognition algorithm (such as Faster R-CNN, YOLO network model, and U-Net network model). Set the first threshold threXray_high and the second threshold threXray_low. When score_Xray > threXray_high, score_Xray = 1; when score_Xray < threXray_low, score_Xray = 0. The first confidence score is score_Xray = 1, or score_Xray = 0. If score_Xray is between threXray_high and threXray_low, it is mapped to a value between 0 and 1.

[0173] At the same time, the point cloud data is recognized to classify the ore into two categories. For example, concave and convex tailings, smooth concentrates, or cubic concentrates, and tailings with irregular shapes. The ore passes through point cloud recognition algorithms (such as PointNet algorithm, VoxelNet, etc.) to obtain the confidence score score_PointCloud that the ore is recognized as concentrate. To convert the continuous confidence score into a discrete class label, the third threshold threPointClou_low and the fourth threshold threPointCloud_high are set. When score_PointCloud > threPointCloud_high, score_PointCloud = 1. When score_PointCloud < threPointCloud_low, score_PointCloud = 0. Among them, the second confidence score is score_PointCloud = 1 or score_PointCloud = 0. If score_PointCloud is between threPointClou_high and threPointClou_low, it is mapped to a value between 0 and 1.

[0174] First, perform a logical OR operation (denoted as ||) on score_Xray and score_PointCloud. If either score is true (i.e., greater than 0), the result of the entire logical OR expression is true. Then, multiply score_Xray by the first weight and divide it by the difference between threXray_high and threXray_low for normalization. Multiply score_PointCloud by the second weight and divide it by the difference between threPointCloud_high and threPointCloud_low for normalization. Add the above two normalized scores to obtain a comprehensive score. Therefore, let the third confidence score be denoted as score, and it is calculated by the following formula:

[0175]

[0176]

[0177]

[0178] Among them, the first weight a and the second weight b are the best parameter combinations obtained through the previous parameter optimization algorithm. For example, a and b are numbers between 0 and 1 respectively.

[0179] Finally, perform operation S940 to determine the final material category recognition result.

[0180] According to the embodiments of the present disclosure, the X-ray image recognition algorithm focuses on the internal structure and composition of the material, while the point cloud recognition algorithm focuses on the external shape and surface features of the material. Therefore, by analyzing and identifying the material from two different dimensions through X-ray images and three-dimensional images, the recognition accuracy can be improved.

[0181] In some embodiments, considering that materials in an adhesion state are related to the accuracy of secondary image segmentation, the material state can be used as one of the input features of the recognition algorithm during recognition.

[0182] For example, during the training and recognition process of the X-ray image recognition algorithm, the characteristic value of the material in the adhesion state is set to 1, and the characteristic value of the material in the non-adhesion state is set to 0. At the same time, for the material in the adhesion state, the concave point detection algorithm is also used to extract the characteristics of its edge concave points. These features, together with the overall image features of the two-dimensional material image, will be input into the X-ray image recognition algorithm to obtain the recognition result. In addition, during the training process, the optimal first threshold threXray_high and the second threshold threXray_low that match can be found according to the material state, and the optimal first weight a and the second weight b that match can be found according to the material state. During the recognition process, at least one of the first threshold threXray_high and the second threshold threXray_low can be dynamically changed according to the material state.

[0183] For another example, during the training and recognition process of the point cloud recognition algorithm, the characteristic value of the material in the adhesion state is set to 1, and the characteristic value of the material in the non-adhesion state is set to 0. At the same time, for the material in the adhesion state, the height continuity feature of its edge is extracted by height difference detection. These features, together with the overall image features of the two-dimensional material image, will be input into the point cloud recognition algorithm to obtain the recognition result. Similarly, during the training process, at least one of the optimal third threshold threPointClou_low and the fourth threshold threPointCloud_high that matches can be found according to the material state.

[0184] The X-ray image recognition algorithm and the point cloud recognition algorithm can be trained synchronously. During the training process, the optimal first weight a and second weight b can be found according to the material state. During the recognition process, at least one of the first weight a and the second weight b can be dynamically changed according to the material state.

[0185] In some embodiments, in a recognition scenario with three or more categories, a two-dimensional radiographic image of the second target material may be recognized to obtain a fourth confidence score for each category, and a three-dimensional point cloud image of the second target material may be recognized to obtain a fifth confidence score for each category. Then, the fourth confidence score and the fifth confidence score for each category are added together to obtain a sixth confidence score for each category. Then, the category with the highest score is selected as the final recognition category.

[0186] According to the embodiments of the present disclosure, the physical state of the material can be taken into account during the training and recognition process, thereby improving the accuracy of recognition. According to the corresponding thresholds and weights of the dynamic changes in the material state, targeted recognition of materials in different material states can be better achieved.

[0187] Fig.10 The flowchart for identifying the category of the second target material according to another embodiment of the present disclosure is schematically shown.

[0188] like Fig.10 As shown, this embodiment includes:

[0189] In operation S1010, the X-ray image is input into a first network branch of an image recognition model to extract a first feature; the first network branch can be constructed based on a convolutional neural network that processes two-dimensional images.

[0190] In operation S1020, the three-dimensional image is input into a second network branch of an image recognition model to extract a second feature, wherein the image recognition model is constructed based on a deep learning network;

[0191] Among them, the second network branch can be used to extract the features of the three-dimensional image point cloud data, and the rich features of the three-dimensional dimension can be extracted. The second network branch can be constructed based on the convolutional neural network for processing three-dimensional images. The three-dimensional image can also be converted into a depth image first, and then the second network branch is used to extract the features of the depth image, and the three-dimensional data processing is converted into two-dimensional data processing, reducing the complexity of the second network branch in processing data. The second network branch can be constructed based on the convolutional neural network for processing two-dimensional images.

[0192] In operation S1030, the first feature and the second feature are concatenated, and the concatenated features are input into a fully connected layer of the image recognition model; for example, the first feature and the second feature may be concatenated using a Concat operation, which is a data structure operation that connects two or more arrays or feature vectors together.

[0193] In operation S1040, the classification layer of the image recognition model processes the output result of the fully connected layer to identify the category of the second target material.

[0194] Exemplarily, the X-ray image and the three-dimensional image are respectively used as the input of the two branches of the deep learning model, and their respective features are extracted. Then, the feature maps of the X-ray image and the three-dimensional image are spliced ​​together to obtain a richer feature expression, which is converted into a new feature space through a fully connected layer. The output result of the fully connected layer is converted into a probability distribution using a softmax function in the classification layer to obtain the final recognition result. For example, in a two-classification scenario, the classification layer outputs the predicted concentrate probability and tailings probability, and the one with a higher probability is taken as the recognition result. In a three-classification scenario, the classification layer outputs the predicted concentrate probability, medium-quality ore probability, and tailings probability, and the one with a higher probability is taken as the recognition result. It can be understood that the present disclosure does not limit the number of categories output by the classification layer, and can output predicted probabilities corresponding to multiple pre-set categories.

[0195] According to the embodiment of the present disclosure, the first network branch and the second network branch are used to extract corresponding features from the two images, and the unique information of each image can be used. Therefore, by fusing the multimodal data of the X-ray image and the three-dimensional image, the target material features are fully captured, and the recognition accuracy is enhanced.

[0196] Exemplarily, an initial image recognition model may be acquired in advance, and the image recognition model may be trained based on a material image sample set, and the trained image recognition model may be applied to operations S1010 to S1040.

[0197] In some embodiments, considering that materials in an adhesion state are related to the accuracy of secondary image segmentation, the material state may be used as one of the input features of the image recognition model during recognition.

[0198] For example, in the training image recognition model stage, the material image sample set includes material image samples in an adhered state and material image samples in a non-adhesive state. In the training and recognition process, the feature value of the material in an adhered state is set to 1, and the feature value of the material in a non-adhesive state is set to 0. In operation S1030, in addition to splicing the first feature and the second feature, the feature value 1 or 0 representing the material state is also spliced, and the spliced ​​features are processed by the fully connected layer and the classification layer in sequence to output the material category.

[0199] According to the embodiments of the present disclosure, the adhesion state of the material may affect the accuracy of the secondary segmentation of the image, thereby affecting the recognition result. By introducing the characteristic value of the material state, a richer feature representation can be learned, and the model can take the material state into consideration during the training and recognition process, thereby improving the recognition accuracy of different material states.

[0200] Fig.11 A block diagram of an electronic device suitable for implementing a material sorting method according to an embodiment of the present disclosure is schematically shown. Fig.11 The electronic device shown is one embodiment of a control module 500 .

[0201] like Fig.11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 to a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0202] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other through bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0203] According to an embodiment of the present disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the I / O interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.

[0204] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0205] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.

[0206] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0207] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0208] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0209] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0210] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0211] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0212] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0213] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A material sorting method, characterized in that: include: Obtain X-ray images and three-dimensional images of the materials to be sorted; Based on at least one of the X-ray image and the three-dimensional image, identifying a material state of each material, wherein the material state includes a non-adhesion state and an adhesion state, and the adhesion state includes a state of contact with at least one other material; For the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material; if material state recognition has not been performed, image segmentation is directly performed on each material; Based on the material state of each material, each material in the image segmented X-ray image is matched with a corresponding material in the image segmented three-dimensional image; Material sorting is performed according to the material matching result between the X-ray image and the three-dimensional image.

2. The method according to claim 1, characterized in that Sorting materials according to the material matching result between the X-ray image and the three-dimensional image includes at least one of the following: If the material matching result shows that at least one of any two successfully matched materials is in a sticky state, and at least one of them has not been processed by image segmentation, the first sorting strategy is executed; If any material in the X-ray image or the three-dimensional image fails to match, executing the first sorting strategy for the material that fails to match; If the material matching result shows that at least one of any two successfully matched materials is in a sticky state and has been image segmented, the second sorting strategy is executed; If the material matching result shows that any two successfully matched materials are in a non-adhesive state, the second sorting strategy is executed.

3. The method according to claim 1, characterized in that Based on the material state of each material, matching each material in the image segmented X-ray image with a corresponding material in the image segmented three-dimensional image includes: Determining a correspondence between the X-ray image and a material in the three-dimensional image based on the X-ray data of the X-ray image and the point cloud data of the three-dimensional image; For any two materials having a corresponding relationship between the X-ray image and the three-dimensional image, if at least one of the two materials is in a state of adhesion, the first matching strategy is executed; if the two materials are not in a state of adhesion, the second matching strategy is executed.

4. The method according to claim 3, characterized in that Executing the first matching strategy includes: Calculating the overlap degree of the two materials, wherein the overlap degree is determined based on at least one of a two-dimensional projection shape and a two-dimensional projection area of ​​the materials; When the overlap degree is greater than or equal to the first value, it is determined that the two materials having a corresponding relationship are matched successfully.

5. The method according to claim 4, characterized in that When the overlap is less than the first value, determining at least one of the two materials as a first target material; Performing a merging degree detection on the first target material and the remaining N materials adjacent to each other in the same image, where N is an integer greater than or equal to 1; The first target material is merged with at least one specific material to obtain a merged material, wherein a merging degree between the specific material and the first target material is greater than or equal to a second value, and the merging process includes marking multiple materials as the same material for rematching.

6. The method according to claim 5, characterized in that When the first target material is a material in the three-dimensional image, performing a merging degree detection on the first target material and other N materials adjacent to each other in the same image includes: The first target material and the N materials are respectively subjected to a height continuity test, and the height continuity is positively correlated with the degree of merging.

7. The method according to claim 5, characterized in that Determining at least one of the two materials as the first target material comprises: comparing the two-dimensional projection area of ​​the material in the X-ray image and the two-dimensional projection area of ​​the material in the three-dimensional image; The material with a smaller two-dimensional projection area is used as the first target material.

8. The method according to claim 3, characterized in that Executing the second matching strategy includes: When a single material in the X-ray image forms a one-to-one correspondence with a single material in the three-dimensional image, confirming that the match is successful; If there is a correspondence between a single material in one of the X-ray image and a plurality of materials in the other image, the plurality of materials are merged into a single material to form a one-to-one correspondence, and the successful matching is confirmed.

9. The method according to claim 2, characterized in that: For the X-ray image and the three-dimensional image, if material state recognition has been performed, image segmentation is performed based on the material state of each material, and if material state recognition has not been performed, image segmentation is directly performed on each material, including: registering the X-ray image with a two-dimensional depth image, the two-dimensional depth image being obtained based on the three-dimensional image; Based on the material state of each material, segment the two-dimensional material image of each material from the registered X-ray image to obtain a third material set; directly segment the depth material image of each material from the registered depth image to obtain a fourth material set; Wherein, based on the material state of each material, matching each material in the image segmented X-ray image with a corresponding material in the image segmented three-dimensional image includes: The material images in the fourth material set are matched based on the material images in the third material set.

10. The method according to claim 2, characterized in that Executing the second sorting strategy includes: for any two materials that are successfully matched, Obtaining the centroid and boundary information of the material in the three-dimensional image; The spraying is performed based on the centroid and boundary information of the material in the three-dimensional image.

11. The method according to claim 10, characterized in that Obtaining the center of mass of the material in the three-dimensional image includes: Calculating the sum of the heights of all pixel points corresponding to the material in the three-dimensional image; The width coordinates of all the pixel points are weighted and then divided by the sum of the heights to obtain the centroid width coordinate; the length coordinates of all the pixel points are weighted and then divided by the sum of the heights to obtain the centroid length coordinate.

12. The method according to claim 11, characterized in that The weighted summation includes: The height of each pixel point is used as a weight and multiplied by its coordinate value to obtain a weighted result, wherein the coordinate value includes a width coordinate value or a length coordinate value; The weighted results of all the pixels are summed.

13. The method according to claim 10, characterized in that Executing the second sorting strategy also includes: using the X-ray image and the three-dimensional image to identify the category of a second target material, wherein the second target material is a material to be identified that complies with the second sorting strategy; Wherein, spraying based on the centroid and boundary information of the material in the three-dimensional image includes: determining a spraying parameter based on the type of the second target material; The spraying is performed according to the spraying parameters based on the centroid and boundary information of the material in the three-dimensional image.

14. The method according to claim 13, characterized in that Identifying the category of the second target material based on the X-ray image and the three-dimensional image includes: Identify the X-ray image to obtain a first confidence score of the second target material, wherein the first confidence score represents a score that the second target material is of the first category; Recognize the three-dimensional image to obtain a second confidence score of the second target material, wherein the second confidence score represents a score that the second target material belongs to the first category; Performing weighted calculation based on the first confidence score and its first weight, and the second confidence score and its second weight, to obtain a third confidence score; When the third confidence score is greater than or equal to a third value, the second target material is determined to be of the first category, otherwise it is of the second category.

15. The method according to claim 14, characterized in that Identifying the category of the second target material based on the X-ray image and the three-dimensional image includes: Inputting the X-ray image into a first network branch of an image recognition model to extract a first feature; Inputting the three-dimensional image into a second network branch of the image recognition model to extract a second feature, wherein the image recognition model is constructed based on a deep learning network; Concatenating the first feature and the second feature, and inputting the concatenated feature into a fully connected layer of the image recognition model; The classification layer of the image recognition model processes the output result of the fully connected layer to identify the category of the second target material.

16. A material sorting device, characterized in that: include: A conveying module, used for conveying materials to be sorted; An X-ray image acquisition module, used to capture the X-ray image of the material to be sorted; A three-dimensional image acquisition module, used to capture the three-dimensional image of the material to be sorted; A control module, configured to acquire the X-ray image and the three-dimensional image, and execute the material sorting method according to any one of claims 1 to 15 to generate a sorting instruction; The blowing module is used to blow the materials leaving the conveying module in response to the sorting instruction of the control module.

Images