Methods for identifying overlapping materials and methods for calculating material area.

Abstract

This invention discloses a method for identifying overlapping materials and calculating material area in the field of image processing technology. The identification method includes: acquiring a material conveying image and determining connected components in the material conveying image; the connected component is composed of at least one closed material region; performing instance segmentation on the connected component to obtain material regions within the connected component; performing erosion processing on the entire connected component image after segmentation of material regions, and randomly extracting any pixel from the eroded independent region; determining the number of the arbitrary pixel located within the connected component, and judging whether the connected component is overlapping material or independent material. This method can identify overlapping and independent materials on a conveyor belt, determine the stacking situation of materials during material transportation, thereby facilitating the calculation of material area and improving material sorting efficiency and production efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for identifying overlapping materials and a method for calculating the area of ​​the materials. Background Technology

[0002] Different sizes of Chinese medicinal herbs have different economic values. Manual sorting is extremely inefficient and difficult to handle processing large quantities of herbs. Related technologies introduce machine vision to actively identify herbs of different sizes, and then use devices such as spray valves to remove herbs of specific sizes. For sliced ​​herbs such as Astragalus membranaceus, conveyor belts are typically used in production environments. However, material stacking can occur on the conveyor belt, making it difficult to determine the area of ​​obscured material and thus hindering effective classification, thereby reducing sorting efficiency and production effectiveness. Summary of the Invention

[0003] One of the technical problems to be solved by the present invention is to provide a method for identifying and judging overlapping areas of materials.

[0004] The second technical problem to be solved by the present invention is to provide a method for accurately calculating the area of ​​materials.

[0005] To achieve the above objectives, a first aspect of the present invention provides a method for identifying overlapping materials, comprising:

[0006] Acquire a material transport image and determine the connected components in the material transport image; the connected components consist of at least one closed material region;

[0007] The connected component is segmented into instances to obtain the material region within the connected component;

[0008] Erosion processing is performed on the entire connected component image after segmenting the material region, and any pixel is randomly extracted from the eroded independent region.

[0009] Determine the number of any pixel points located within the connected region, and determine whether the connected region is an overlapping material or an independent material.

[0010] According to the method for identifying overlapping materials in embodiments of the present invention, by processing the material transport image, a connected region consisting of a closed area of ​​at least one material is determined, and then the connected region is further processed to count the number of any pixel points within the connected region, thereby determining whether the connected region belongs to overlapping materials or independent materials. This method can determine the stacking situation of materials during material transportation, which is beneficial for calculating the material area and improving the sorting effect and production efficiency of materials.

[0011] To achieve the above objectives, a second aspect of the present invention provides a method for calculating the area of ​​a material, comprising:

[0012] Based on the number of any pixel points within the connected region, it is determined whether the connected region is an overlapping material or an independent material, and the area of ​​the corresponding material is determined based on the calculation rules for overlapping materials or independent materials, respectively.

[0013] According to the material area calculation method of the present invention, by determining whether the connected region is an overlapping material or an independent material, the area of ​​the corresponding material is determined by using the corresponding area calculation rules. This method can accurately calculate the area of ​​the material in different stacking states, thereby effectively determining the category of the material, which is beneficial for sorting the material and improving the sorting effect and production efficiency.

[0014] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] Figure 1 This is an architecture diagram of an experimental platform;

[0016] Figure 2 This is a flowchart illustrating a method for identifying overlapping materials in one embodiment;

[0017] Figure 3 Image showing material transfer before and after corrosion in one embodiment;

[0018] Figure 4 This is a schematic diagram of sample data used to train an instance segmentation model in one embodiment;

[0019] Figure 5 This is the annotation result of the polygonal outline of the material in the sample data of one embodiment;

[0020] Figure 6 This is a schematic diagram of the material outline of a connected region in one embodiment;

[0021] Figure 7 This is a flowchart illustrating a method for calculating material area in one embodiment;

[0022] Figure 8 This is a flowchart illustrating the area calculation of an individual material in one embodiment;

[0023] Figure 9 Here are the RGB images before and after removing the background in one embodiment;

[0024] Figure 10 Here is an HSV image of a connected component in one embodiment;

[0025] Figure 11 Here is an HSV image of a connected component after erosion processing in one embodiment;

[0026] Figure 12 In one embodiment, the epidermal region is extracted based on color recognition.

[0027] Figure 13 This is a schematic diagram of the process for extracting the epidermal region in one embodiment;

[0028] Figure 14 This is a schematic diagram of the epidermal region after edge extraction in one embodiment;

[0029] Figure 15 This is a schematic diagram of contour annotation in the training data in one embodiment;

[0030] Figure 16 This is a schematic diagram of the structure of a look-ahead semantic segmentation network in one embodiment;

[0031] Figure 17 This is a schematic diagram of overlapping rows in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] like Figure 1 As shown, when sorting Chinese medicinal herbs such as Astragalus membranaceus, due to the irregular structure of the material, it is impossible to use a channel-type color sorter. Generally, a... Figure 1 The conveyor belt structure shown uses material transport and identification on the belt, followed by material rejection and sorting at the output end. However, due to the overlap of sheet-like medicinal materials during transport, accurate material identification is affected, thus impacting the sorting results. Therefore, accurately identifying overlapping materials and performing targeted sorting is a problem that needs to be solved.

[0034] The implementation details of the technical solutions in the embodiments of this application are described in detail below.

[0035] This embodiment provides a method for identifying overlapping materials, including:

[0036] Obtain material transport images and determine the connected components in the material transport images;

[0037] The connected component is segmented into instances to obtain the material region within the connected component;

[0038] Erosion processing is performed on the entire connected component image after segmenting the material region, and any pixel is randomly extracted from the eroded independent region.

[0039] Determine the number of any pixel points located within the connected region, and determine whether the connected region is an overlapping material or an independent material.

[0040] This embodiment performs instance segmentation on connected components in an image and combines it with image erosion operations to extract an arbitrary pixel from each segmented instance region. By determining the number of arbitrary pixels within the connected component, it identifies whether the region is an overlapping area, thus effectively recognizing overlapping materials. This allows for the collection and further processing of independent and overlapping materials as needed, improving sorting quality.

[0041] Specifically, in one embodiment, such as Figure 2 As shown, a method for identifying overlapping materials is provided, which may include the following steps:

[0042] Step S101: Obtain the material conveying image and determine the connected components in the material conveying image.

[0043] Here, the material conveying images are taken by industrial cameras, which can record the conditions of different materials on the conveyor belt.

[0044] For the acquired material handling images, the connected components in the images are calculated. To determine the connected components, a connected component labeling algorithm can be used to process the material handling images and label all connected components. In practical applications, functions provided by computer vision libraries (such as OpenCV) can also be used to calculate all connected components C in the material handling images. A single connected component c is denoted as .

[0045] The connected domain includes at least one material or multiple materials that have overlapping and / or adjacency relationships.

[0046] In practical applications, the intervals between two different materials on a conveyor belt vary. If the interval between two different materials is small, the boundary between the two materials will stick together. In the process of image processing, it is easy to confuse sticking materials with overlapping materials. Therefore, in order to accurately identify the connected components in the material conveying image, it is necessary to separate the sticking materials to prevent the connected components corresponding to each sticking material from being identified as the connected components corresponding to the overlapping materials.

[0047] Based on this, in one embodiment, the material transport image is subjected to etched processing to separate the adhered areas. For example... Figure 3 As shown, Figure 3 Images of the material transfer process before and after corrosion are shown. The specific processing steps for these material transfer images are as follows:

[0048] First, the material handling image is converted into a binary image of material handling. Here, all interconnected pixels forming a connected component are pixels with a value of 1 in the binary image. Therefore, the material handling image needs to be converted into a binary image before determining the connected components. (Reference) Figure 3 From the material conveying images, the blue portion of the image can be identified as the background. Therefore, the background color can be used to perform binary image conversion. The specific conversion process is as follows:

[0049] (1) Traverse each pixel of the material conveying image and obtain the components of the current pixel in the three channels of the RGB color space as r, g, and b.

[0050] (2) Calculate the proportion value of each pixel according to the given formula, where the formula is:

[0051]

[0052] The above formula calculates the proportion of the blue channel in the total color values ​​(the sum of blue, green, and red channels), where b, g, and r are the values ​​of the blue, green, and red channels of the current pixel, respectively. To avoid division by zero, a small constant of 10 is added to the formula. -7 .

[0053] (3) Set threshold conditions. `color_val = 70` is used to determine if the value of the blue channel is greater than this threshold. This threshold condition filters out pixels with higher blue channel values. Only when the blue value `b` of a pixel is greater than 70 will that pixel be considered a possible background, thus effectively excluding background pixels with high blue values. `val_ratio = 45` is also set to determine if the calculated ratio value is greater than this threshold. This threshold condition filters out pixels with a large proportion of blue in the overall color. Only when the ratio value is greater than 45 is that pixel considered part of the background.

[0054] (4) Generate a preliminary binary image based on the set threshold conditions, where dst = (b > color_val) & (ratio > val_ratio). Here, only pixels that simultaneously satisfy the condition that the blue channel is greater than 70 and the blue proportion is greater than 45 will be marked as background pixels.

[0055] (5) Invert the initially generated binary image, i.e., dst = ~dst. The inversion operation changes the pixels that meet the conditions (background) to 0 and the material points to 1. Thus, in the final binary image, the pixels with a value of 1 belong to the material points and the pixels with a value of 0 belong to the background.

[0056] Next, the generated binary material transport map (dst) is etched to separate the adhering materials. Specifically, in some embodiments, a 5x5 square etch kernel is used in the etch process to define the coverage area during the etch operation. The etch kernel is used to iterate the material transport map twice, where the number of iterations is set empirically.

[0057] Assuming the erosion symbol is - and the erosion kernel is K, for each pixel in the binary image dst of material transport, erosion is performed using the erosion kernel K, and the pixel value is updated. Then the operational rules for the erosion process are as follows: Under this operation rule, the pixel values ​​of a region will only be updated when the erosion kernel K completely covers the foreground region of the material transport binary map dst. The new value of each pixel is the minimum value of the pixels within the covered region. For example, if the pixel values ​​of all covered regions of the erosion kernel K are 1 (foreground), then the pixel values ​​of that region remain 1; if the pixel values ​​of any covered region are 0 (background), then the pixel values ​​of that region are updated to 0.

[0058] After two erosion operations, the adhered material regions in the material transport binary graph can be separated to form independent connected regions, which is beneficial for extracting connected regions from the material transport binary graph.

[0059] Step S102: Perform instance segmentation on the connected component to obtain the material region within the connected component.

[0060] A connected component typically contains one or two materials, and occasionally three. Therefore, it is necessary to determine the materials contained within the connected component. Here, by segmenting the connected component into instances, the boundary contours of each material in the connected component are identified, thereby obtaining all material regions within the connected component.

[0061] In practical applications, instance segmentation can be achieved using instance segmentation models. Instance segmentation models are deep learning models in computer vision designed to simultaneously perform object detection and pixel-level segmentation. Specifically, a Mask R-CNN instance segmentation model is built and then trained. During the training process, multiple datasets such as... Figure 4 The sample data shown is used as training data. Then, according to the instance segmentation requirements, the outlines of the materials in the sample data are labeled to obtain the following results: Figure 5 The annotation results of the material polygon outlines in the sample data shown are used to train the Mask R-CNN instance segmentation model, and finally the trained Mask R-CNN instance segmentation model is obtained.

[0062] The trained Mask R-CNN instance segmentation model can process connected components, identifying each material instance within the connected component and generating a segmentation mask for it. The mask is a binary image representing the shape of the material, where material regions are represented by 1s and background regions by 0s. Therefore, the final output of the Mask R-CNN instance segmentation model is as follows: Figure 6 The material outline of the connected domain shown is represented by a polygon, which represents the material region, that is, the position of the material in the connected domain.

[0063] It should be noted that the instance segmentation model can directly identify the closed regions corresponding to materials, including the closed regions corresponding to each material in overlapping materials. Specifically, the instance segmentation prediction result can effectively handle overlapping materials and distinguish different material instances, while the connected component analysis mainly focuses on identifying pixel regions in the material transport image and does not consider the distinction between overlapping materials or the instance information of the materials.

[0064] Step S103: Perform erosion processing on the entire connected component image after segmenting the material region, and randomly extract any pixel from the eroded independent region.

[0065] For humans, directly observing how many instance-segmented material regions are within a connected component can determine whether they are independent or overlapping materials. However, when using a computer for identification, such a simple judgment cannot be made. In this embodiment, an arbitrary point is extracted from each material region. By determining whether the arbitrary point is within the connected component, the number of arbitrary points within the connected component is determined, and thus the identification result of whether the connected component is an independent or overlapping material is obtained.

[0066] Understandably, based on the relationship between connected components and instance segmentation prediction results, large connected components can be divided into smaller material regions. When a large connected component contains multiple smaller material regions, the connected component is considered to contain overlapping materials. Based on this, an arbitrary pixel is selected within each material region. By analyzing the number of such arbitrary pixels within the connected component, the quantity of materials within the connected component is determined, thus indicating whether the connected component belongs to overlapping materials. However, the pixels representing materials cannot be simply extracted using mathematical formulas understood by the program. For example, points can be extracted by adding 1 to the coordinates of an edge point, or by averaging the coordinates of multiple boundary points, but the extracted points may not necessarily be within the material region.

[0067] Here, a method for extracting pixels belonging to materials through erosion is provided. Specifically, the connected component image after segmenting the material region is converted into a binary image, that is, the material is represented by a binary image, where pixels inside the material region have a value of 1 (foreground) and pixels outside have a value of 0 (background). The conversion process can refer to the steps in the above embodiment for converting the material transport image into a material transport binary image. Then, the entire connected component image converted to a binary image is eroded. Erosion causes the boundaries to shrink inward, thereby removing some small noise and fine connections, changing the shape and size of the connected component and the material region within it, so that any point in the eroded material region is within the material region segmented in the original instance. A 5*5 square erosion kernel can be used for erosion. After the erosion is completed, any pixel is extracted from the eroded independent region within the connected component. The extracted pixel belongs to the material within that connected component.

[0068] Step S104: Determine the number of any pixel points located within the connected component, and determine whether the connected component is an overlapping material or an independent material.

[0069] By counting the number of any pixel within a connected component, we can determine how many materials are contained within that component, and thus determine whether the connected component consists of overlapping or independent materials.

[0070] It is understandable that only one pixel is extracted for each material region. If there is only one pixel in a connected region, then the connected region contains only one material, that is, the connected region is an independent material; if there are multiple pixels in a connected region, then the connected region contains multiple materials, that is, the connected region is an overlapping material.

[0071] In practical applications, besides determining whether a connected component belongs to overlapping or independent materials based on the number of any pixels within it, other methods can be used to determine whether a connected component is overlapping. For example, shape feature analysis can be performed on the instance segmentation prediction results to determine the independence of objects based on these features, or by calculating the degree of overlap.

[0072] In one embodiment, during pixel extraction, the number of iterations of the erosion operation is randomly determined. M (M≥1) erosion operations can be performed on the connected component image after instance segmentation. The erosion operation may shrink the connected components and material regions. The remaining polygonal regions within the connected components are the material regions retained after the erosion operation. Pixels located within the remaining polygonal regions of the connected components all belong to the material. A material pixel is randomly selected from these pixels for subsequent determination of overlapping or independent materials. After extracting the material pixels, the extracted material pixels from all material regions within the same connected component are summarized and calculated. Specifically, all selected pixels are traversed one by one and assigned to the corresponding connected components based on their coordinate values, thus counting the pixels contained in the connected component. This counts the number of material pixels extracted from the same connected component, reflecting the number of material regions contained in the same connected component, and is used for subsequent determination of whether the connected component belongs to overlapping materials.

[0073] In one embodiment, the number of iterations M in the erosion operation is a definite number. M iterations of the erosion operation are performed to find the last pixel in the eroded material region. Based on this, if the (M+1)th erosion operation is performed on the connected component image after instance segmentation, then there exists a material region within the connected component where all pixels have been eroded to 0. Therefore, a random material pixel is selected from the remaining material region after the Mth erosion operation. This ensures that any selected material pixel is within the material region, and the selected pixel can be used to accurately determine whether the connected component belongs to overlapping or independent materials. The number of iterations M for selecting random pixels is different for different material regions after instance segmentation, thus gradually extracting random pixels from each material region as the erosion progresses.

[0074] In one embodiment, based on the above-described method for identifying overlapping materials, a method for calculating the area of ​​materials is also proposed.

[0075] like Figure 7 As shown, the method for calculating the area of ​​materials may include the following steps:

[0076] Step S201: Based on the number of any pixel in the connected component, determine whether the connected component is an overlapping material or an independent material.

[0077] Step S202: Determine the area of ​​the corresponding material based on the calculation rules for overlapping materials or independent materials respectively.

[0078] Here, a connected component is composed of at least one closed material region; that is, a connected component may contain one material or multiple overlapping materials. Therefore, it is necessary to determine whether a connected component contains overlapping or independent materials. Understandably, if a connected component contains only one material, it contains only one material region; conversely, if a connected component contains overlapping materials, it contains multiple material regions. Based on this, an arbitrary pixel is extracted from each material region within the connected component. By counting the number of extracted arbitrary pixels within the connected component, if the number of arbitrary pixels is 1, it indicates that the connected component contains only one material region, belonging to independent materials; if the number of arbitrary pixels is greater than 1, it indicates that the connected component contains multiple material regions, belonging to overlapping materials.

[0079] The method for determining whether a connected region is an overlapping material can be referred to the specific description of the relevant embodiments of the overlapping material identification method described above, and will not be repeated here.

[0080] After determining whether a connected region consists of overlapping or independent materials, the area of ​​the materials within the connected region is calculated according to the calculation rules for overlapping or independent materials. It is understandable that when the connected region consists of independent materials, the outlines of the materials and their corresponding areas are clearly visible; therefore, the area of ​​each independent material can be directly calculated from its subgraph. However, when the connected region consists of overlapping materials, the outlines of the materials are obscured due to occlusion between them. Further processing of the subgraph is required to calculate the area of ​​each material. Therefore, the area calculation methods differ for different types of connected regions, and the specific calculation process will be explained through different embodiments.

[0081] In one embodiment, such as Figure 8 As shown, Figure 8 The diagram illustrates the process of calculating the area of ​​an independent material. Connected regions typically include a light yellow cut surface and a dark yellow skin; the skin's influence on the area needs to be removed when calculating the material's area.

[0082] Step S301: Binarize the material conveying image, separate the background based on the region of the connected components, and extract the RGB image of the connected components.

[0083] like Figure 9 As shown, Figure 9The images before and after background removal are shown. Material conveying images are RGB images, and the background color in these images is blue. Therefore, the intensity of the blue channel and its proportion in the RGB color space can be used to determine which pixels in the material conveying image belong to the background. Pixels belonging to the background are set to 0 (black), while foreground pixels remain unchanged. This allows for background separation in the material conveying image based on the connected component regions. The specific process for removing the background can be found in the background separation formula used in the above embodiment when converting the material conveying image to a binary image.

[0084] After binarization, the background in the material conveying image can be separated based on the regions of the connected components, thus allowing the extraction of the RGB image of the connected components from the material conveying image, denoted as img. hq The connected component RGB image is extracted from the material conveying image based on the minimum positive bounding moment of the connected component. The minimum positive bounding moment is a rectangle that completely encloses the connected component. Therefore, the coordinates of the top-left corner of the connected component (x...) are... min ,y min ) and the coordinates of the lower right corner (x max ,y max The formula for calculating ) is:

[0085] x min =minx i

[0086] x max =maxx i

[0087] y min =miny i

[0088] y max =maxy i

[0089] In the above formula, the minimum x-coordinate of all pixels in the connected domain represents the leftmost boundary of the minimum positive bounding moment, the maximum x-coordinate of all pixels in the connected domain represents the rightmost boundary of the minimum positive bounding moment, the minimum y-coordinate of all pixels in the connected domain represents the uppermost boundary of the minimum positive bounding moment, and the maximum y-coordinate of all pixels in the connected domain represents the lowermost boundary of the minimum positive bounding moment.

[0090] In the material handling image, the coordinates of pixel p are (x... i ,y i These coordinates are relative to the entire material transport image. When a connected component is extracted, the coordinates of the top-left corner of the connected component are (x...). min ,y minTherefore, it is evident that the coordinate system of the material conveying image differs from the coordinate system of the RGB image of the connected component. The coordinates of this pixel need to be updated to be relative to the RGB image of the connected component. Given that the coordinates of pixel p are (x... i ,y i The updated coordinates are:

[0091] x′ i =x i -x min

[0092] y′ i =y i -y min

[0093] Where, x′ i and y′ i These are the updated coordinates. Assuming pixel p has coordinates (150, 200) in the material conveying image and the top-left corner of the connected component has coordinates (100, 150), the new coordinates of pixel p in the connected component can be determined as (50, 50) according to the above relationship.

[0094] Step S302: Convert the RGB image of the connected components to the HSV space, and perform erosion processing on the resulting HSV image.

[0095] Then, the RGB image of the connected components is converted to HSV space to obtain the HSV image of the connected components after removing the background, denoted as img. hq-hsv ,like Figure 10 As shown, Figure 10 The HSV image of the connected components is shown. The HSV color space can separate color information (hue) from lightness information (brightness), making color processing more intuitive. For the surface and cut surfaces of materials, hue can help to better distinguish different color areas.

[0096] Next, a 3x3 square is selected as the erosion kernel to perform erosion processing on the HSV image, as follows: Figure 11 As shown, Figure 11 The image shows an HSV image of a connected component after erosion. The erosion operation applies an erosion kernel to each pixel of the HSV image, checks the pixel values ​​within the kernel's coverage area, and if all pixels covered by the kernel are foreground, the center pixel remains foreground; otherwise, the center pixel is set to background. By eroding the HSV image, the edges of connected components shrink inward, removing pixels where the edges blend with the background. This helps to clearly separate material from the background and reduces edge noise.

[0097] In one embodiment, the erosion process is performed only once, meaning that each pixel is processed only once.

[0098] Step S303: Extract the skin region from the connected region based on the color difference between the cut surface and the skin of the material.

[0099] Extracting the skin region from the eroded HSV image can be achieved through color segmentation. Since there is a significant color difference between the cut surface and the skin, meaning their color characteristics are not identical, thresholds for hue, saturation, and lightness in the HSV color space can be set based on the color characteristics of the skin region. Typically, the skin color may fall within a specific hue range; therefore, a mask can be generated using the defined HSV range to extract the skin regions that meet the criteria.

[0100] In one embodiment, the saturation (s) and brightness (v) of each pixel are extracted from the HSV image. If the HSV component of a pixel satisfies the condition (sv) > thr, it is considered a skin pixel; otherwise, it is considered a material pixel. Here, thr is set to 0. Based on this, the process of extracting candidate skin pixels can be represented as follows:

[0101] img hq-hsv-edge =img hq-hsv &&(sv)>thr

[0102] In the above formula, img hq-hsv-edge HSV image representing the epidermal region, img hq-hsv The HSV image representing the connected components is denoted as S, and the extracted epidermal region is denoted as S. edge When the value of (sv) is greater than thr, it indicates that the saturation of that pixel is relatively high compared to its brightness, usually meaning that the pixel is a region with a bright color. Since thr is set to 0, this condition actually judges whether the saturation is greater than the brightness. Therefore, only when a pixel in the material's HSV image meets the set condition (sv) > thr will that pixel be retained in the saturation. edge In the middle. For example Figure 12 As shown, Figure 12 The epidermal region extracted based on color recognition is shown.

[0103] In one embodiment, considering that the cut surface of the material may exhibit circular textures similar in color to the skin, relying solely on color might incorrectly extract the non-skin portion of the material. Given that the skin always appears at the edge of the material, additional location information is introduced, using both color features and location information to extract the skin region of the material. For example... Figure 13 As shown, the extraction process for the epidermal region is as follows:

[0104] Step S401: Take any contour point in the skin area of ​​the material edge. If the pixel is in the connected region, add it to the set of skin pixels to be removed.

[0105] Here, for any contour point within the skin region of the material edge, it is determined whether the pixel is within a connected component. If so, it is added to the set S of skin pixels to be removed. edge-c The inner part indicates that the point belongs to the skin that needs to be removed.

[0106] Step S402: Traverse the eight neighboring pixels. If a pixel is located within the skin region based on color recognition and has not been visited, add it to the set of skin pixels to be removed. Repeat the step of traversing the neighborhood of the pixel to search until there are no new matching pixels to be removed in the neighborhood. Use the area of ​​all pixels in the set of skin pixels to be removed as the area of ​​the skin region.

[0107] Traverse the eight neighborhood pixels of any contour point p = (x, y), where p = (x, y) and its eight neighborhood pixels are (x-1, y), (x+1, y), (x, y-1), (x, y+1), (x-1, y-1), (x-1, y+1), (x+1, y-1), and (x+1, y+1). If these pixels are unvisited and located in the color-based skin region S... edge If these pixels are inside, then they are added to the set S of the surface pixels to be removed. edge-c , where set S edge-c The pixels within this range are those of the skin region after edge extraction. If, during the search process, neighboring pixels are found to have already been visited or do not belong to the skin region S... edge If the value is not found, skip it and proceed to find the next neighboring pixel.

[0108] Repeat the search for eight pixels in the neighborhood of each arbitrary contour point until no new matching pixels to be removed exist in the neighborhood. This indicates that the search is complete. Figure 14 As shown, Figure 14 The epidermal region after edge extraction is shown.

[0109] After completing the search, the set S of skin pixels to be removed will be... edge-c The area of ​​all pixels within the set S is taken as the area of ​​the skin region, i.e., determined by the set S. edge-c All the pixels in the graph constitute the skin region in the connected domain, and the area of ​​the skin region can be calculated using these pixels.

[0110] For some materials, the color of the entire product is not completely consistent, and there may even be some areas inside that are similar in color to the surface. Therefore, when identifying by color alone, the outer ring and some areas inside may be identified as the surface area. By taking any point on the outer edge and searching for continuous surface areas based on the neighborhood, we can avoid the internal material points being incorrectly rejected and improve the accuracy of the peeling results.

[0111] Step S304: The material area is obtained based on the difference between the area of ​​the connected region and the area of ​​the skin region.

[0112] The connected domain is divided into the cut surface and the skin region. The area corresponding to the cut surface region represents the actual area of ​​the material. Based on this, the material area S hq Equal to the area S of the connected region cc Subtract the area S of the epidermal region edge-c .

[0113] It should be noted that, in addition to Astragalus membranaceus, if the actual area calculation of other independent materials also needs to consider the edge area, the area calculation method of independent materials in the above embodiments is also applicable.

[0114] In one embodiment, a method for calculating the area of ​​overlapping materials is provided.

[0115] First, extract the contours of the connected components. Here, we can refer to step S301 to extract the RGB image of the connected components from the material conveying image.

[0116] An RGB image of a connected component records images of multiple materials on a conveyor belt that partially or completely overlap. In cases of material overlap, some edges may be occluded, making it difficult to accurately identify the complete outline and determine which parts belong to which materials, thus affecting the calculation of the material area. Therefore, calculating the material area of ​​overlapping materials requires segmenting each material within the overlapping material.

[0117] Here, a front- and back-look semantic segmentation network is pre-trained. This network is a deep learning model designed to handle material segmentation tasks in images and can identify the contours of each material in overlapping materials.

[0118] It should be noted that the training of the foreground and background semantic segmentation network depends on labeled training data. In routine segmentation modeling processes, such as... Figure 15In the outline annotation diagram of the training data shown, during the regular annotation process, both occluded and occluded materials are annotated along their outlines. The segmentation network also focuses on the visible material outlines from the current viewpoint. When a pixel is spatially segmented due to material overlap, causing a sudden change in its gradient value, it is predicted as part of the material outline. Therefore, when calculating the area based on this prediction, the connected component area of ​​the occluded material is smaller than its actual connected component, resulting in a significant decrease in sorting performance. Furthermore, since the trained front-view and back-view semantic segmentation network can segment overlapping materials from both front-view and back-view perspectives, the annotation of the training data also needs to adapt to the training objectives of the front-view and back-view semantic segmentation network. Here, as... Figure 15 In the schematic diagram of contour annotation shown, during the annotation process of training data, the occluded materials in the training data are still annotated using the conventional method, that is, annotated according to the contour of the visible part of the occluded material. For the occluded materials in the training data, the shape of the occluded material is simulated from the rear view angle based on the visible area of ​​the occluded material. This means that the possible shape and contour of the occluded material need to be considered, and then the contour of the occluded area of ​​the occluded material is inferred and annotated along the contour of the occluded material.

[0119] After the training data is labeled, the labeled training data is input into the front- and back-look semantic segmentation network to be trained, so that the front- and back-look semantic segmentation network can learn the occlusion relationship of overlapping materials, thereby having the ability to segment occluded materials and occluded materials.

[0120] The working principle of the front-and-back semantic segmentation network is explained below. It is understandable that in the case of overlapping materials, some materials may be occluded by others, making it difficult to accurately identify and segment each material. The front-and-back semantic segmentation network has the ability to learn the occlusion relationships between overlapping materials. It can model the occlusion relationships of overlapping materials in the input material sub-image, that is, it can identify which parts of the overlapping materials are occluded and which parts are visible. This modeling of occlusion relationships helps improve the accuracy of segmentation. By modeling occlusion relationships, the front-and-back semantic segmentation network can decouple overlapping materials, that is, it can process overlapping materials separately, allowing the features of each material to be identified individually. In this case, the front-and-back semantic segmentation network can extract the contour of each material from two different perspectives (front view and back view). Understandably, in the forward view, some materials may be obscured by other materials, while in the rear view, these obscured parts may be revealed. Therefore, the forward and rear views provide different viewing angles, allowing the forward and rear view semantic segmentation networks to capture different features of the materials from different perspectives. The forward view provides the frontal features of the materials, while the rear view provides the shape and position of the obscured parts. By combining the information from these two perspectives and utilizing the complementarity of multi-view data, the forward and rear view semantic segmentation networks can more accurately identify the true outline of each material.

[0121] like Figure 16 As shown, Figure 16 A schematic diagram of the front-view and back-view semantic segmentation network is shown, which includes a downsampling network (N1), a front-view upsampling network (N2), and a back-view upsampling network (N3). The downsampling network includes four convolutional modules with a stride of 2 and three C3 layers. The four convolutional modules are respectively a convolutional module with 3 input channels and 64 output channels, a convolutional module with 64 input channels and 128 output channels, a convolutional module with 128 input channels and 256 output channels, and a convolutional module with 256 input channels and 512 output channels. The three C3 layers are respectively a C3 layer with a depth of 4, a C3 layer with a depth of 6, and a C3 layer with a depth of 2. The forward upsampling network and the backward upsampling network include 5 convolutional layers with a stride of 1, 3 C3 layers with a depth of 3, and 4 upsampling layers. The 5 convolutional layers are respectively a convolutional layer with 512 input channels and 256 output channels, a convolutional layer with 256 input channels and 128 output channels, a convolutional layer with 128 input channels and 64 output channels, a convolutional layer with 64 input channels and 32 output channels, and a convolutional layer with 32 input channels and 2 output channels. The 3 C3 layers are a C3 layer with 256 output channels, a C3 layer with 128 output channels, and a C3 layer with 64 output channels.

[0122] It's important to note that convolutional modules and convolutional layers in segmentation models are two different structures. A convolutional module (ConvBNSiLU) is a common neural network layer combination, integrating convolutional layers (Conv), batch normalization (BN), and the SiLU (Sigmoid Linear Unit) activation function. A convolutional layer (Conv) is a standard convolutional layer, containing only convolution operations. While activation functions can be added after convolutional layers, the Conv layer itself does not contain an activation function. Furthermore, the C3 layer in segmentation models is a modular network structure, typically including residual blocks (bottlenecks), skip connections, and feature fusion. The depth of the C3 layer refers to the number of bottlenecks it contains. The C3 layer can divide the input data into two parts. One part of the input data is processed by Bottleneck, while the other part skips Bottleneck processing. Then, the data processed by Bottleneck is concatenated with the data that has not been processed by Bottleneck. Optionally, after concatenation, a convolutional layer can be used to perform convolution processing on the concatenated data to fuse features and avoid information redundancy.

[0123] The total loss L during model training is divided into two parts: the first loss value L caused by the output of the forward upsampling network. 前 The second loss value L caused by the output of the back-look upsampling network 后 Among them, the first loss value L 前 The second loss value L is calculated at the output layer of the forward-looking upsampling network. 后 It is calculated at the output layer of the back-look upsampling network. Both use cross-entropy as the loss function. For each channel c, the cross-entropy loss is defined as:

[0124]

[0125] Among them, y c p represents the true label, indicating the true value of category c. c It is the probability value predicted by the model, representing the probability that the sample belongs to category c, where C is the total number of channels.

[0126] Then, by adjusting the first loss value L 前 Second loss value L 后 After performing a weighted summation, the final total loss L is obtained, which is defined as:

[0127] L=(1-w)*L 前 +w*L 后

[0128] Here, w is set to 0.7.

[0129] Here, the total loss L is weighted and summed to balance the fore-look and back-look losses during training. This ensures that the back-look loss has a greater impact when segmenting occluded materials, thus improving the robustness and accuracy of the segmentation network. The total loss L is used as the optimization objective to update the parameters of the segmentation network through backpropagation to minimize the loss and improve the segmentation performance.

[0130] During the inference phase, the acquired material sub-images are input into the front- and back-look semantic segmentation network for processing. The front- and back-look semantic segmentation network can output the corresponding segmentation results, that is, the contour of each material in the overlapping materials.

[0131] In practical applications, overlapping materials can consist of two or more materials. To more clearly illustrate the data processing process of the look-ahead semantic segmentation network, the following example uses overlapping materials consisting of two materials, combined with... Figure 16 The data processing of the foreground and background semantic segmentation network is explained.

[0132] The main task of a downsampling network is to extract deep features from the connected components of the input. Downsampling networks typically include multiple convolutional layers (Conv) and max pooling layers. Through these layers, the resolution of the feature map is progressively reduced while the number of channels is increased to capture richer feature information. The specific process by which a downsampling network extracts the first deep features of a sub-map is as follows:

[0133] i) The material sub-image (img) has a size of H×W×3. The downsampling network performs the first convolution on the material sub-image through a convolutional module (ConvBNSiLU). In the first convolution, ConvBNSiLU has 3 input channels, 64 output channels, and a stride of 2, thereby reducing the length and width of the feature map to half of its original size. That is, the size of the feature map output by ConvBNSiLU is [size missing]. ConvBNSiLU is a common neural network layer combination that integrates convolutional layers, batch normalization (BN), and the SiLU (Sigmoid Linear Unit) activation function. Next, a C3 layer of depth 4 is applied to increase the network's receptive field while maintaining the same resolution and number of channels, resulting in a structure of size [size missing]. The feature map.

[0134] ii) Using ConvBNSiLU to measure the size of The feature map is subjected to a second convolution. In this second convolution, ConvBNSiLU has 64 channels as input, 128 channels as output, and a stride of 2. This reduces the length and width of the feature map to one-quarter of its original size. In other words, the second convolution will output a size of... The feature map is then processed through a C3 layer of depth 6 to obtain a feature map of size . The feature map.

[0135] iii) Using ConvBNSiLU to process the size of The feature map is subjected to a third convolution. In this third convolution, ConvBNSiLU has 128 channels as input, 256 channels as output, and a stride of 2. This reduces the length and width of the feature map to 1 / 8 of its original size. In other words, the third convolution will output a size of... The feature map is then processed through a C3 layer of depth 2, resulting in a feature map of size [size missing]. The feature map.

[0136] iv) Finally, the size is obtained by using ConvBNSiLU. The feature map is subjected to a fourth convolutional layer. In this fourth convolution, ConvBNSiLU has 256 channels as input, 512 channels as output, and a stride of 2. This reduces the length and width of the feature map to 1 / 16 of its original size, meaning the fourth convolution will output a size of [missing value]. The feature map.

[0137] Through the above steps, the downsampling network completes the deep feature extraction of connected components, and the final output feature map is at 1 / 16 resolution with 512 channels. This process effectively extracts important features.

[0138] The forward upsampling network, based on the downsampling network, upsamples the first deep feature output by the downsampling network. After upsampling, the forward upsampling network concatenates the upsampled feature map (i.e., the first upsampled result) with the first deep feature extracted by the downsampling network, ultimately restoring the feature map resolution to the same size as the input material sub-image and outputting the outline of the occluded material from the forward viewpoint. The specific processing procedure of the forward upsampling network for the feature map is as follows:

[0139] i) Downsampling networks through convolutional layers (Conv) The feature map undergoes a first convolutional layer. In this first layer, Conv has 512 channels as input, 256 channels as output, and a stride of 1, resulting in a new 1 / 16 feature map. Next, nearest neighbor interpolation is used to modify the output of Conv. The feature map undergoes a first-level upsampling (2x upsampling) to obtain a size of... Feature map. The feature map obtained from the first layer upsampling. Feature maps and downsampling networks The feature maps are concatenated along the channel dimension to obtain... Feature maps are generated to increase the number of features describing the image. After passing through a C3 layer with a depth of 3 and half the number of output channels, a size of [size missing] is obtained. The feature map.

[0140] ii) Next, the C3 output of the forward upsampling network... The feature map undergoes a second convolution. In this second convolution, Conv has 256 channels as input, 128 channels as output, and a stride of 1, resulting in a new 1 / 8 feature map. Then, the output of Conv is processed using nearest neighbor interpolation. The feature map is upsampled a second time (2x upsampling) to obtain a size of Feature map. The feature map obtained by upsampling from the second layer. Feature maps and downsampling networks The feature maps are concatenated along the channel dimension, and then passed through a C3 layer of depth 3, with half the number of output channels, to obtain... Feature map.

[0141] iii) Subsequently, the C3 output of the forward upsampling network... The feature map undergoes a third convolution. In this third convolution, Conv has 128 channels as input, 64 channels as output, and a stride of 1, resulting in a new 1 / 4 feature map. Then, nearest neighbor interpolation is used to refine the Conv output. The feature map is upsampled at the third layer (2x upsampling) to obtain a size of Feature map. The feature map obtained by upsampling from the third layer. Feature maps and downsampling networks The feature maps are concatenated along the channel dimension, and then passed through a C3 layer of depth 3, with half the number of output channels, to obtain... Feature map.

[0142] iv) Subsequently, the C3 output of the forward upsampling network... The feature map undergoes a fourth convolution. In this fourth convolution, Conv has 64 channels as input, 32 channels as output, and a stride of 1, resulting in a new half-feature map. Then, nearest-neighbor interpolation is used to modify the Conv output. The feature map is upsampled at the fourth layer (2x upsampling) to obtain a feature map of size H×W×32. The H×W×32 feature map is passed through a convolutional layer with 32 input channels, C output channels, and a stride of 1 to obtain the output result about the front view, i.e., the outline of the occluded material.

[0143] Forward-looking analysis typically refers to observing overlapping materials from the front. Overlapping materials can be categorized into occluding materials and occluded materials. Positionally, the occluding material is located in front of the occluded material, allowing the occluding material to partially obscure the occluded material from the front-looking perspective. Forward-looking upsampling networks process extracted deep features to segment overlapping materials from the front-looking angle, thus obtaining the outline of the occluded material from the front-looking perspective. In practical applications, the C value in the forward-looking upsampling network is set to 2, indicating that each pixel has two possible categories: one is the occluding material in the forward-looking view, and the other is either the background or the occluded material in the forward-looking view.

[0144] Through the above steps, forward upsampling can effectively restore image resolution, enhance segmentation accuracy, and adapt to complex occlusion scenarios by progressive upsampling and multi-scale feature stitching, thereby achieving segmentation of overlapping materials from the forward viewpoint.

[0145] The back-view upsampling network, based on the front-view upsampling network, upsamples the first deep features output by the front-view upsampling network. After upsampling, the back-view upsampling network concatenates the upsampled feature map (i.e., the second upsampling result) with the second deep features extracted by the back-view upsampling network. Finally, it restores the resolution of the feature map to the same size as the input material sub-image and outputs the outline of the occluded material from the back view. The specific processing procedure of the back-view upsampling network for the feature map is as follows:

[0146] i) Output of the downsampling network Based on the feature maps, the back-look upsampling network performs upsampling and stitching operations on the feature maps using a similar process to the front-look upsampling network. The difference lies in the prediction difficulty. Since predicting occluded materials is relatively easy, the front-look upsampling network can directly stitch the first deep feature extracted by the downsampling network (which already contains multi-scale visual information about the materials) together. However, predicting occluded materials is more challenging, requiring the back-look upsampling network to stitch the second deep feature (which already contains relevant information about the occluded materials) together with the front-look upsampling network to fully utilize the occlusion relationships of the target region already learned by the front-look upsampling network.

[0147] ii) The final output of the back-view upsampling network is the prediction result for the back-view, the size of which is the same as the original. Figure 1 The number of channels C is 2, which means that each pixel has two possible categories: one is the occluded material in the rear view, and the other is the background or the occluded material in the rear view.

[0148] Rear view usually refers to observing overlapping materials from the back. It can be understood that when observing overlapping materials from the front, it is difficult to observe the part of the obscured object due to the presence of obstructing objects; however, when observing overlapping materials from the rear, the part of the obscured object can be observed, thus obtaining the outline information of the obscured object.

[0149] The outlines of both the occluding and hidden materials can be extracted and can be represented by a series of coordinate points, indicating the shape of the material boundaries. Using the outline of each material, the area of ​​each material is calculated. There are several methods for area calculation; for example, ready-made functions provided by image processing libraries can be used to calculate the area of ​​materials. These functions can directly receive the outline data of the materials and return the corresponding area.

[0150] It should be noted that the material area actually comprises the cut surface area and the skin area. When calculating the area of ​​the obscured material, the influence of the skin area on the material area needs to be considered. That is, the skin area within the obscured material needs to be determined, and then the actual area of ​​the obscured material is obtained based on the difference between the area of ​​the material area and the area of ​​the skin area. However, the influence of the skin area can be ignored when calculating the area of ​​the obscured material.

[0151] In the above embodiments, by determining whether the connected region is an independent material or an overlapping material, and then determining the area of ​​the material according to the calculation rules for independent or overlapping materials, the area of ​​independent or overlapping materials can be accurately calculated, which is beneficial for the effective sorting of independent and overlapping materials, thereby improving the material sorting effect and production efficiency.

[0152] In practical applications, as the material moves with the conveyor belt, the distance between the material and the left and right boundaries of the conveyor belt remains constant, and the material will not exceed the left and right boundaries. Therefore, the material will not be truncated at the left and right edges in material transport images acquired at different times. However, the distance between the material and the top and bottom boundaries of the conveyor belt changes as the conveyor belt moves. Therefore, the top and bottom edges of the material transport images acquired at different times may be truncated. If the material edges in the material transport images are incomplete, it is difficult to accurately calculate the material area.

[0153] Based on this, in one embodiment, materials with complete edges are acquired through overlapping rows. The industrial camera saves a material transport image every fixed number of rows. Therefore, the industrial camera can first acquire a material transport image at a first-time interval, and then acquire the next material transport image at a fixed-row interval (i.e., a second-time interval). Two temporally adjacent material transport images contain overlapping rows of width L, where the overlapping rows represent the same image area in the two adjacent material transport images.

[0154] When there are materials with incomplete edges in a material handling image, the material with complete edges can be obtained in the next adjacent material handling image by using the overlapping rows of two adjacent material handling images. Here, the overlapping rows contain the image area corresponding to the material with incomplete edges. It should be noted that the height of the overlapping rows should be greater than the height of the positive bounding moment of a material. This means that if the material is truncated at the bottom edge of the previous material handling image, the material will still be fully displayed at row H2 of the next adjacent material handling image.

[0155] like Figure 17 As shown, Figure 17 A schematic diagram of overlapping rows is shown, in which, Figure 17 The image contains Figure A and Figure B. Taking Figure A as an example, the total number of rows H in Figure A can be divided into image rows H1 and overlapping rows H2 (i.e., H = H1 + H2). Image rows H1 are located from the first to the H1th row of the image, and overlapping rows H2 are located from the H1th to the H2th row of the image. Overlapping rows H2 in Figure B are located from the first to the H2th row of the image. Therefore, material 1 in Figure A corresponds to material 3 in Figure B, and material 2 in Figure A also corresponds to material 4 in Figure B.

[0156] The specific process for obtaining materials with intact edges is as follows: Identify the coordinates of the positive circumscribed moments corresponding to the materials in the overlapping rows. Since the materials are not truncated at the left and right edges, we can further obtain only the coordinates of the top and bottom edges of the positive circumscribed moments corresponding to the materials, denoted as y. min and y max And the top and bottom edge coordinates Y of the currently acquired material conveying image (Figure A) 1-0 and Y 1-1 And the top and bottom edge coordinates Y of the next adjacent material transfer image (Figure B). 2-0 and Y 2-1 By using y min and y max respectively with Y 1-0 Y 1-1 Y 2-0 and Y 2-1 By comparing the images, it is possible to determine in which material transport image the material was fully recorded.

[0157] Taking material number 1 as an example, if the coordinates of material number 1 in the current material transfer image satisfy y min >Y 1-0 &&y max <Y 1-1 At this point, Figure A shows the complete material No. 1, so the area calculation of material No. 1 is based on the material area in Figure A.

[0158] Taking material No. 2 as an example, if the coordinates of the material in Figure A satisfy y min >Y 1-0 &&y max =Y 1-1 At this point, material 2 is cut off at the bottom edge of Figure A and will appear near row H2 of Figure B. That is, material 4 in Figure B is the complete shape of material 1 in the first material transfer picture. Therefore, the area calculation of material 2 is based on the material area in Figure B.

[0159] Taking material No. 3 as an example, if the coordinates of material No. 3 in Figure B satisfy y min =Y 2-0 At this point, the material is cut off at the top edge of Figure B, but it has already appeared completely near row H1 in Figure A. Therefore, the area calculation of material No. 3 is based on the material area in Figure A.

[0160] In the above embodiments, by setting overlapping rows, the problem of material truncation caused by image acquisition frequency can be effectively avoided. In this way, even if there is a truncation at the bottom edge of a material conveying image in adjacent material conveying images, the next material conveying image can still capture the complete material, thereby enabling the extraction of a complete material sub-image.

[0161] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0162] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0163] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0164] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0165] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for identifying overlapping materials, characterized in that, include: Acquiring a material conveying image and determining the connected components in the material conveying image includes: converting the material conveying image into a material conveying binary image based on the color information in the material conveying image; and performing erosion processing on the material conveying binary image to separate the adhering materials in the material conveying image. The connected component is segmented into instances to obtain the material region within the connected component; Erosion processing is performed on the entire connected component image after segmenting the material region. Randomly extract any pixel from the eroded independent region. This includes: performing an erosion operation on the connected component image after segmentation of the instance for M iterations, so that the number of pixels in each segmented material region is less than or equal to a preset threshold; when the erosion operation of a certain material region in the connected component reaches the M+1th iteration, its number of pixels is exactly 0. Determine the number of any pixel points located within the connected region, and determine whether the connected region is an overlapping material or an independent material.

2. The method for identifying overlapping materials according to claim 1, characterized in that, The process involves eroding the entire connected component image after segmenting the material region, and randomly extracting any pixel from the eroded independent region, including: Randomly select a material pixel within the remaining material area; The material pixels extracted from all material regions within the same connected domain are summed and counted.

3. The method for identifying overlapping materials according to claim 1, characterized in that, after the Mth erosion operation, a material pixel is randomly selected from the image of the material region.

4. The method for identifying overlapping materials according to claim 1, characterized in that, There are overlapping rows in two adjacent material transport images.

5. A method for calculating the area of ​​overlapping materials based on the method for identifying overlapping materials according to any one of claims 1-4, characterized in that, Based on the number of any pixels within the connected component, it is determined whether the connected component is an overlapping material or an independent material, and the area of ​​the corresponding material is determined based on the calculation rules for overlapping materials and independent materials, respectively; wherein, The area of ​​the independent material is calculated based on the difference between the area of ​​the connected region and the area of ​​the skin region; The area of ​​the overlapping materials is obtained by extracting the contours of the connected components, modeling the occlusion relationship of the overlapping materials based on the trained front- and back-view semantic segmentation network, decoupling and extracting the contours of the occluding materials and the occluded materials from the front and back views respectively, and calculating the corresponding areas respectively; wherein, when calculating the area of ​​the occluding materials, the surface region is removed, and when calculating the area of ​​the occluded materials, it is calculated based on the material contours.

6. The material area calculation method according to claim 5, characterized in that, When the connected domain is an independent material, it includes: The material conveying image is binarized, the background is separated according to the region of the connected component, and the RGB image of the connected component is extracted. The RGB image of the connected components is converted to HSV space, and the resulting HSV image is then subjected to erosion processing. Based on the color difference between the cut surface and the skin of the material, the skin region is extracted from the connected domain; The material area is obtained by calculating the difference between the area of ​​the connected region and the area of ​​the skin region.

7. The material area calculation method according to claim 6, characterized in that, The method for erosion processing of HSV images is to use an erosion kernel of 3. A 3-square shape undergoes one erosion operation; The method for extracting the skin region is as follows: if the HSV component of a pixel satisfies SV>thr, then it is a skin point; otherwise, it is a material point, where thr is an empirical threshold.

8. The material area calculation method according to claim 6, characterized in that, extracting the skin region from the connected region based on the color difference between the cut surface and the skin of the material, further includes, Take any contour point within the skin region of the material edge. If the pixel is within a connected component, add it to the set of skin pixels to be removed. Traverse the eight neighboring pixels. If a pixel is located within a color-based skin region and has not been visited, add it to the set of pixels to be removed. Repeat the search until there are no new pixels in the neighborhood that meet the criteria. The area of ​​all pixels in the set of pixels to be removed is then taken as the area of ​​the skin region.

9. The material area calculation method according to claim 5, characterized in that, When the connected component consists of overlapping materials, the process includes: extracting the contours of the connected component; modeling the occlusion relationship of the overlapping materials based on a trained front- and back-look semantic segmentation network; decoupling and extracting the contours of the occluding and occluded materials from the front and back views respectively; and calculating the corresponding areas for each. Specifically, modeling the occlusion relationship of the overlapping materials based on a trained front- and back-look semantic segmentation network, and decoupling and extracting the contours of the occluding and occluded materials from the front and back views respectively, includes: The segmentation model includes a downsampling network, a front-view upsampling network, and a back-view upsampling network. Based on the downsampling network, a first deep feature about the connected components is extracted. The front-view upsampling network then upsamples the first deep feature based on the downsampling network, and the first upsampling result is concatenated with the first deep feature to output the outline of the occluded material in the front view. The back-view upsampling network then upsamples the first deep feature based on the front-view upsampling network, and the second upsampling result is concatenated with the second deep feature extracted by the front-view upsampling network to output the outline of the occluded material in the back view.

Images