Recognition method of overlapped materials and material area calculation method

Abstract

The invention discloses an overlapped material identification method and a material area calculation method in the technical field of image processing, and the identification method comprises the steps: obtaining a material transmission picture, and determining a connected domain in the material transmission picture; the connected region is composed of at least one closed material region; performing instance segmentation on the connected domain to obtain a material area in the connected domain; carrying out corrosion processing on the whole connected domain image after the material region is segmented, and randomly extracting any pixel point from the corroded independent region; and determining the number of any pixel points in the connected domain, and judging whether the connected domain is an overlapped material or an independent material. By means of the method, the overlapped materials and the independent materials on the conveying belt can be recognized, the stacking condition of the materials can be determined in the material conveying process, then calculation of the material area is facilitated, and the sorting effect and production efficiency of the materials are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for identifying overlapping materials and a method for calculating the area of ​​materials. Background Art

[0002] Chinese herbal medicines of different sizes have different economic values. Manual sorting is extremely inefficient and difficult to cope with the processing scenarios of a large number of Chinese herbal medicines. In related technologies, machine vision technology is introduced to actively identify Chinese herbal medicines of different sizes, and then spray valves and other equipment are used to eliminate Chinese herbal medicines of specific sizes. For flaky medicinal materials such as astragalus, conveyor belts are generally used for transportation in production environments. Materials may be stacked on the conveyor belts, and it is difficult to determine the area of ​​the obscured materials, resulting in the inability to effectively determine the category to which they belong, thereby reducing the sorting effect and production efficiency of the materials. Summary of the invention

[0003] One of the technical problems to be solved by the present invention is to provide a method capable of identifying and judging the overlapping areas of materials.

[0004] The second technical problem to be solved by the present invention is to provide a method for accurately calculating the area of ​​a material.

[0005] In order to achieve the above object, a first embodiment of the present invention provides a method for identifying overlapping materials, comprising:

[0006] Acquire a material transfer picture, and determine a connected domain in the material transfer picture; the connected domain is composed of at least one closed material area;

[0007] Performing instance segmentation on the connected domain to obtain a material region in the connected domain;

[0008] The connected domain image after segmenting the material area is corroded as a whole, and any pixel point is randomly extracted from the independent area after corrosion;

[0009] The number of the arbitrary pixel points located in the connected domain is determined, and the connected domain is judged to be an overlapping material or an independent material.

[0010] According to the overlapping material recognition method of the embodiment of the present invention, by processing the material transmission picture, a connected domain composed of a closed area of ​​at least one material is determined, and then the connected domain is processed, and the number of arbitrary pixel points in the connected domain is counted, and then it is determined whether the connected domain belongs to overlapping materials or independent materials. The stacking situation of materials can be determined during the material transportation process, which is beneficial to the calculation of the material area and improves the material sorting effect and production efficiency.

[0011] To achieve the above object, a second embodiment of the present invention provides a material area calculation method, comprising:

[0012] Based on the number of the arbitrary pixel points in the connected domain, the connected domain is judged to be an overlapping material or an independent material, and the area of ​​the corresponding material is determined based on the calculation rules of the overlapping material or the independent material.

[0013] According to the material area calculation method of the embodiment of the present invention, by judging whether the connected domain is an overlapping material or an independent material, and using the corresponding area calculation rule to determine the area of ​​the corresponding material, the area of ​​the material in different stacking states can be accurately calculated, thereby effectively determining the category of the material, which is beneficial to sorting the material and improving the material sorting effect and production efficiency.

[0014] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an architecture diagram of an experimental platform;

[0016] Figure 2 is a schematic flow chart of a method for identifying overlapping materials in one embodiment;

[0017] Figure 3 Transmitting pictures of the material before and after corrosion in one embodiment;

[0018] Figure 4 is a schematic diagram of sample data used to train an instance segmentation model in one embodiment;

[0019] Figure 5 is a labeling result of a material polygonal contour in sample data in an embodiment;

[0020] Figure 6 A schematic diagram of a material profile of a connected domain in one embodiment;

[0021] Figure 7 A schematic diagram of a flow chart of a material area calculation method in one embodiment;

[0022] Figure 8 A schematic diagram of a process for calculating the area of ​​an independent material in one embodiment;

[0023] Fig. 9 RGB images before and after background removal in one embodiment;

[0024] Fig.10 is an HSV image of a connected domain in one embodiment;

[0025] Fig.11 is an HSV image of a connected domain after corrosion processing in one embodiment;

[0026] Fig.12 The epidermal region extracted based on color recognition in one embodiment;

[0027] Fig.13 A schematic diagram of a process for extracting an epidermal area in one embodiment;

[0028] Fig.14 A schematic diagram of an epidermal region after edge extraction in one embodiment;

[0029] Fig.15 A schematic diagram of contour annotation in training data in one embodiment;

[0030] Fig.16 Schematic diagram of the structure of a front-back view semantic segmentation network in one embodiment;

[0031] Fig.17 Schematic diagram of overlapping rows in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] like Figure 1 As shown in the figure, when sorting Chinese medicinal materials such as Astragalus, due to the irregular structure of the materials, it is impossible to use a channel type color sorter for sorting. Figure 1 The crawler structure shown in the figure is used for transmission, and the materials are identified on the crawler, and the materials are rejected and sorted at the output end of the crawler. However, since the sheet-shaped medicinal materials overlap during the transmission process, it will affect the accurate identification of the materials, and thus affect the sorting results. Therefore, how to accurately identify the overlapping materials and sort them in a targeted manner is a problem that needs to be solved.

[0034] The implementation details of the technical solution of the embodiment of the present application are described in detail below.

[0035] This embodiment provides a method for identifying overlapping materials, including:

[0036] Acquire a material transfer picture, and determine a connected domain in the material transfer picture;

[0037] Performing instance segmentation on the connected domain to obtain a material region in the connected domain;

[0038] The connected domain image after segmenting the material area is corroded as a whole, and any pixel point is randomly extracted from the independent area after corrosion;

[0039] The number of the arbitrary pixel points located in the connected domain is determined, and the connected domain is judged to be an overlapping material or an independent material.

[0040] This embodiment performs instance segmentation processing on the connected domain in the image, combines it with the image corrosion operation, extracts an arbitrary pixel point from each segmented instance area, and determines whether it is an overlapping area by judging the number of arbitrary pixels in the connected domain. This effectively realizes the identification of overlapping materials, so that independent materials and overlapping materials can be collected and further processed according to needs, thereby improving the sorting quality.

[0041] Specifically, in one embodiment, Figure 2 As shown, a method for identifying overlapping materials is provided, and the method may include the following steps:

[0042] Step S101, obtaining a material transfer picture, and determining a connected domain in the material transfer picture.

[0043] Here, the material transfer picture is taken by an industrial camera, which can record the conditions of different materials on the conveyor belt.

[0044] For the acquired material transfer picture, the connected domain in the material transfer picture is calculated. For the determination of the connected domain, the connected domain marking algorithm can be used to process the material transfer picture and mark all the connected domains. In practical applications, the functions provided by the computer vision library (such as the OpenCV library) can also be used to calculate all the connected domains C in the material transfer picture. A single connected domain c is recorded as

[0045] The connected domain includes at least one material or multiple materials that are overlapping and / or adjacent to each other.

[0046] In practical applications, the intervals between two different materials on a conveyor belt vary. If the interval between two different materials is small, the boundaries of the two materials will be adhered to each other. In the process of image processing, the adhered materials are easily confused with the overlapping materials. Therefore, in order to accurately identify the connected domains in the material transmission image, the adhered materials need to be separated to prevent the connected domains corresponding to the adhered materials from being identified as the connected domains corresponding to the overlapping materials.

[0047] Based on this, in one embodiment, the material transfer image is subjected to corrosion treatment to separate the adhesion area. Figure 3 As shown, Figure 3 The material transfer pictures before and after corrosion are shown. The specific processing process of the material transfer picture is as follows:

[0048] First, the material transfer picture is converted into a binary image of material transfer. Among them, all the interconnected pixels that constitute the connected domain refer to the pixels with a value of 1 in the binary image. Therefore, before determining the connected domain, the material transfer picture needs to be converted into a binary image. Figure 3 In the material transmission picture, the blue part in the material transmission picture can be determined as the background, so the background color can be used to realize the conversion of the binary image. The conversion process is as follows:

[0049] (1) Traverse each pixel of the material transfer image to obtain the components of the current pixel in the three channels of the RGB color space, which are r, g, and b.

[0050] (2) Calculate the ratio value of each pixel according to the given formula, where the formula is:

[0051]

[0052] In the above formula, the proportion of the blue channel in the total color value (the sum of the blue, green and red channels) is calculated, where b, g, and r are the values ​​of the blue, green and red channels of the current pixel respectively. In order to avoid division by zero, a small constant 10 is added to the above formula -7 .

[0053] (3) Setting the threshold condition, where color_val=70 is set to determine whether the value of the blue channel is greater than the threshold. This threshold condition selects pixels with higher blue channel values. When the blue value b of a pixel is greater than 70, the pixel is considered to be a possible background, thereby effectively excluding background pixels with higher blue values. Val_ratio=45 is also set to determine whether the calculated ratio value is greater than the threshold. This threshold condition selects pixels with a larger proportion of blue in the overall color. Only when the ratio value is greater than 45, the pixel is considered to be part of the background.

[0054] (4) Generate a preliminary binary image according to the set threshold conditions, where dst = (b>color_val) & (ratio>val_ratio. Here, only pixels that satisfy both the blue channel greater than 70 and the blue ratio greater than 45 will be marked as background points.

[0055] (5) Perform a negation operation on the initially generated binary image, that is, dst = ~dst. The negation operation changes the pixels that meet the conditions (background) to 0 and the material points to 1, so that in the final binary image, the pixels with a value of 1 belong to the material points, and the pixels with a value of 0 belong to the background.

[0056] Next, the generated binary image of material transfer (i.e., dst) is etched to separate the adhering materials. Specifically, in some embodiments, the etch kernel used in the etch process is a 5*5 square, which is used to define the coverage area in the etch operation. The etch kernel is used to iterate the etch process twice on the binary image of material transfer, where the number of iterations is set according to experience.

[0057] Assume that the erosion symbol is - and the erosion kernel is K. For each pixel in the binary image dst of the material transfer, the erosion kernel K is used to perform erosion processing and update the pixel value. Then the operation rule of the erosion process is: Under this operation rule, it means that only when the corrosion kernel K completely covers the foreground area of ​​the material transfer binary image dst, the pixel value of the area will be updated, and the new value of each pixel is the minimum value of the pixels in the covered area. For example, if the pixel values ​​of all covered areas of the corrosion kernel K are 1 (foreground), the pixel value of the area remains 1; if the pixel value of any covered area is 0 (background), the pixel value of the area is updated to 0.

[0058] After two corrosion operations, the material areas that are adhered in the material transfer binary map can be separated to form independent connected domains, which is conducive to extracting connected domains from the material transfer binary map.

[0059] Step S102, performing instance segmentation on the connected domain to obtain the material region in the connected domain.

[0060] A connected domain may generally contain one or two materials, and occasionally three materials, so it is necessary to determine the materials contained in the connected domain. Here, by performing instance segmentation on the connected domain, the boundary contour of each material in the connected domain is identified, thereby obtaining all material regions in the connected domain.

[0061] In practical applications, instance segmentation can be achieved through instance segmentation models. The instance segmentation model is a deep learning model in the field of computer vision that aims to perform object detection and pixel-level segmentation at the same time. Specifically, a Mask R-CNN instance segmentation model is built, and then the instance segmentation model is trained. During the training process of the instance segmentation model, multiple Figure 4 The sample data shown in the figure is used as training data. Then, according to the instance segmentation requirements, the contours of the materials in the sample data are labeled to obtain the following Figure 5 The labeling results of the material polygon contours in the sample data shown are used to train the Mask R-CNN instance segmentation model using the labeled sample data, and finally a trained Mask R-CNN instance segmentation model is obtained.

[0062] The trained Mask R-CNN instance segmentation model can process the connected domain, identify each material instance in the connected domain and generate a segmentation mask for it. The mask refers to a binary image used to represent the shape of the material, where the material area is 1 and the background area is 0. The final output of the Mask R-CNN instance segmentation model is as follows: Figure 6 The material outline of the connected domain shown is represented by a polygon, which represents the material area, that is, the position of the material in the connected domain.

[0063] It should be noted that the instance segmentation model can directly identify the closed area corresponding to the material, including the closed area corresponding to each material in the overlapping materials. Among them, the instance segmentation prediction result can effectively handle overlapping materials and distinguish different material instances, while the connected domain mainly focuses on identifying the pixel area in the material transmission picture, does not focus on the distinction of overlapping materials, and does not consider the instance information of the material.

[0064] Step S103, corroding the entire connected domain image after segmenting the material area, and randomly extracting any pixel point from the independent area after corrosion.

[0065] For humans, by directly observing how many material regions are segmented by instances in a connected domain, it is possible to know whether it is an independent material or an overlapping material. However, when using a computer for recognition, it is impossible to make such a simple judgment. In this embodiment, it is proposed to extract an arbitrary point in each material region, and by judging whether the arbitrary point is in the connected domain, the number of arbitrary points in the connected domain is judged, and then the recognition result of whether the connected domain is an independent material or an overlapping material is obtained.

[0066] It is understandable that, based on the relationship between the connected domain and the instance segmentation prediction results, a large connected domain can be segmented into small material regions. When there are multiple small material regions in a large connected domain, the connected domain is considered to be an overlapping material. Based on this, an arbitrary pixel point is selected in each material region, and the number of such arbitrary pixel points in the connected domain is analyzed to determine the number of materials in the connected domain, thereby determining whether the connected domain belongs to an overlapping material. However, the pixel points of the material cannot be simply extracted through mathematical formulas understood by the program. For example, points can be extracted by adding 1 to the coordinates of a certain edge point, or by averaging the coordinates of multiple boundary points. However, the extracted points are not necessarily in the material region.

[0067] Here, a method of extracting the pixel points belonging to the material by corrosion processing is provided. Specifically, the connected domain image after the material area is segmented is converted into a binary image, that is, the material is represented by a binary image, the pixel value inside the material area is 1 (foreground), and the pixel value outside is 0 (background), wherein the conversion process here can refer to the steps of converting the material transfer picture into the binary image of material transfer in the above embodiment. Then the connected domain image converted into a binary image is corroded as a whole, and the corrosion processing will cause the boundary to shrink inward, thereby removing some small noise and small connecting parts, so that the shape and size of the connected domain and the material area in the connected domain change, so that any point of the corroded material area is within the material area segmented by the original instance. Among them, the corrosion processing can be performed using a square with a corrosion kernel of 5*5 in size. After the corrosion processing is completed, any pixel point is extracted from the independent area after corrosion in the connected domain, wherein the pixel point extracted here is the pixel point belonging to the material in the connected domain.

[0068] Step S104, determining the number of arbitrary pixel points located in the connected domain, and determining whether the connected domain is an overlapping material or an independent material.

[0069] Here, by counting the number of arbitrary pixel points in a connected domain, it is possible to determine how many materials are contained in the connected domain, and then determine whether the connected domain belongs to overlapping materials or independent materials.

[0070] It can be understood that only one pixel point is extracted from each material area. If there is one pixel point in the connected domain, the connected domain only contains one material, that is, the connected domain is an independent material; if there are multiple pixels points in the connected domain, the connected domain contains multiple materials, that is, the connected domain is an overlapping material.

[0071] In practical applications, in addition to judging whether a connected domain belongs to overlapping materials or independent materials based on the number of arbitrary pixels in the connected domain, other methods can also be used to determine whether a connected domain is an overlapping material, such as by performing shape feature analysis on the instance segmentation prediction results, judging the independence of objects based on these features, or by calculating the degree of overlap.

[0072] In one embodiment, in the process of extracting pixel points, the number of iterations of the corrosion operation is randomly determined. The connected domain image after instance segmentation can be iteratively performed M (M≥1) corrosion operations. The corrosion operation may shrink the connected domain and the material area, wherein the remaining polygonal area in the connected domain is the material area retained after the corrosion operation. The pixel points located in the remaining polygonal area of ​​the connected domain all belong to the material, and a material pixel point is randomly selected from them for subsequent judgment of overlapping materials or independent materials. After the extraction of material pixel points is completed, the material pixel points extracted from all material areas in the same connected domain are summarized and calculated. The specific method is to traverse all selected pixel points one by one, and assign them to the corresponding connected domain according to the coordinate value, so as to realize the counting of the pixel points contained in the connected domain. That is, the number of material pixel points extracted from the same connected domain is counted, which reflects the number of material areas contained in the same connected domain, and is used for the subsequent judgment of whether the connected domain belongs to overlapping materials.

[0073] In one embodiment, the number of iterations M of the corrosion operation is a clear number, and the M-times corrosion operation is performed to find the last pixel point of the corroded material area. On this basis, if the M+1th corrosion operation is performed on the connected domain image after instance segmentation, then there is a material area in the connected domain whose pixel points have been corroded to 0, so a random material pixel point in the remaining material area during the Mth corrosion operation is taken, which can ensure that any selected material pixel point must be in the material area, and the material pixel point can be used to accurately determine whether the connected domain belongs to overlapping materials or independent materials. For different material areas after instance segmentation, the number of iterations M for selecting random pixel points is different, so that as the corrosion proceeds, random pixel points of each material area are gradually extracted.

[0074] In one embodiment, based on the above-mentioned overlapping material identification method, a material area calculation method is also proposed.

[0075] like Figure 7 As shown, the material area calculation method may include the following steps:

[0076] Step S201: Based on the number of any pixels in a connected domain, determine whether the connected domain is an overlapping material or an independent material.

[0077] Step S202, determining the area of ​​the corresponding material based on the calculation rules of the overlapping materials or the independent materials.

[0078] Here, the connected domain is composed of at least one closed material area, that is, the connected domain may contain one material, or may contain multiple overlapping materials. Therefore, it is necessary to determine whether the connected domain is an overlapping material or an independent material. It can be understood that if the connected domain is a separate material, then the connected domain only contains one material area, and if the connected domain is an overlapping material, then the connected domain will contain multiple material areas. Based on this, an arbitrary pixel point is arbitrarily extracted from each material area in the connected domain, and the number of arbitrary pixel points extracted in the connected domain is counted. If the number of arbitrary pixel points in the connected domain is 1, it means that the connected domain only contains one material area and belongs to an independent material; if the number of arbitrary pixel points in the connected domain is greater than 1, it means that the connected domain contains multiple material areas and belongs to overlapping materials.

[0079] The method for determining whether the connected domain is an overlapping material may refer to the specific description of the above-mentioned overlapping material identification method embodiment, which will not be repeated here.

[0080] After determining whether the connected domain belongs to overlapping materials or independent materials, the material area in the connected domain is calculated according to the calculation rules of overlapping materials or independent materials. It can be understood that when the connected domain is an independent material, the outline of the material and the corresponding area are clearly visible, so the material area corresponding to the independent material can be directly calculated based on the material sub-graph. In the case where the connected domain is an overlapping material, due to the occlusion between the materials, the outline of the material is blocked, and the material sub-graph needs to be further processed to calculate the material area corresponding to each material. Based on this, the area calculation method used for different types of connected domains is different, and the specific calculation process will be explained through different embodiments.

[0081] In one embodiment, Figure 8 As shown, Figure 8 The figure shows the process flow of area calculation of independent materials. The connected domain usually includes the light yellow cut surface and the dark yellow skin. When calculating the area of ​​the material, the influence of the skin on the area needs to be removed.

[0082] Step S301, binarize the material transfer picture, separate the background according to the area of ​​the connected domain, and extract the RGB image of the connected domain.

[0083] like Fig. 9 As shown, Fig. 9The RGB images before and after the background is removed are shown. The material transfer picture belongs to the RGB image, and the color of the background in the material transfer picture is blue, so the intensity of the blue channel and its proportion in the RGB color space can be used to determine which pixels in the material transfer picture belong to the background, and the pixels belonging to the background are set to 0 (black), while the foreground pixels remain unchanged, so that the background in the material transfer picture can be separated according to the area of ​​the connected domain. Among them, the specific process of removing the background can refer to the separation background formula used in the process of converting the material transfer picture into a binary image in the above embodiment.

[0084] After the binarization process is completed, the background in the material transfer picture can be separated according to the area of ​​the connected domain, so that the RGB image of the connected domain can be extracted from the material transfer picture, which is recorded as img hq The connected domain RGB image is extracted from the material transfer image according to the minimum positive circumscribed moment of the connected domain. The minimum positive circumscribed moment is a rectangle that can completely surround the connected domain. The coordinate of the upper left corner of the connected domain (x min ,y min ) and the lower right corner coordinate (x max ,y max ) is calculated as:

[0085] x min =minx i

[0086] x max =maxx i

[0087] y min =miny i

[0088] y max =maxy i

[0089] In the above formula, the minimum x-coordinate of all pixels in the connected domain represents the leftmost boundary of the minimum positive circumscribed moment, the maximum x-coordinate of all pixels in the connected domain represents the rightmost boundary of the minimum positive circumscribed moment, the minimum y-coordinate of all pixels in the connected domain represents the uppermost boundary of the minimum positive circumscribed moment, and the maximum y-coordinate of all pixels in the connected domain represents the lowermost boundary of the minimum positive circumscribed moment.

[0090] In the material transfer picture, the coordinates of pixel point p are (x i ,y i ), these coordinates are relative to the entire material transfer image. When a connected domain is extracted, the coordinates of the upper left corner of the connected domain are (x min ,y min), it can be seen that the coordinate system of the material transfer image is different from the coordinate system of the RGB image of the connected domain, and the coordinates of the pixel point need to be updated to the coordinates relative to the RGB image of the connected domain. i ,y i ), the updated coordinates are:

[0091] x′ i =x i -x min

[0092] y′ i =y i -y min

[0093] Among them, x′ i and y′ i is the updated coordinate. Assuming that the coordinates of pixel point p in the material transfer image are (150,200), and the coordinates of the upper left corner of the connected domain are (100,150), according to the above relationship, the new coordinates of pixel point p in the connected domain can be determined to be (50,50).

[0094] Step S302: convert the RGB image of the connected domain into the HSV space, and perform corrosion processing on the obtained HSV image.

[0095] Then, the RGB image of the connected domain is converted to the HSV space to obtain the HSV image of the connected domain after removing the background, which is recorded as img hq-hsv ,like Fig.10 As shown, Fig.10 The HSV image of the connected domain is shown. The HSV color space can separate color information (hue) from brightness information (brightness), making it more intuitive when processing colors. For the surface and cut surface of the material, hue can help better distinguish different color areas.

[0096] Next, a square of size 3*3 is selected as the erosion core to erode the HSV image, as shown in Fig.11 As shown, Fig.11 The HSV image of the connected domain after corrosion processing is shown. The corrosion operation applies the corrosion kernel to each pixel of the HSV image, checks the pixel values ​​in the area covered by the corrosion kernel, and if all pixels covered by the corrosion kernel are foreground, the center pixel remains as foreground; otherwise, the center pixel is set to background. By corroding the HSV image, the edge of the connected domain shrinks inward, thereby removing the pixels where the edge of the connected domain blends with the background, which helps to clearly separate the material from the background and reduce edge noise.

[0097] In one embodiment, the erosion process is performed only once, which means that each pixel is processed only once.

[0098] Step S303: extracting the surface area in the connected domain according to the color difference between the cut surface and the surface of the material.

[0099] Extracting the epidermis area from the eroded HSV image can be achieved by color segmentation. There is a large color difference between the incision surface and the epidermis, that is, the color characteristics of the two are not the same. Therefore, the hue, saturation and brightness thresholds in the HSV color space can be set according to the color characteristics of the epidermis area. Usually, the color of the epidermis may be within a certain hue range, so a mask is generated using the set HSV range to extract the epidermis area that meets the conditions.

[0100] In one embodiment, the saturation (s) and brightness (v) of each pixel are extracted from the HSV image. If the HSV component of the pixel satisfies the condition (sv)>thr, it is an epidermal point; if it does not meet the condition, it is a material point. Wherein, thr is set to 0. Based on this, the process of extracting epidermal candidate pixels can be expressed as:

[0101] img hq-hsv-edge =img hq-hsv &&(sv)>thr

[0102] In the above formula, img hq-hsv-edge HSV image representing the epidermal area, img hq-hsv The HSV image represents the connected domain, and the extracted epidermal area is denoted as S edge When the value of (sv) is greater than thr, it means that the saturation of the pixel is higher than the brightness, which usually means that the pixel is in a brighter color area. Since thr is set to 0, this condition actually determines whether the saturation is greater than the brightness. Therefore, only when a pixel in the material HSV image meets the set condition (sv)>thr, the pixel will be retained in S edge In. Fig.12 As shown, Fig.12 The epidermal region extracted based on color recognition is shown.

[0103] In one embodiment, considering that a circular texture similar to the color of the skin may appear on the cut surface of the material, the non-skin part of the material may be mistakenly extracted by relying solely on color. Considering that the skin must appear at the edge of the material, additional position information is introduced to extract the skin area of ​​the material through color characteristics and position information. Fig.13 As shown in the figure, the extraction process of the epidermal area is:

[0104] Step S401, taking any contour point in the epidermis region of the material edge, if the pixel point is in the connected domain, adding it to the epidermis pixel point set to be removed.

[0105] Here, for any contour point in the surface area of ​​the material edge, it is determined whether the pixel point is in the connected domain. If so, it is added to the set of surface pixel points to be removed S edge-c It indicates that the point belongs to the pixel of the epidermis that needs to be removed.

[0106] Step S402, traverse the eight pixels in its neighborhood. If they are in the epidermal area based on color recognition and have not been visited, add them to the set of epidermal pixels to be removed, and repeat the step of traversing the neighborhood of the pixel to search until there are no new pixels to be removed that meet the conditions in the neighborhood. The area of ​​all pixels in the set of epidermal pixels to be removed is taken as the area of ​​the epidermal area.

[0107] Traverse the eight pixels in the neighborhood of any contour point. For any contour point p = (x, y), the eight pixels in its neighborhood are (x-1, y), (x+1, y), (x, y-1), (x, y+1), (x-1, y-1), (x-1, y+1), (x+1, y-1), (x+1, y+1). If these pixels have not been visited and are located in the epidermal region S based on color recognition, edge If the pixel points are within , then these pixels are added to the epidermis pixel set S to be removed. edge-c , where the set S edge-c The pixels in the area are the epidermal pixels after edge extraction. If the neighboring pixels are found to have been visited or do not belong to the epidermal area S during the search process, edge , then skip and search for the next neighboring pixel.

[0108] Repeat the search for eight pixels in the neighborhood of each arbitrary contour point until there are no new pixels that meet the conditions to be removed in the neighborhood, indicating that the search is completed, such as Fig.14 As shown, Fig.14 The epidermal region after edge extraction is shown.

[0109] After the search is completed, the set of epidermal pixels to be removed S edge-c The area of ​​all pixels in the epidermis is taken as the area of ​​the epidermis, that is, the set S edge-c All the pixels in the connected domain constitute the epidermal region, so the area of ​​the epidermal region is calculated using these pixels.

[0110] For some materials, the color of the entire product is not completely consistent. There may even be some internal areas that are close to the color of the skin. Therefore, when only color recognition is used, the outer circle and some internal areas may be identified as skin areas. By taking any point on the periphery and searching for continuous skin areas based on the neighborhood, it is possible to avoid internal material points from being mistakenly removed and improve the accuracy of the peeling results.

[0111] Step S304, obtaining the material area based on the difference between the connected domain area and the surface area.

[0112] The connected domain is divided into the incision surface and the epidermis area. The area corresponding to the incision surface area represents the actual area of ​​the material. Based on this, the material area S hq Equal to the connected domain area S cc Subtract the epidermal area S edge-c .

[0113] It should be noted that, in addition to Astragalus, if the actual area calculation of other independent materials also needs to take the edge area into consideration, the area calculation method of the independent materials in the above embodiment is also applicable.

[0114] In one embodiment, a method for calculating the area of ​​overlapping materials is provided.

[0115] First, the contour of the connected domain is extracted, wherein reference may be made to step S301 to extract an RGB image of the connected domain from the material transfer picture.

[0116] The RGB image of the connected domain records the images of multiple materials on the conveyor belt that are partially or completely overlapped. In the case of overlapping materials, some edges of the materials may be blocked, resulting in the inability to accurately identify their complete outlines. It is also difficult to determine which parts belong to which material, thus affecting the calculation of the material area. Based on this, the premise of calculating the material area of ​​overlapping materials is to segment each of the overlapping materials.

[0117] Here, a front- and back-view semantic segmentation network is pre-trained. This network is a deep learning model designed to handle material segmentation tasks in images and can identify the contours corresponding to each material in overlapping materials.

[0118] It should be noted that the training of the front-view and back-view semantic segmentation network depends on labeled training data. Fig.15In the contour annotation diagram of the training data shown, in the conventional annotation process, both the occluding material and the occluded material are annotated along the material contour. The segmentation network also focuses on the material contour visible from the current perspective. When a pixel point is spatially phased due to material overlap, causing a sudden change in the gradient value, it will be predicted as the material contour. Therefore, when calculating the area based on the prediction result, the connected domain area of ​​the occluded material is smaller than its true connected domain, which will cause a serious decline in the sorting effect. Moreover, since the trained front- and back-view semantic segmentation network can achieve the segmentation of overlapping materials from two different perspectives, front view and back view, based on this, when annotating the training data, it is also necessary to adapt to the training objectives of the front- and back-view semantic segmentation network. Here, as Fig.15 In the contour annotation schematic diagram shown, during the annotation process of training data, the conventional annotation method is still used for the occluded materials in the training data, that is, annotation is performed according to the contour of the visible part of the occluded material. For the occluded materials in the training data, the shape of the occluded materials observed from a rear-view angle is simulated according to the visible area of ​​the occluded materials, which means that the possible shape and contour of the occluded materials need to be considered, and then the contour of the occluded area of ​​the occluded materials is inferred, and annotation is performed along the contour of the occluded materials.

[0119] After completing the labeling of the training data, the labeled training data is input into the front-back view semantic segmentation network to be trained, so that the front-back view semantic segmentation network can learn the occlusion relationship of overlapping materials, thereby having the ability to segment the occluding materials and the occluded materials.

[0120] The working principle of the front-to-back view semantic segmentation network is explained below. It is understandable that in the case of overlapping materials, some materials may be occluded by other materials, making it difficult to accurately identify and segment each material. The front-to-back view semantic segmentation network has the ability to learn the occlusion relationship between overlapping materials, and can model the occlusion relationship of overlapping materials in the input material sub-image, that is, it can identify which parts of the overlapping materials are occluded and which parts are visible. This occlusion relationship modeling helps to improve the accuracy of segmentation. By modeling the occlusion relationship, the front-to-back view semantic segmentation network can decouple overlapping materials, that is, it can process overlapping materials separately so that the features of each material can be identified separately. In this case, the front-to-back view semantic segmentation network can extract the contour of each material from two different perspectives (front view and back view). It is understandable that in the front view, some materials may be obscured by other materials, while in the rear view, these obscured parts may be revealed. Therefore, the front view and rear view provide different observation angles, allowing the front and rear view semantic segmentation networks to capture different features of the materials through different perspectives. Among them, the front view provides the front features of the material, and the rear view can provide the shape and position of the obscured parts. By combining the information of these two perspectives and utilizing the complementarity of multi-perspective data, the front and rear view semantic segmentation networks can more accurately identify the true outline of each material.

[0121] like Fig.16 As shown, Fig.16 The schematic diagram of the structure of the front-view and back-view semantic segmentation network is shown, wherein the front-view and back-view semantic segmentation network includes a downsampling network (N1), a front-view upsampling network (N2) and a back-view upsampling network (N3). In the downsampling network, there are 4 convolution modules with a step size of 2 and 3 C3 layers, wherein the 4 convolution modules respectively include a convolution module with an input channel number of 3 and an output channel number of 64, a convolution module with an input channel number of 64 and an output channel number of 128, a convolution module with an input channel number of 128 and an output channel number of 256, and a convolution module with an input channel number of 256 and an output channel number of 512, and the 3 C3 layers respectively include a C3 layer with a depth of 4, a C3 layer with a depth of 6, and a C3 layer with a depth of 2. In the forward-view upsampling network and the backward-view upsampling network, there are 5 convolutional layers with a stride of 1, 3 C3 layers with a depth of 3, and 4 upsampling layers. The 5 convolutional layers are respectively a convolutional layer with an input channel number of 512 and an output channel number of 256, a convolutional layer with an input channel number of 256 and an output channel number of 128, a convolutional layer with an input channel number of 128 and an output channel number of 64, a convolutional layer with an input channel number of 64 and an output channel number of 32, and a convolutional layer with an input channel number of 32 and an output channel number of 2. The 3 C3 layers are a C3 layer with an output channel number of 256, a C3 layer with an output channel number of 128, and a C3 layer with an output channel number of 64.

[0122] It should be noted that the convolution module and convolution layer in the segmentation model are two different structures. The convolution module (ConvBNSiLU) is a common combination of neural network layers, which combines the convolution layer (Conv), batch normalization (BN, Batch Normalization) and activation function SiLU (Sigmoid Linear Unit) layer. The convolution layer (Conv) is a standard convolution layer that only contains convolution operations. Although activation functions can be added after the convolution layer, the Conv layer itself does not contain activation functions. In addition, the C3 layer in the segmentation model is a modular network structure, which generally includes residual blocks (Bottleneck), jump connections and feature fusion, where the depth of the C3 layer refers to the number of Bottleneck contained in the C3 layer. The C3 layer can divide the input data into two parts, one part of the input data is processed by Bottleneck, and the other part directly skips Bottleneck processing, and then the data processed by Bottleneck is spliced ​​with the data not processed by Bottleneck. Optionally, after splicing, a convolution layer can be used to perform convolution processing on the spliced ​​data to fuse features and avoid information redundancy.

[0123] The total loss L during model training is divided into two parts: the first loss value L caused by the output of the forward-looking upsampling network 前 and the second loss value L caused by the output of the back-view upsampling network 后 Among them, the first loss value L 前 It is calculated at the output layer of the forward-looking upsampling network. The second loss value L 后 It is calculated at the output layer of the back-view upsampling network. Both use cross entropy as the loss function. For each channel c, the cross entropy loss is defined as:

[0124]

[0125] Among them, y c is the true label, indicating the true value of category c, p c is the probability value predicted by the model, indicating the probability that the sample belongs to category c, and C is the total number of channels.

[0126] Then, by calculating the first loss value L 前 and the second loss value L 后 Perform weighted summation to obtain the final total loss L, which is defined as:

[0127] L=(1-w)*L 前 +w*L 后

[0128] Among them, w is set to 0.7.

[0129] Here, the total loss L is weighted summed to balance the loss of the front and back views during the training process of the front-view and back-view semantic segmentation network, ensuring that the back-view loss has a greater impact when dealing with the segmentation task of occluded materials, thereby improving the robustness and accuracy of the segmentation network. The total loss L will be used as the optimization target to update the parameters of the segmentation network through the back propagation algorithm to minimize the loss, thereby improving the segmentation performance of the segmentation network.

[0130] In the inference stage, the acquired material sub-images are input into the front-back view semantic segmentation network for processing. The front-back view semantic segmentation network can output the corresponding segmentation results, that is, the corresponding contours of each material in the overlapping materials.

[0131] In practical applications, overlapping materials can be composed of more than two materials. In order to more clearly illustrate the data processing process of the front-view and back-view semantic segmentation network, the following takes overlapping materials composed of two materials as an example. Fig.16 The data processing of the front-view and back-view semantic segmentation network is explained.

[0132] The main task of the downsampling network is to extract deep features from the connected domain of the input. The downsampling network usually includes multiple convolutional layers (Conv) and maximum pooling layers (Max Pooling). Through multiple convolutional layers and pooling layers, the resolution of the feature map is gradually reduced, while the number of channels of the feature map is increased to capture richer feature information. The specific process of extracting the first deep feature of the material sub-graph by the downsampling network is:

[0133] i) The size of the material sub-image (img) is H×W×3. The downsampling network performs the first layer of convolution on the material sub-image through the convolution module (ConvBNSiLU). In the first layer of convolution, the input of ConvBNSiLU is 3 channels, the output is 64 channels, and the step size is 2, so that the length and width of the feature map can be reduced to 1 / 2 of the original. That is, the size of the feature map output by ConvBNSiLU is Among them, ConvBNSiLU is a common combination of neural network layers, combining convolutional layers, batch normalization (BN) and activation function SiLU (Sigmoid Linear Unit). Then, after a C3 layer with a depth of 4, the receptive field of the network is increased without changing the resolution and the number of channels, and the size is obtained. feature map.

[0134] ii) Through ConvBNSiLU The second layer of convolution is performed on the feature map of . In the second layer of convolution, the input of ConvBNSiLU is 64 channels, the output is 128 channels, and the step size is 2, which can reduce the length and width of the feature map to 1 / 4 of the original. That is, the second layer of convolution will output a size of Then, after a C3 layer with a depth of 6, we get a feature map of size feature map.

[0135] iii) Through ConvBNSiLU The third layer of convolution is performed on the feature map. In the third layer of convolution, the input of ConvBNSiLU is 128 channels, the output is 256 channels, and the step size is 2, which can reduce the length and width of the feature map to 1 / 8 of the original. That is, the third layer of convolution will output a size of Then, after a C3 layer with a depth of 2, we get a feature map of size feature map.

[0136] iv) Finally, through ConvBNSiLU The fourth convolution is performed on the feature map of . In the fourth convolution, the input of ConvBNSiLU is 256 channels, the output is 512 channels, and the step size is 2, which can reduce the length and width of the feature map to 1 / 16 of the original. That is, the fourth convolution will output a size of feature map.

[0137] Through the above steps, the downsampling network completes the deep feature extraction of the connected domain, and the final output feature map is 1 / 16 resolution and 512 channels. This process effectively extracts important features.

[0138] The forward-looking upsampling network upsamples the first deep features output by the downsampling network on the basis of the downsampling network. After upsampling, the forward-looking upsampling network concatenates the upsampled feature map (i.e., the first upsampling result) with the first deep features extracted from the downsampling network, and finally restores the resolution of the feature map to the same size as the input material sub-image, and outputs the occluded material outline at the forward viewing angle. The specific processing process of the forward-looking upsampling network on the feature map is as follows:

[0139] i) Downsampling network through convolutional layer (Conv) The feature map is convolved in the first layer. In the first layer of convolution, the input of Conv is 512 channels, the output is 256 channels, and the step size is 1, and a new 1 / 16 feature map is obtained. Then, the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the first layer (2 times upsampling) to obtain a size of The feature map of the first layer is obtained by upsampling Feature Maps and Downsampling Networks The feature map is concatenated in the channel dimension to obtain feature map, thereby increasing the number of features describing the image. After a C3 layer with a depth of 3 and half the output channels, the size is feature map.

[0140] ii) Next, the C3 output of the forward-looking upsampling network The feature map is convolved for the second layer. In the second layer of convolution, the input of Conv is 256 channels, the output is 128 channels, and the step size is 1, and a new 1 / 8 feature map is obtained. Then, the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the second layer (2 times upsampling) to obtain a size of Feature map. The second layer is upsampled Feature Maps and Downsampling Networks The feature map is concatenated in the channel dimension and then passes through a C3 layer with a depth of 3 and half the output channels to obtain Feature map.

[0141] iii) Then, the output of C3 of the forward-looking upsampling network is The feature map is convolved in the third layer. In the third layer of convolution, the input of Conv is 128 channels, the output is 64 channels, and the step size is 1, and a new 1 / 4 feature map is obtained. Then the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the third layer (2 times upsampling) to obtain a size of Feature map. The third layer is upsampled to obtain Feature Maps and Downsampling Networks The feature map is concatenated in the channel dimension and then passes through a C3 layer with a depth of 3 and half the output channels to obtain Feature map.

[0142] iv) Then, the output of C3 of the forward-looking upsampling network is The feature map is convolved in the fourth layer. In the fourth layer of convolution, the input of Conv is 64 channels, the output is 32 channels, and the step size is 1, and a new 1 / 2 feature map is obtained. Then the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the fourth layer (2 times upsampling) to obtain a feature map of size H×W×32. The H×W×32 feature map passes through a convolutional layer with 32 input channels, C output channels, and a stride of 1 to obtain the output result about the front view, that is, the outline of the occluded material.

[0143] Forward vision usually refers to observing overlapping materials from the front. Overlapping materials can be divided into occluding materials and occluded materials. The occluding materials are located in front of the occluded materials, so that it can be observed that the occluding materials occlude the part of the occluded materials in the forward view. The forward-looking upsampling network processes the extracted deep features and can segment the overlapping materials in the forward view to obtain the outline of the occluding materials in the forward view. In practical applications, C in the forward-looking upsampling network is set to 2, indicating that each pixel has two possible categories: one category is the occluding material in the forward view, and the other category is the background or the occluded material in the forward view.

[0144] Through the above steps, forward-view upsampling can effectively restore image resolution, enhance segmentation accuracy, and adapt to complex occlusion scenes through progressive upsampling and multi-scale feature stitching, thereby achieving segmentation of overlapping materials in the forward perspective.

[0145] Based on the front-view upsampling network, the rear-view upsampling network upsamples the first deep features output by the front-view upsampling network. After upsampling, the rear-view upsampling network concatenates the upsampled feature map (i.e., the second upsampling result) with the second deep features extracted from the rear-view upsampling network, and finally restores the resolution of the feature map to the same size as the input material sub-image, and outputs the outline of the occluded material in the rear-view angle. The specific processing process of the rear-view upsampling network on the feature map is as follows:

[0146] i) At the output of the downsampling network Based on the feature map, the rear-view upsampling network uses similar steps to the front-view upsampling network to upsample and splice the feature map. The difference is that the prediction of occluded materials is less difficult, so the front-view upsampling network can be directly spliced ​​with the first deep features extracted by the downsampling network (which already contains the multi-scale visual information of the material). The prediction of occluded materials is more difficult, and the rear-view upsampling network needs to be spliced ​​with the second deep features of the front-view upsampling network (which already contains the relevant information of the occluded material) to make full use of the occlusion relationship of the target area that has been learned by the front-view upsampling network.

[0147] ii) The back-view upsampling network finally outputs the prediction result about the back-view, and its size is the same as the original Figure 1 The number of channels C is 2, which means that each pixel has two possible categories, one is the occluded material under rear-view, and the other is the background or the occluded material under rear-view.

[0148] Rear view usually refers to observing overlapping materials from the back. It can be understood that when observing overlapping materials from the front, it is difficult to observe the obstructed part of the obstructed object due to the existence of the obstructing object; while when observing overlapping materials from the rear, the obstructed part of the obstructed object can be observed, thereby obtaining the contour information of the obstructed object.

[0149] After the occluding material and the occluded material are extracted, the wheel width can be composed of a series of coordinate points, which represent the shape of the material boundary. Using the contour corresponding to each material, the area corresponding to each material is calculated respectively. Among them, there are many methods for area calculation. For example, the ready-made function provided by the image processing library can be used to calculate the area of ​​the material. The function can directly receive the contour data of the material and return the area corresponding to the material.

[0150] It should be noted that the material area actually includes the cut surface area and the epidermis area. In the process of calculating the area of ​​the obstructed material, the influence of the epidermis area on the material area needs to be considered, that is, the epidermis area in the obstructed material needs to be determined, and then the actual area of ​​the obstructed material is obtained according to the difference between the area of ​​the material area and the area of ​​the epidermis area. For the calculation of the area of ​​the obstructed material, the influence of the epidermis area can be ignored.

[0151] In the above embodiment, by determining whether the connected domain is an independent material or an overlapping material, and then determining the area of ​​the material according to the calculation rules for independent materials or overlapping materials, the area of ​​the independent material or overlapping materials can be accurately calculated, which is conducive to the effective sorting of independent materials and overlapping materials, thereby improving the material sorting effect and production efficiency.

[0152] In actual applications, when the material moves with the conveyor belt, the distance between the material and the left and right boundaries of the conveyor belt remains unchanged, and the material will not exceed the left and right boundaries of the conveyor belt. Therefore, the material will not be cut off at the left and right edges of the material transmission pictures obtained at different times. However, the distance between the material and the upper and lower boundaries of the conveyor belt will change with the movement of the conveyor belt. Therefore, the upper and lower edges of the material transmission pictures obtained at different times may be cut off. If the edge of the material in the material transmission picture is incomplete, it is difficult to accurately calculate the material area.

[0153] Based on this, in one embodiment, materials with complete edges are obtained through overlapping rows. The industrial camera will save a material transfer picture every fixed number of rows, so the industrial camera can first obtain a material transfer picture at the first time, and then obtain the next material transfer picture at a fixed number of rows (i.e., the second time). There are overlapping rows with a width of L in two material transfer pictures that are adjacent in time, where the overlapping rows represent the same image area in the two adjacent material transfer pictures.

[0154] When there is a material with incomplete edges in the material transfer picture, the material with complete edges can be obtained in the next adjacent material transfer picture through the overlapping lines in the two adjacent material transfer pictures, wherein the overlapping lines here include the image area corresponding to the material with incomplete edges. It should be noted that the height of the overlapping lines should be greater than the height of the positive circumscribed moment of a material, which means that if the material is cut off at the bottom edge of the previous material transfer picture, the material can still be fully displayed at the H2 line of the next adjacent material transfer picture.

[0155] like Fig.17 As shown, Fig.17 A schematic diagram of overlapping rows is shown, where Fig.17 The figure A and the figure B are included respectively. Taking the figure A as an example, the total number of rows H of the figure A can be divided into the image row H1 and the overlapping row H2 (that is, H=H1+H2), wherein the image row H1 is located at the 1st to H1th rows of the image, the overlapping row H2 is located at the H1th to H2th rows of the image, and the overlapping row H2 of the figure B is located at the 1st to H2th rows of the image, so that the material No. 1 in the figure A corresponds to the material No. 3 in the figure B, and the material No. 2 in the figure A corresponds to the material No. 4 in the figure B.

[0156] The specific process of obtaining materials with complete edges is as follows: identify the coordinates of the positive circumscribed moment corresponding to the material in the overlapping rows. Since the material will not be truncated at the left and right edges, we can further obtain only the upper and lower edge coordinates of the positive circumscribed moment corresponding to the material, recorded as y min and max . And currently get the upper and lower edge coordinates Y of the material transfer picture (Figure A) 1-0 and Y 1-1 And the upper and lower edge coordinates Y of the next adjacent material transfer picture (Figure B) 2-0 and Y 2-1 By changing y min and max Respectively with Y 1-0 , Y 1-1 , Y 2-0 and Y 2-1 A comparison is made so that it can be determined in which material transport picture the material is completely recorded.

[0157] Taking material No. 1 as an example, if the coordinates of material No. 1 in the current material transfer picture meet the y min >Y 1-0 &&y max <Y 1-1 At this time, the complete material No. 1 can be seen in Figure A, so the area calculation of material No. 1 is based on the material area in Figure A.

[0158] Taking material No. 2 as an example, if the coordinates of the material in Figure A satisfy y min >Y 1-0 &&y max =Y 1-1 At this time, material No. 2 is cut off at the lower edge of Figure A and will appear near the H2th row of Figure B, that is, material No. 4 in Figure B is the complete shape of material No. 1 in the first material transfer picture, so the area calculation of material No. 2 is based on the material area in Figure B.

[0159] Taking material No. 3 as an example, if the coordinates of material No. 3 in Figure B satisfy y min =Y 2-0 At this time, the material is cut off at the upper edge of Figure B, but it appears completely near row H1 in Figure A. Therefore, the area calculation of material No. 3 is based on the material area in Figure A.

[0160] In the above embodiment, by setting overlapping rows, the problem of material truncation caused by the image acquisition frequency can be effectively avoided. In this way, in adjacent material transfer pictures, even if a material transfer picture is truncated at the bottom edge, the next material transfer picture can still capture the complete material, so that a complete material sub-picture can be extracted.

[0161] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0162] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0163] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0164] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0165] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for identifying overlapping materials, characterized in that: include: Acquire a material transfer picture, and determine a connected domain in the material transfer picture; Performing instance segmentation on the connected domain to obtain a material region in the connected domain; The connected domain image after segmenting the material area is corroded as a whole, and any pixel point is randomly extracted from the independent area after corrosion; The number of the arbitrary pixel points located in the connected domain is determined, and the connected domain is judged to be an overlapping material or an independent material.

2. The method for identifying overlapping materials according to claim 1, characterized in that: The determining of the connected domain in the material transfer picture includes: According to the color information in the material transfer picture, converting the material transfer picture into a material transfer binary picture; The material transport binary image is corroded.

3. The overlapping material identification method according to claim 1, characterized in that: The entire connected domain image after the material area is segmented is corroded, and any pixel point is randomly extracted from the independent area after the corrosion, including: Perform an erosion operation on the connected domain image after instance segmentation. After iterating M times, randomly select a material pixel point in the remaining material area. The material pixel points extracted from all material areas in the same connected domain are aggregated and counted.

4. The overlapping material recognition method according to claim 3 is characterized in that when a material area in the connected domain is corroded for the M+1th time, its pixel point is exactly 0, and after the Mth corrosion operation, a pixel point of the material is randomly selected in the image of the material area.

5. The method for identifying overlapping materials according to claim 1, characterized in that: There are overlapping rows between two adjacent material transfer pictures.

6. A material area calculation method based on the overlapping material identification method according to any one of claims 1 to 5, characterized in that: Based on the number of the arbitrary pixel points in the connected domain, the connected domain is judged to be an overlapping material or an independent material, and the area of ​​the corresponding material is determined based on the calculation rules of the overlapping material or the independent material.

7. The material area calculation method according to claim 6, characterized in that: When the connected domain is an independent material, it includes Binarizing the material transport image, separating the background according to the area of ​​the connected domain, and extracting the RGB image of the connected domain; Convert the RGB image of the connected domain to the HSV space, and perform corrosion processing on the obtained HSV image; Extracting the epidermis region in the connected domain according to the color difference between the cut surface and the epidermis of the material; The material area is obtained based on the difference between the connected domain area and the skin area.

8. The material area calculation method according to claim 7, characterized in that: The method of corroding the HSV image is to perform an erosion operation using an erosion kernel of a 3*3 square; The method for extracting the epidermis area is that if the HSV component of the pixel point satisfies SV>thr, it is an epidermis point, otherwise it is a material point, and thr is the empirical threshold.

9. The material area calculation method according to claim 7 is characterized in that the step of extracting the surface area in the connected domain according to the color difference between the cut surface and the surface of the material further comprises: Take any contour point in the skin area of ​​the material edge. If the pixel point is in the connected domain, add it to the set of skin pixels to be removed. Traverse the eight pixels in its neighborhood. If they are in the epidermal area identified based on color and have not been visited, add them to the set of epidermal pixels to be removed and repeat the search until there are no new pixels to be removed that meet the conditions in the neighborhood. The area of ​​all pixels in the set of epidermal pixels to be removed is taken as the area of ​​the epidermal area.

10. The material area calculation method according to claim 6, characterized in that: When the connected domain is an overlapping material, the method includes extracting the contour of the connected domain, modeling the occlusion relationship of the overlapping materials based on a trained front and rear view semantic segmentation network, decoupling and extracting the contours of the occluding material and the occluded material from the front and rear view angles, and calculating the corresponding areas respectively; wherein, modeling the occlusion relationship of the overlapping materials based on the trained front and rear view semantic segmentation network, decoupling, and extracting the contours of the occluding material and the occluded material from the front and rear view angles, respectively, includes: The segmentation model includes a downsampling network, a front-view upsampling network and a rear-view upsampling network. The first deep feature about the connected domain is extracted based on the downsampling network; the first deep feature is upsampled on the basis of the downsampling network through the front-view upsampling network, and the first upsampling result is spliced ​​with the first deep feature to output the contour of the occluded material at the front viewing angle; the first deep feature is upsampled on the basis of the front-view upsampling network through the rear-view upsampling network, and the second upsampling result is spliced ​​with the second deep feature extracted by the front-view upsampling network to output the contour of the occluded material at the rear-view angle.

Images