Method and device for calculating area of overlapped materials
By segmenting the pictures of overlapping materials and calculating the material area with front and back perspective information, the problem of inaccurate area identification caused by material overlap is solved, and the accuracy and efficiency of material sorting are improved.
Patent Information
- Application Number
- CN202411875051.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-11
AI Technical Summary
During the processing of Chinese herbal medicines, material overlapping results in the inability to accurately identify the true area of overlapping materials, affecting the sorting effect and production efficiency.
A method of calculating the area of overlapping materials is adopted. By obtaining the picture of overlapping materials and inputting it to a pre-trained segmentation model, the outline of each material is obtained, and the area of each material is calculated based on the information of front and back perspectives.
Accurate area estimation of overlapping materials is achieved, and the accuracy and production efficiency of material sorting are improved.
Smart Images
Figure CN119810177A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for calculating the area of overlapping materials. Background Art
[0002] Chinese herbal medicines of different sizes have different economic values. Manual sorting is extremely inefficient and difficult to cope with the processing scenarios of a large number of Chinese herbal medicines. In related technologies, machine vision technology is introduced to actively identify Chinese herbal medicines of different sizes, and then spray valves and other equipment are used to eliminate Chinese herbal medicines of specific sizes. For sheet-like medicinal materials such as astragalus, conveyor belts are generally used for transportation in production environments. Materials may overlap on the conveyor belts, making it impossible to effectively identify the true area of the overlapping materials. Overlapping materials are easily eliminated during the sorting process, resulting in low production efficiency and poor sorting effects. Summary of the invention
[0003] One of the technical problems to be solved by the present invention is to provide a method capable of identifying the real area of overlapping materials.
[0004] In order to achieve the above object, a first embodiment of the present invention provides a method for calculating the area of overlapping materials, comprising:
[0005] Obtaining overlapping material images, and inputting the overlapping material images into a pre-trained segmentation model to segment and obtain the contours corresponding to each material in the overlapping materials; wherein the segmentation model is designed to model the occlusion relationship of the overlapping materials, decouple, and extract the material contours from the front and rear perspectives;
[0006] According to the contour corresponding to each material in the overlapping materials, the area corresponding to each material in the overlapping materials is calculated respectively.
[0007] According to the method for calculating the area of overlapping materials in an embodiment of the present invention, the contour corresponding to each material in the overlapping materials is segmented, and the contour corresponding to each material obtained by segmentation is used to calculate the area of each material in the overlapping materials. This can more accurately estimate the area of each material in an overlapping state, which is beneficial for sorting each material in an overlapping state by using the area of the material, thereby improving the processing efficiency of the material.
[0008] In addition, the method for calculating the area of overlapping materials according to the present invention may also have the following additional technical features:
[0009] In the above scheme, the segmentation model includes a downsampling network, a forward-looking upsampling network and a backward-looking upsampling network. The overlapping material images are input into the pre-trained segmentation model to segment and obtain the contours corresponding to each material in the overlapping materials, including:
[0010] Extracting first deep features about the overlapping material images based on the downsampling network;
[0011] By using the front-view upsampling network, the first deep feature is upsampled on the basis of the downsampling network, and the first upsampling result is spliced with the first deep feature to output the outline of the occluded material at the front viewing angle;
[0012] Through the rear-view upsampling network, the first deep feature is upsampled based on the front-view upsampling network, and the second upsampling result is spliced with the second deep feature extracted by the front-view upsampling network to output the outline of the obscured material at the rear-view angle.
[0013] In the above scheme, the downsampling network includes 4 convolution modules with a step size of 2 and 3 C3 layers, wherein the number of input channels of the convolution modules is 3, 64, 128, and 256, respectively, the number of output channels of the convolution modules is 64, 128, 256, and 512, respectively, and the depth of the C3 layer is 4, 6, and 2, respectively;
[0014] The forward-view upsampling network and the backward-view upsampling network both include 5 convolutional layers with a step size of 1, 3 C3 layers with a depth of 3, and 4 upsampling layers, wherein the number of input channels of the convolutional layers are 512, 256, 128, 64, and 32, respectively, and the number of output channels of the convolutional layers are 256, 128, 64, 32, and 2, respectively; the number of output channels of the C3 layers are 256, 128, and 64, respectively.
[0015] In the above scheme, during the training process of the segmentation model, the method further includes:
[0016] Annotating along the contour of the obstructing material in the training data, and simulating the contour of the obstructed area of the obstructed material in the rear view according to the visible area of the obstructed material in the training data, and annotating the obstructed material; the obstructing material is placed in front of the obstructed material;
[0017] The labeled training data is input into the segmentation model to be trained for iterative training.
[0018] In the above solution, the step of inputting the labeled training data into the segmentation model to be trained for iterative training includes:
[0019] At the end of one iteration of training, a first loss value is calculated according to an output result of the forward-looking upsampling network, and a second loss value is calculated according to an output result of the backward-looking upsampling network;
[0020] The first loss value and the second loss value are weightedly summed to obtain a total loss value of the segmentation model in this iterative training.
[0021] In the above scheme, the method further comprises:
[0022] Each material in the overlapping materials is sorted according to the area corresponding to each material in the overlapping materials.
[0023] In the above solution, obtaining overlapping material images includes:
[0024] A material conveying picture of the material being conveyed on a conveyor belt is acquired, and the overlapping material picture is identified in the material conveying picture.
[0025] In the above scheme, there is an overlapping row with a width of L between two adjacent material transfer pictures; the overlapping row represents the same image area in the two adjacent material transfer pictures; wherein, if there is overlapping material with incomplete edges in the material transfer picture, then the overlapping material with complete edges is obtained in the next adjacent material transfer picture.
[0026] To achieve the above object, a second embodiment of the present invention provides a device for calculating the area of overlapping materials, comprising:
[0027] Acquisition module, used to obtain overlapping material images;
[0028] A contour segmentation module is used to input the overlapping material images into a pre-trained segmentation model to segment and obtain the contours corresponding to each material in the overlapping materials; wherein the segmentation model is designed to model the occlusion relationship of the overlapping materials, decouple and extract the material contours from the front and rear perspectives;
[0029] The area calculation module is used to calculate the area corresponding to each material in the overlapping materials according to the contour corresponding to each material in the overlapping materials.
[0030] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 This is the architecture diagram of the experimental platform;
[0032] Figure 2 It is a schematic diagram of a flow chart of a method for calculating the area of overlapping materials in one embodiment;
[0033] Figure 3 A schematic diagram of the structure of a segmentation model in one embodiment;
[0034] Figure 4It is a schematic diagram of contour marking in one embodiment;
[0035] Figure 5 is a schematic diagram of overlapping rows in one embodiment;
[0036] Figure 6 4 is a structural block diagram of a device for calculating the area of overlapping materials in one embodiment. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0038] like Figure 1 As shown, Figure 1 The diagram shows the architecture of the experimental platform, which is used to sort materials. The experimental platform mainly includes a feeding device, a conveyor belt, a camera, a spray valve, a main control, etc. The feeding device is responsible for conveying the materials to be sorted in the container to the conveyor belt. The materials move on the conveyor belt, and the conveyor belt transports the materials from the feeding device to the detection area of the industrial camera. The industrial camera collects the image of the materials on the conveyor belt, detects and identifies the area size of the materials on the conveyor belt in real time through the material image, and then sprays the materials to different sorting boxes (silo 1 and silo 2) through the spray valve, thereby realizing the sorting of materials.
[0039] It should be noted that Figure 1 The materials processed by the experimental platform are usually medicinal materials, such as astragalus, etc., but it also supports the processing of other different types of materials.
[0040] During the working process of the experimental platform, the movement posture of the material may change during the unloading and transmission process, resulting in overlapping. If the experimental platform is used to sort materials that do not meet the size requirements, the overlapping materials will be removed, resulting in poor sorting effect. A large amount of materials need to be re-sorted, reducing production efficiency.
[0041] If the experimental platform is used to sort materials according to their area size, since the identification and sorting operations are performed after the materials are thrown out of the conveyor belt, this aerial identification and sorting method can only perform sorting once, that is, sorting into large materials and small materials. As for overlapping materials, since the obscured parts cannot be seen, the area calculation of the obscured materials is affected, the area of the obscured materials cannot be effectively identified, and it is difficult to sort the overlapping materials, resulting in poor sorting effects and reduced production efficiency.
[0042] Based on this, in order to effectively sort overlapping materials, a method suitable for Figure 1The method for calculating the area of overlapping materials of the experimental platform shown can effectively calculate the area of each material in an overlapping state.
[0043] The implementation details of the technical solution of the embodiment of the present application are described in detail below.
[0044] In one embodiment, Figure 2 As shown, a method for calculating the area of overlapping materials is provided, and the method may include the following steps:
[0045] Step S101, obtaining overlapping material images, and inputting the overlapping material images into a pre-trained segmentation model to segment and obtain the contour corresponding to each material in the overlapping materials.
[0046] Overlapping material images refer to images of multiple materials partially or completely overlapping on a conveyor belt. In practical applications, overlapping material images can be captured by industrial cameras. When materials overlap, the edges of some materials may be blocked, making it impossible to accurately identify their complete outlines. At the same time, the blockage makes it difficult to determine which parts belong to which material, thus affecting the calculation of the area of each material.
[0047] Based on this, a segmentation model is pre-trained. The segmentation model is a deep learning model designed to handle material segmentation tasks in images and can identify the contours of each material in overlapping materials. The acquired overlapping material images are input into the segmentation model for processing, and the segmentation model can output the corresponding segmentation results, that is, the contours of each material in the overlapping materials.
[0048] The working principle of the segmentation model is explained below. It is understandable that in the case of overlapping materials, some materials may be obscured by other materials, making it difficult to accurately identify and segment each material. The segmentation model has the ability to learn the occlusion relationship between overlapping materials, and can model the occlusion relationship of overlapping materials in the input overlapping material image, that is, it can identify which parts of the overlapping materials are obscured and which parts are visible. This occlusion relationship modeling helps to improve the accuracy of segmentation. By modeling the occlusion relationship, the segmentation model can decouple overlapping materials, that is, it can process overlapping materials separately so that the features of each material can be identified separately. In this case, the segmentation model can extract the outline of each material from two different perspectives (front view and rear view). It is understandable that in the front view, some materials may be obscured by other materials, while in the rear view, these obscured parts may be revealed. Therefore, the front view and the rear view provide different observation angles, allowing the segmentation model to capture different features of the material through different perspectives. Among them, the front view provides the front features of the material, and the rear view can provide the shape and position of the obscured part. By combining the information of these two perspectives and utilizing the complementarity of multi-perspective data, the segmentation model can more accurately identify the true outline of each material.
[0049] like Figure 3 As shown, Figure 3 The schematic diagram of the structure of the segmentation model is shown, wherein the segmentation model includes a downsampling network (N1), a front-view upsampling network (N2) and a rear-view upsampling network (N3). In the downsampling network, there are 4 convolution modules with a step size of 2 and 3 C3 layers, wherein the 4 convolution modules respectively include a convolution module with an input channel number of 3 and an output channel number of 64, a convolution module with an input channel number of 64 and an output channel number of 128, a convolution module with an input channel number of 128 and an output channel number of 256, and a convolution module with an input channel number of 256 and an output channel number of 512, and the 3 C3 layers respectively include a C3 layer with a depth of 4, a C3 layer with a depth of 6, and a C3 layer with a depth of 2. In the forward-view upsampling network and the backward-view upsampling network, there are 5 convolutional layers with a stride of 1, 3 C3 layers with a depth of 3, and 4 upsampling layers. The 5 convolutional layers are respectively a convolutional layer with an input channel number of 512 and an output channel number of 256, a convolutional layer with an input channel number of 256 and an output channel number of 128, a convolutional layer with an input channel number of 128 and an output channel number of 64, a convolutional layer with an input channel number of 64 and an output channel number of 32, and a convolutional layer with an input channel number of 32 and an output channel number of 2. The 3 C3 layers are a C3 layer with an output channel number of 256, a C3 layer with an output channel number of 128, and a C3 layer with an output channel number of 64.
[0050] It should be noted that the convolution module and convolution layer in the segmentation model are two different structures. The convolution module (ConvBNSiLU) is a common combination of neural network layers, which combines the convolution layer (Conv), batch normalization (BN, Batch Normalization) and activation function SiLU (Sigmoid Linear Unit) layer. The convolution layer (Conv) is a standard convolution layer that only contains convolution operations. Although activation functions can be added after the convolution layer, the Conv layer itself does not contain activation functions. In addition, the C3 layer in the segmentation model is a modular network structure, which generally includes residual blocks (Bottleneck), jump connections and feature fusion, where the depth of the C3 layer refers to the number of Bottleneck contained in the C3 layer. The C3 layer can divide the input data into two parts, one part of the input data is processed by Bottleneck, and the other part directly skips Bottleneck processing, and then the data processed by Bottleneck is spliced with the data not processed by Bottleneck. Optionally, after splicing, a convolution layer can be used to perform convolution processing on the spliced data to fuse features and avoid information redundancy.
[0051] In actual applications, overlapping materials can be composed of more than two materials. In order to more clearly illustrate the data processing process of the segmentation model, the following takes overlapping materials composed of two materials as an example. Figure 3 The data processing of the segmentation model is described.
[0052] The main task of the downsampling network is to extract deep features from the input overlapping material images. The downsampling network usually includes multiple convolutional layers (Conv) and maximum pooling layers (Max Pooling). Through multiple convolutional layers and pooling layers, the resolution of the feature map is gradually reduced, while the number of channels of the feature map is increased to capture richer feature information. The specific process of the downsampling network extracting the first deep features of the overlapping material image is:
[0053] i) The size of the overlapping material image (img) is H×W×3. The downsampling network performs the first layer of convolution on the overlapping material image through ConvBNSiLU. In the first layer of convolution, the input of ConvBNSiLU is 3 channels, the output is 64 channels, and the step size is 2, so that the length and width of the feature map can be reduced to 1 / 2 of the original. That is, the size of the feature map output by ConvBNSiLU is Next, after a C3 layer with a depth of 4, the receptive field of the network is increased without changing the resolution and the number of channels, resulting in a size of feature map.
[0054] ii) Through ConvBNSiLU The second layer of convolution is performed on the feature map of . In the second layer of convolution, the input of ConvBNSiLU is 64 channels, the output is 128 channels, and the step size is 2, which can reduce the length and width of the feature map to 1 / 4 of the original. That is, the second layer of convolution will output a size of Then, after a C3 layer with a depth of 6, we get a feature map of size feature map.
[0055] iii) Through ConvBNSiLU The third layer of convolution is performed on the feature map. In the third layer of convolution, the input of ConvBNSiLU is 128 channels, the output is 256 channels, and the step size is 2, which can reduce the length and width of the feature map to 1 / 8 of the original. That is, the third layer of convolution will output a size of Then, after a C3 layer with a depth of 2, we get a feature map of size feature map.
[0056] iv) Finally, the ConvBNSiLU is used to The fourth convolution is performed on the feature map of . In the fourth convolution, the input of ConvBNSiLU is 256 channels, the output is 512 channels, and the step size is 2, which can reduce the length and width of the feature map to 1 / 16 of the original. That is, the fourth convolution will output a size of feature map.
[0057] Through the above steps, the downsampling network completes the deep feature extraction of the overlapping material images, and the final output feature map is 1 / 16 resolution and 512 channels. This process effectively extracts important features in the image.
[0058] The forward-looking upsampling network upsamples the first deep features output by the downsampling network based on the downsampling network. After upsampling, the forward-looking upsampling network concatenates the upsampled feature map (i.e., the first upsampling result) with the first deep features extracted from the downsampling network, and finally restores the resolution of the feature map to the same size as the input overlapping material image, and outputs the outline of the occluded material in the forward-looking perspective. The specific processing process of the forward-looking upsampling network on the feature map is as follows:
[0059] i) Downsampling the network through the convolutional layer (Conv) The feature map is convolved in the first layer. In the first layer of convolution, the input of Conv is 512 channels, the output is 256 channels, and the step size is 1, and a new 1 / 16 feature map is obtained. Then, the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the first layer (2 times upsampling) to obtain a size of The feature map of the first layer is obtained by upsampling Feature Maps and Downsampling Networks The feature map is concatenated in the channel dimension to obtain feature map, thereby increasing the number of features describing the image. After a C3 layer with a depth of 3 and half the output channels, the size is feature map.
[0060] ii) Next, the C3 output of the forward-looking upsampling network The feature map is convolved for the second layer. In the second layer of convolution, the input of Conv is 256 channels, the output is 128 channels, and the step size is 1, and a new 1 / 8 feature map is obtained. Then, the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the second layer (2 times upsampling) to obtain a size of Feature map. The second layer is upsampled Feature Maps and Downsampling Networks The feature map is concatenated in the channel dimension and then passes through a C3 layer with a depth of 3 and half the output channels to obtain Feature map.
[0061] iii) Then, the output of C3 of the forward-looking upsampling network is The feature map is convolved in the third layer. In the third layer of convolution, the input of Conv is 128 channels, the output is 64 channels, and the step size is 1, and a new 1 / 4 feature map is obtained. Then the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the third layer (2 times upsampling) to obtain a size of Feature map. The third layer is upsampled to obtain Feature Maps and Downsampling Networks The feature map is concatenated in the channel dimension and then passes through a C3 layer with a depth of 3 and half the output channels to obtain Feature map.
[0062] iv) Then, the output of C3 of the forward-looking upsampling network is The feature map is convolved in the fourth layer. In the fourth layer of convolution, the input of Conv is 64 channels, the output is 32 channels, and the step size is 1, and a new 1 / 2 feature map is obtained. Then the output of Conv is interpolated by the nearest neighbor. The feature map is upsampled by the fourth layer (2 times upsampling) to obtain a feature map of size H×W×32. The H×W×32 feature map passes through a convolutional layer with 32 channels as input, C channels as output, and a stride of 1 to obtain the output result about the front view.
[0063] Forward vision usually refers to observing overlapping materials from the front. Overlapping materials can be divided into occluding materials and occluded materials. The occluding materials are located in front of the occluded materials, so that it can be observed that the occluding materials occlude the part of the occluded materials in the forward view. The forward-looking upsampling network processes the extracted deep features and can segment the overlapping materials in the forward view to obtain the outline of the occluding materials in the forward view. In practical applications, C in the forward-looking upsampling network is set to 2, indicating that each pixel has two possible categories: one category is the occluding material in the forward view, and the other category is the background or the occluded material in the forward view.
[0064] Through the above steps, forward-view upsampling can effectively restore image resolution, enhance segmentation accuracy, and adapt to complex occlusion scenes through progressive upsampling and multi-scale feature stitching, thereby achieving segmentation of overlapping materials in the forward perspective.
[0065] Based on the forward-view upsampling network, the backward-view upsampling network upsamples the first deep features output by the forward-view upsampling network. After upsampling, the backward-view upsampling network concatenates the upsampled feature map (i.e., the second upsampling result) with the second deep features extracted from the backward-view upsampling network, and finally restores the resolution of the feature map to the same size as the input overlapping material image, and outputs the outline of the occluded material at the rear-view angle. The specific processing process of the backward-view upsampling network on the feature map is as follows:
[0066] i) At the output of the downsampling network Based on the feature map, the rear-view upsampling network uses similar steps to the front-view upsampling network to upsample and splice the feature map. The difference is that the prediction of occluded materials is less difficult, so the front-view upsampling network can be directly spliced with the first deep features extracted by the downsampling network (which already contains multi-scale visual information of overlapping materials). The prediction of occluded materials is more difficult, and the rear-view upsampling network needs to be spliced with the second deep features of the front-view upsampling network (which already contains relevant information about the occluded materials) to make full use of the occlusion relationship of the target area that has been learned by the front-view upsampling network.
[0067] ii) The back-view upsampling network finally outputs the prediction result about the back-view, and its size is the same as the original Figure 1 The number of channels C is 2, which means that each pixel has two possible categories, one is the occluded material under rear-view, and the other is the background or the occluded material under rear-view.
[0068] Rear view usually refers to observing overlapping materials from the back. It can be understood that when observing overlapping materials from the front, it is difficult to observe the obstructed part of the obstructed object due to the existence of the obstructing object; while when observing overlapping materials from the rear, the obstructed part of the obstructed object can be observed, thereby obtaining the contour information of the obstructed object.
[0069] Through the above steps, the back-view upsampling network can effectively handle the segmentation problem of occluded objects and improve the accuracy of segmentation.
[0070] In one embodiment, the material on the conveyor belt may be a single material, or may be an overlapping material composed of multiple different materials overlapping. It is understandable that the material transfer picture taken by the industrial camera records the different materials on the conveyor belt, that is, the material transfer picture contains single materials and / or overlapping materials. Based on this, it is necessary to identify overlapping material pictures that only contain overlapping materials in the material transfer picture, wherein the contours of all materials in the material transfer picture can be extracted, and by counting the number of extracted contours, if there is only one contour, it may be a single material, and if there are multiple contours, it may be overlapping materials, thereby identifying the overlapping materials in the material transfer picture, and then extracting the pixel area corresponding to the overlapping materials, so that the overlapping material picture can be obtained.
[0071] It should be noted that in actual applications, when the interval between two different materials is small, the boundaries of the two materials will adhere to each other, and there is a certain degree of difficulty in distinguishing between the adhesion between materials and overlapping materials. Based on this, in addition to the identification of overlapping materials and single materials, it is also necessary to identify the adhesion materials, so as to improve the recognition accuracy of overlapping materials.
[0072] When the material moves with the conveyor belt, the distance between the material and the left and right boundaries of the conveyor belt remains unchanged, and the material will not exceed the left and right boundaries of the conveyor belt. Therefore, the material will not be cut off at the left and right edges of the material transmission pictures obtained at different times. However, the distance between the material and the upper and lower boundaries of the conveyor belt will change with the movement of the conveyor belt. Therefore, the upper and lower edges of the material transmission pictures obtained at different times may be cut off. If the truncated overlapping material picture is extracted from the material transmission picture, and the truncated overlapping material picture is used to calculate the area corresponding to each material in the overlapping material, then the calculated area corresponding to each material will be smaller than its actual area.
[0073] Based on this, in one embodiment, the concept of overlapping rows is introduced, and the overlapping rows are used to obtain complete overlapping material pictures. In practical applications, the industrial camera will save a material transfer picture every fixed number of rows, so the industrial camera can first obtain a material transfer picture at the first time, and then obtain the next material transfer picture at a fixed number of rows (i.e., the second time). There is an overlapping row with a width of L between two material transfer pictures that are adjacent in time. The overlapping row is used to indicate the same image area in the two material transfer pictures, that is, the image area where the overlapping row of the first material transfer picture is located is the same as the image area where the overlapping row of the second material transfer picture is located.
[0074] It is understandable that if overlapping materials with incomplete edges appear in the current material transfer picture, then overlapping materials with complete edges can be obtained in the next adjacent material transfer picture through overlapping rows in two adjacent pictures, where the overlapping rows here include the image area corresponding to the overlapping materials with incomplete edges. It should be noted that the height of the overlapping rows should be greater than the height of the positive circumscribed moment of an overlapping material, which means that if the overlapping material is cut off at the bottom edge of the previous material transfer picture, the overlapping material can still be fully displayed at the row of the next material transfer picture.
[0075] like Figure 5 As shown, Figure 5 A schematic diagram of overlapping rows is shown, where Figure 5 The graphs A and B are included in the graphs. Taking graph A as an example, the total number of rows H of graph A can be divided into image rows H1 and overlapping rows H2 (i.e., H = H i +H2), wherein image row H1 is located at the 1st to H1th rows of the image, overlapping row H2 is located at the H1th to H2th rows of the image, and overlapping row H2 of Figure B is located at the 1st to H2th rows of the image, so that material No. 1 in Figure A corresponds to material No. 3 in Figure B, and material No. 2 in Figure A corresponds to material No. 4 in Figure B.
[0076] Next, identify the coordinates of the positive circumscribed moment corresponding to the overlapping material in the overlapping row. Since the overlapping material will not be truncated at the left and right edges, we can further obtain only the upper and lower edge coordinates of the positive circumscribed moment corresponding to the overlapping material, which are recorded as y min and max . And get the upper and lower edge coordinates Y of the current material transfer image 1-0 and Y 1-1 And the upper and lower edge coordinates Y of the next material transfer picture 2-0 and Y 2-1 By changing y min and max Respectively with Y 1-0 , Y 1-1 , Y 2-0and Y 2-1 A comparison is made to determine in which material transport picture the overlapping material is completely recorded.
[0077] If the coordinates of the overlapping materials in the current material transfer picture meet the y min >Y 1-0 &&y max <Y 1-1 ,At this time, the complete material can be seen in the current material transfer picture, so the image area where the overlapping material is located is extracted from the current material transfer picture as the overlapping material picture.
[0078] If the coordinates of the overlapping materials in the current material transfer picture meet the y min >Y 1-0 &&y max =Y 1-1 At this time, the overlapping material is cut off at the lower edge of the current material transfer picture and will appear near the H2th row of the next material transfer picture, thereby extracting the image area where the overlapping material is located in the next material transfer picture as the overlapping material picture.
[0079] If the coordinates of the overlapping materials in the next material transfer picture meet the y min =Y 2-0 At this time, the overlapping material is cut off at the upper edge of the next material transfer picture, but it has completely appeared near the H1th row in the current material transfer picture, so that the image area where the overlapping material is located is extracted in the current material transfer picture as the overlapping material picture.
[0080] In the above embodiment, by setting the overlapping rows, the problem of overlapping material truncation caused by the image acquisition frequency can be effectively avoided. In this way, in the continuous material conveying pictures, even if a material conveying picture is truncated at the bottom edge, the next material conveying picture can still capture the complete overlapping material, so that the complete overlapping material picture can be extracted.
[0081] In one embodiment, the training of the segmentation model relies on labeled training data. Figure 4In the contour annotation schematic diagram shown, in the conventional annotation process, both the occluding material and the occluded material are annotated along the material contour. The segmentation model also focuses on the material contour visible from the current perspective. When a pixel point is spatially phased due to material overlap, resulting in a sudden change in the gradient value, it will be predicted as the material contour. Therefore, when calculating the area based on the prediction result, the connected domain area of the occluded material is smaller than its true connected domain, which will cause a serious decline in the sorting effect. Moreover, since the trained segmentation model can achieve the segmentation of overlapping materials from two different perspectives, front view and rear view, based on this, when annotating the training data, it is also necessary to adapt to the training objectives of the segmentation model.
[0082] Here, if Figure 4 In the outline annotation schematic diagram shown, in the process of annotating the overlapping material training pictures, the conventional annotation method is still used for the occluded materials in the training data, that is, annotation is performed according to the outline of the visible part of the occluded material. For the occluded materials in the training data, the shape of the occluded material observed from a rear-view angle is simulated based on the visible area of the occluded material, which means that the possible shape and outline of the occluded material need to be considered, and then the outline of the occluded area of the occluded material is inferred, and annotation is performed along the outline of the occluded material.
[0083] After the training data is labeled, the labeled training data is input into the segmentation model to be trained, so that the segmentation model can learn the occlusion relationship of overlapping materials, thereby having the ability to segment the occluding materials and the occluded materials.
[0084] In one embodiment, the total loss L during model training is divided into two parts: the first loss value L caused by the output of the forward-looking upsampling network and 前 and the second loss value L caused by the output of the back-view upsampling network 后 Among them, the first loss value L 前 It is calculated at the output layer of the forward-looking upsampling network. The second loss value L 后 It is calculated at the output layer of the back-view upsampling network. Both use cross entropy as the loss function. For each channel c, the cross entropy loss is defined as:
[0085]
[0086] Among them, y c is the true label, indicating the true value of category c, p c is the probability value predicted by the model, indicating the probability that the sample belongs to category c, and C is the total number of channels.
[0087] Then, by calculating the first loss value L 前 and the second loss value L 后Perform weighted summation to obtain the final total loss L, which is defined as:
[0088] L=(1-w)*L 前 +w*L 后
[0089] Among them, w is set to 0.7.
[0090] Here, the total loss L is weighted summed so that the segmentation model can balance the loss of front and back vision during training, ensuring that the impact of back vision loss is greater when dealing with the segmentation task of occluded materials, thereby improving the robustness and accuracy of the segmentation model. The total loss L will be used as the optimization target to update the parameters of the segmentation model through the back propagation algorithm to minimize the loss, thereby improving the segmentation performance of the segmentation model.
[0091] Step S102, calculating the area corresponding to each material in the overlapping materials according to the contour corresponding to each material in the overlapping materials.
[0092] Here, the contour corresponding to each material can be composed of a series of coordinate points, which represent the shape of the material boundary. Using the contour corresponding to each material, the area corresponding to each material is calculated respectively. Among them, there are many methods for area calculation. For example, the ready-made function provided by the image processing library can be used to calculate the area of the material. The function can directly accept the contour data of the material and return the area corresponding to the material.
[0093] In one embodiment, after calculating the area corresponding to each material in the overlapping materials, the overlapping materials can be sorted according to the area of each material. Figure 1 As shown, silo 1 is used to hold materials with a larger area, and silo 2 is used to hold materials with a smaller area. If the area of the material is larger than the set threshold, the material is sorted into silo 1; if the area of the material is smaller than the threshold, the material is sorted into silo 2, thereby realizing the sorting of each material in the overlapping materials and improving the production efficiency and sorting results.
[0094] In the case of taking material carry-out into consideration, for example, when using nozzles to blow up for material sorting, normal materials move directly into the receiving bin, and abnormal materials are blown out of the normal material queue by the nozzles. Due to the change in motion state, they fall into the waste bin. Since there is inevitably a certain range of influence when blowing, the motion state of normal materials around the target material may also be changed, and thus enter the waste bin. Since normal material carry-out is inevitable, we can only ensure that all materials entering the receiving bin are normal materials, but we cannot ensure that all materials entering the waste bin are abnormal materials. Therefore, when collecting, materials that meet the size requirements can be collected as normal materials, and materials that do not meet the size requirements (small size or overlapping materials with small size materials) can be collected as abnormal materials.
[0095] In the above embodiment, the acquired overlapping material images are input into the segmentation model, and the segmentation model can model the occlusion relationship of the overlapping materials, so that the contour corresponding to each material in the overlapping materials can be accurately predicted in two different perspectives, front view and rear view. Then, according to the contour corresponding to each material in the overlapping materials, the area corresponding to each material is calculated respectively, so that the area of each material in the overlapping state can be accurately calculated, which is conducive to the sorting of overlapping materials and improves production efficiency and sorting effect.
[0096] In one embodiment, a device for calculating the area of overlapping materials is provided, referring to Figure 6 As shown, the overlapping material area calculation device 600 may include: an acquisition module 601, a contour segmentation module 602, an area calculation module 603, a model training module 604 and a material sorting module 605.
[0097] Wherein, the acquisition module 601 is used to acquire overlapping material images;
[0098] The contour segmentation module 602 is used to input the overlapping material image into a pre-trained segmentation model, and segment to obtain the contour corresponding to each material in the overlapping materials; wherein the segmentation model is designed to model the occlusion relationship of the overlapping materials, decouple and extract the material contour from the front and rear perspectives;
[0099] The area calculation module 603 is used to calculate the area corresponding to each material in the overlapping materials according to the contour corresponding to each material in the overlapping materials.
[0100] In one embodiment, the segmentation model includes a downsampling network, a front-view upsampling network and a rear-view upsampling network. The contour segmentation module 602 is specifically used to extract a first deep feature about the overlapping material image based on the downsampling network; through the forward-view upsampling network, the first deep feature is upsampled on the basis of the downsampling network, and the first upsampling result is spliced with the first deep feature to output the contour of the occluded material at the front perspective; through the rear-view upsampling network, the first deep feature is upsampled on the basis of the forward-view upsampling network, and the second upsampling result is spliced with the second deep feature extracted by the forward-view upsampling network to output the contour of the occluded material at the rear perspective.
[0101] In one embodiment, the downsampling network includes 4 convolution modules with a stride of 2 and 3 C3 layers, wherein the number of input channels of the convolution modules is 3, 64, 128, and 256, respectively, the number of output channels of the convolution modules is 64, 128, 256, and 512, respectively, and the depth of the C3 layer is 4, 6, and 2, respectively;
[0102] The forward-view upsampling network and the backward-view upsampling network both include 5 convolutional layers with a step size of 1, 3 C3 layers with a depth of 3, and 4 upsampling layers, wherein the number of input channels of the convolutional layers are 512, 256, 128, 64, and 32, respectively, and the number of output channels of the convolutional layers are 256, 128, 64, 32, and 2, respectively; the number of output channels of the C3 layers are 256, 128, and 64, respectively.
[0103] In one embodiment, the model training module 604 is used to label along the contour of the occluding material in the training data, and according to the visible area of the occluded material in the training data, simulate the contour of the occluded area of the occluded material in the rear perspective, and label the occluded material; the occluding material is placed in front of the occluded material; and the labeled training data is input into the segmentation model to be trained for iterative training.
[0104] In one embodiment, the model training module 604 is specifically used to, at the end of one iterative training, calculate a first loss value based on the output result of the forward-looking upsampling network, and calculate a second loss value based on the output result of the backward-looking upsampling network; perform weighted summation on the first loss value and the second loss value to obtain a total loss value of the segmentation model in this iterative training.
[0105] In one embodiment, the material sorting module 605 is used to sort each material in the overlapping materials according to the area corresponding to each material in the overlapping materials.
[0106] In one embodiment, the acquisition module 601 is specifically used to acquire a material transfer picture of the material being transferred on a conveyor belt, and identify overlapping material pictures in the material transfer picture.
[0107] In one embodiment, there is an overlapping row with a width of L between two adjacent material transfer pictures; the overlapping row represents the same image area in the two adjacent material transfer pictures; wherein, if there is overlapping material with incomplete edges in the material transfer picture, then the overlapping material with complete edges is obtained in the next adjacent material transfer picture, so the width L of the overlapping row should be greater than the maximum size of the material, generally limited to the maximum size of conventional materials, so that when identifying the material, there is no need to fuse the parts on the two pictures, thereby reducing errors.
[0108] The specific definition of the overlapping material area calculation device 600 can refer to the definition of the overlapping material area calculation method mentioned above, which will not be repeated here. Each module in the overlapping material area calculation device 600 can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0109] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0110] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0112] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0113] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A method for calculating the area of overlapping materials, characterized in that: include: Obtaining overlapping material images, and inputting the overlapping material images into a pre-trained segmentation model to segment and obtain the contours corresponding to each material in the overlapping materials; wherein the segmentation model is designed to model the occlusion relationship of the overlapping materials, decouple, and extract the material contours from the front and rear perspectives; According to the contour corresponding to each material in the overlapping materials, the area corresponding to each material in the overlapping materials is calculated respectively.
2. The method for calculating the area of overlapping materials according to claim 1, characterized in that: The segmentation model includes a downsampling network, a forward-looking upsampling network, and a backward-looking upsampling network. The overlapping material images are input into the pre-trained segmentation model to segment and obtain the contours corresponding to each material in the overlapping materials, including: Extracting first deep features about the overlapping material images based on the downsampling network; By using the front-view upsampling network, the first deep feature is upsampled on the basis of the downsampling network, and the first upsampling result is spliced with the first deep feature to output the outline of the occluded material at the front viewing angle; Through the rear-view upsampling network, the first deep feature is upsampled based on the front-view upsampling network, and the second upsampling result is spliced with the second deep feature extracted by the front-view upsampling network to output the outline of the obscured material at the rear-view angle.
3. The method for calculating the area of overlapping materials according to claim 2, characterized in that: The downsampling network includes 4 convolution modules with a step size of 2 and 3 C3 layers, wherein the number of input channels of the convolution modules is 3, 64, 128, and 256, respectively, the number of output channels of the convolution modules is 64, 128, 256, and 512, respectively, and the depth of the C3 layer is 4, 6, and 2, respectively; The forward-view upsampling network and the backward-view upsampling network both include 5 convolutional layers with a step size of 1, 3 C3 layers with a depth of 3, and 4 upsampling layers, wherein the number of input channels of the convolutional layers are 512, 256, 128, 64, and 32, respectively, and the number of output channels of the convolutional layers are 256, 128, 64, 32, and 2, respectively; the number of output channels of the C3 layers are 256, 128, and 64, respectively.
4. The method for calculating the area of overlapping materials according to claim 2, characterized in that: During the training of the segmentation model, the method further includes: Annotating along the contour of the obstructing material in the training data, and simulating the contour of the obstructed area of the obstructed material in the rear view according to the visible area of the obstructed material in the training data, and annotating the obstructed material; the obstructing material is placed in front of the obstructed material; The labeled training data is input into the segmentation model to be trained for iterative training.
5. The method for calculating the area of overlapping materials according to claim 4, characterized in that: The step of inputting the labeled training data into the segmentation model to be trained for iterative training includes: At the end of one iteration of training, a first loss value is calculated according to an output result of the forward-looking upsampling network, and a second loss value is calculated according to an output result of the backward-looking upsampling network; The first loss value and the second loss value are weightedly summed to obtain a total loss value of the segmentation model in this iterative training.
6. The method for calculating the area of overlapping materials according to claim 1, characterized in that: The method further comprises: Each material in the overlapping materials is sorted according to the area corresponding to each material in the overlapping materials.
7. The method for calculating the area of overlapping materials according to claim 1, characterized in that: The obtaining of overlapping material images includes: A material conveying picture of the material being conveyed on a conveyor belt is acquired, and the overlapping material picture is identified in the material conveying picture.
8. The method for calculating the area of overlapping materials according to claim 7, characterized in that: There is an overlapping line with a width of L between two adjacent material conveying pictures; the overlapping line represents the same image area in the two adjacent material conveying pictures.
9. A device for calculating the area of overlapping materials, characterized in that: include: Acquisition module, used to obtain overlapping material images; A contour segmentation module is used to input the overlapping material images into a pre-trained segmentation model to segment and obtain the contours corresponding to each material in the overlapping materials; wherein the segmentation model is designed to model the occlusion relationship of the overlapping materials, decouple and extract the material contours from the front and rear perspectives; The area calculation module is used to calculate the area corresponding to each material in the overlapping materials according to the contour corresponding to each material in the overlapping materials.