Turnover box residual tobacco leaf identification method and equipment based on machine vision and medium

Abstract

The invention belongs to the technical field of visual identification. The invention provides a turnover box residual tobacco identification method and device based on machine vision and a medium. The method comprises the steps that firstly, a turnover box image is obtained; inputting the image of the turnover box into a contour recognition module of a tobacco leaf recognition model to obtain initial residual information; inputting the turnover box image and the initial residual information into a type identification module of a tobacco leaf identification model to obtain a residual type, a residual space state and a residual position; through the visual identification technology, the type, the state and the position of the residual tobacco leaves are accurately identified, information is provided for automatic cleaning, and therefore the cleaning efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of visual recognition technology, and in particular relates to a method, device and medium for identifying residual tobacco leaves in a turnover box based on machine vision. Background Art

[0002] The vacuum rehumidification process is an important production process in the silk-making workshop. This process is mainly used to increase the moisture content of the material, improve the temperature and toughness of the material, and remove green impurities. In actual production, there are often wet tobacco leaves remaining on the inner surface of the turnover box used in this process. If it is not cleaned in time and thoroughly, it will not only cause a large amount of material waste, but also there is a high risk of water-stained tobacco leaves. What's more, it will cause the problem of different grades of tobacco leaves being mixed with each other, which seriously threatens the product quality.

[0003] At present, most cigarette factories deal with residual materials on the inner surface of turnover boxes mainly by manual cleaning, such as using shovels, brooms, and brushes to clean the residues. This cleaning method has the disadvantages of being time-consuming and labor-intensive, labor-intensive, low in production efficiency, and the cleaning effect cannot be guaranteed. In response to the difficulties in this industry, some scholars or technicians have conducted relevant research, such as redesigning breathable turnover boxes, applying chemical coatings to the inner surface of turnover boxes to prevent adhesion, setting automatic nozzles based on PLC to clean specific tracks, and using brush rollers to clean specific paths. These methods have solved the problem of cleaning the residues in turnover boxes to a certain extent, but there is still a lot of room for improvement in terms of processing speed, processing effect, intelligence, and integration. Summary of the invention

[0004] In view of this, the present invention provides a method, device and medium for identifying residual tobacco leaves in a turnover box based on machine vision, aiming to solve the problem of low efficiency in processing residual materials on the inner surface of the turnover box in the prior art.

[0005] A first aspect of the present invention provides a method for identifying residual tobacco leaves in a turnover box based on machine vision, comprising:

[0006] Get the turnover box image;

[0007] Input the turnover box image into the contour recognition module of the tobacco leaf recognition model to obtain initial residual information;

[0008] The turnover box image and initial residue information are input into the type recognition module of the tobacco leaf recognition model to obtain the residue type, residue spatial state and residue position.

[0009] In a possible implementation, the type recognition module includes an image feature extraction unit, a contour association feature extraction unit and a fusion unit; the turnover box image and the initial residue information are input into the type recognition module of the tobacco leaf recognition model to obtain the residue type, the residue space state and the residue position, including:

[0010] Input the initial residual information and the turnover box image into the image feature extraction unit to obtain a multi-level feature map;

[0011] The multi-level feature map and the initial residual information are input into the contour association feature extraction unit to determine the spatial feature information;

[0012] The spatial feature information and multi-level feature maps are input into the fusion unit to obtain the residue type, residue spatial state and residue position.

[0013] In a possible implementation, the initial residual information and the turnover box image are input into the image feature extraction unit to obtain a multi-level feature map, including:

[0014] Segment the turnover box image according to the initial residual information to obtain a segmented image;

[0015] According to the initial residual information corresponding to each segmented image, the scale of the first convolution kernel corresponding to each segmented image is selected;

[0016] Input each segmented image into the corresponding first convolution kernel to obtain basic local features;

[0017] The basic local features are input into the multi-layer second convolution kernel to obtain a multi-level feature map.

[0018] In a possible implementation, the multi-level feature map and the initial residual information are input into a contour-related feature extraction unit to determine spatial feature information, including:

[0019] Input the initial residual information into the contour-associated feature extraction unit to obtain the contour internal features;

[0020] Input the multi-level feature map into the contour correlation feature extraction unit to obtain the spatial correlation feature;

[0021] The spatial feature information is determined based on the features within the contour and the spatial correlation features.

[0022] In a possible implementation, the spatial feature information and the multi-level feature map are input into the fusion unit to obtain the residue type, the residue spatial state and the residue position, including:

[0023] Determine a first feature according to the spatial feature information and the multi-level feature image;

[0024] The first feature is input into the fusion unit to obtain the residue type, residue space state and residue position.

[0025] In a possible implementation, the method further includes:

[0026] Get dynamic images and camera shooting area in real time;

[0027] The residue type, residual spatial state and dynamic image of the residual tobacco leaves in the shooting area are input into the lightweight neural network to obtain the residue correction information.

[0028] In a possible implementation, the method further includes:

[0029] According to the dynamic IOU post-processing algorithm, the lightweight neural network is optimized.

[0030] A second aspect of the present invention provides a device for identifying residual tobacco leaves in a turnover box based on machine vision, comprising:

[0031] An image acquisition module, used to acquire images of turnover boxes;

[0032] The first recognition module is used to input the turnover box image into the contour recognition module of the tobacco leaf recognition model to obtain initial residual information;

[0033] The second recognition module is used to input the turnover box image and initial residue information into the type recognition module of the tobacco leaf recognition model to obtain the residue type, residue space state and residue position.

[0034] The third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for identifying residual tobacco leaves in a turnover box based on machine vision as described in the first aspect above are implemented.

[0035] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for identifying residual tobacco leaves in a turnover box based on machine vision as described in the first aspect above.

[0036] The method, device and medium for identifying residual tobacco leaves in turnover boxes based on machine vision provided by the embodiment of the present invention first obtain an image of the turnover box; then input the image of the turnover box into the contour recognition module of the tobacco leaf recognition model to obtain initial residual information; then input the image of the turnover box and the initial residual information into the type recognition module of the tobacco leaf recognition model to obtain the residual type, residual spatial state and residual position. The present invention uses visual recognition technology to accurately identify the type, state and position of residual tobacco leaves, provide information for automated cleaning, and thus improve cleaning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 It is a flow chart of the implementation of the method for identifying residual tobacco leaves in a turnover box based on machine vision provided by an embodiment of the present invention;

[0039] Figure 2 It is a structural schematic diagram of a turnover box residual tobacco leaf identification device based on machine vision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0041] Figure 1 1 is a flowchart of the method for identifying residual tobacco leaves in a turnover box based on machine vision provided by an embodiment of the present invention. Figure 1 As shown, the method for identifying residual tobacco leaves in a turnover box based on machine vision includes:

[0042] S110, acquiring an image of a turnover box;

[0043] S120, inputting the turnover box image into a contour recognition module of a tobacco leaf recognition model to obtain initial residual information;

[0044] S130, inputting the turnover box image and the initial residue information into the type recognition module of the tobacco leaf recognition model to obtain the residue type, the residue space state and the residue position.

[0045] In the embodiment of the present invention, the target of tobacco leaf residue identification in the turnover box is mainly tobacco leaves and their fragments. Tobacco leaves have specific shapes, textures and colors. Its shape is usually slender and curly, the texture shows a natural vein distribution, and the color is mainly a tobacco-specific color range from yellow-green to dark brown. Residues of different types and states usually need to be cleaned in different ways. The tobacco leaf residues in the turnover box may be in different stacking states, with overlap, squeezing, etc. This makes identification more difficult because some tobacco leaves may be blocked by other tobacco leaves, making it difficult to fully obtain their shape and texture information. There are several difficulties in cleaning the residual tobacco leaves in the turnover box:

[0046] After tobacco leaves are processed and transported, a large number of tiny fragments will be generated. These fragments may be embedded in the gaps, corners and tiny depressions on the surface of the turnover box. For example, after a turnover box is used for a long time, its plastic surface may have some small pits due to wear and tear, and tobacco leaf fragments are easy to get stuck in it.

[0047] Moreover, tobacco leaves have a certain adsorption property and will be adsorbed on the inner wall of the turnover box. This is because there are some tiny fluff and sticky substances on the surface of tobacco leaves, which can absorb dust and other impurities, and also make tobacco leaf fragments tightly attached to the turnover box, which is difficult to remove by simple cleaning.

[0048] In the turnover box, the tobacco residue may pile up together and form a compact state. In particular, when the tobacco leaves at the bottom of the turnover box are pressed by the tobacco leaves above, they will be compacted. This compacted tobacco pile not only increases the difficulty of cleaning, but also makes the tobacco leaves inside difficult to reach. For example, in a large tobacco processing plant, the turnover boxes may be stacked frequently, and the tobacco residue in the bottom turnover box will become like a hard block after a long period of pressure, requiring greater force to break it up and clean it.

[0049] The types of tobacco leaf residues in the turnover boxes may include: large pieces of intact tobacco leaf residues, broken tobacco leaf fragments, and compacted tobacco leaf piles.

[0050] Large intact tobacco leaf residues usually maintain a relatively complete leaf shape, with obvious petioles and leaf parts, and the leaves may have a certain degree of curling. In terms of size, the length of the intact leaf is generally about 10-30 cm, and the width is about 5-15 cm. For example, the leaves of flue-cured tobacco are relatively large, and the color is mostly golden yellow to dark brown.

[0051] Broken tobacco leaf fragments are irregular in shape and vary greatly in size. Small fragments may be only a few millimeters in size, while large fragments may be several centimeters. The edges of the fragments are usually jagged, which may be caused by tearing or crushing during transportation, loading and unloading. The color is similar to that of whole tobacco leaves, but due to the increased surface area after crushing, it may be more susceptible to oxidation and the color is slightly darker.

[0052] Compacted tobacco piles appear to be in a piled state and compacted. In appearance, they are tightly packed together to form blocks or clumps. Their shape may vary depending on the shape of the turnover box, which may be flat blocks at the bottom of the turnover box or irregular clumps in the corners. The color may be darker due to the tight stacking, and there may be slight signs of mold and black or gray spots due to poor internal air circulation.

[0053] The residual space state may include: tobacco residues at the bottom of the turnover box, tobacco residues on the side walls of the turnover box, and tobacco residues in the gaps of the turnover box.

[0054] The tobacco residues at the bottom of the crate are mainly distributed on the bottom plane of the crate, and may be spread flat on the bottom due to gravity, or piled up in the corners. In crates with drainage holes, the tobacco residues may block the drainage holes.

[0055] Tobacco residues on the side walls of the turnover box are attached to the side walls of the turnover box. They may be left due to friction and collision between the tobacco leaves and the side walls during transportation. They may be scattered or continuous, especially near the top opening, where tobacco leaves are more likely to remain on the side walls due to loading and unloading.

[0056] Tobacco residues in the gaps of turnover boxes mainly exist in various gaps of turnover boxes, such as the gap between the box cover and the box body, the joint gap between the side walls, the gap around the drainage holes, etc. The tobacco residues in these gaps are usually small fragments, because it is difficult for large tobacco leaves to enter the gaps.

[0057] In addition, the residual space state may also include the residual area and residual thickness of the residual tobacco leaves there, which are not limited here. In particular, for a compacted tobacco leaf pile, the larger the residual area and the higher the thickness, the more cleaning efforts are needed.

[0058] In an embodiment of the present invention, the contour recognition module uses a small-size convolution kernel of 5×5, which can effectively capture the detailed features of the local area of ​​the image, such as the edge and texture of the tobacco leaf. With the stacking of the convolution layer, the network gradually extracts higher-level and more abstract semantic features, transitioning from simple lines to block features that can characterize the outline of the tobacco leaf. The pooling layer is connected after the convolution layer, and the maximum pooling or average pooling method is often used. The maximum pooling selects the maximum value in the local area, which can retain the most significant features of the image, while halving the size of the feature map, greatly reducing the amount of data and computational complexity; the average pooling calculates the average value of the local area, which plays a role in smoothing features. Through the pooling operation, not only the resource consumption of subsequent calculations is reduced, but also the extracted features are made more robust. After multiple rounds of convolution and pooling, the highly abstract feature map is expanded and input into the fully connected layer. The fully connected layer connects all the feature points, integrates the information of each local feature, provides a comprehensive feature basis for the final contour prediction, and integrates the previously scattered and different-level features into a unified feature vector. At the end of the fully connected layer, two parallel branches are set to perform different tasks. One branch is used for classification, and the Softmax function is used to output the probability that each pixel belongs to the outline of a tobacco leaf, and to determine whether the pixel is at the edge of the tobacco leaf; the other branch is responsible for regression, predicting the coordinate offset of the outline point, and more accurately locating the pixel position initially determined to be the outline, to assist in outlining the complete outline as the initial residual information. Based on the obtained outline, it can only be determined that there is residue in a certain place, but the specific type of residue needs further analysis.

[0059] In some embodiments, the type recognition module includes an image feature extraction unit, a contour association feature extraction unit and a fusion unit; the turnover box image and the initial residue information are input into the type recognition module of the tobacco leaf recognition model to obtain the residue type, residue spatial state and residue position, including: inputting the initial residue information and the turnover box image into the image feature extraction unit to obtain a multi-level feature map; inputting the multi-level feature map and the initial residue information into the contour association feature extraction unit to determine the spatial feature information; inputting the spatial feature information and the multi-level feature map into the fusion unit to obtain the residue type, residue spatial state and residue position.

[0060] In some embodiments, the initial residual information and the turnover box image are input into an image feature extraction unit to obtain a multi-level feature map, including: segmenting the turnover box image according to the initial residual information to obtain a segmented image; selecting the scale of the first convolution kernel corresponding to each segmented image according to the initial residual information corresponding to each segmented image; inputting each segmented image into the corresponding first convolution kernel to obtain basic local features; inputting the basic local features into multiple layers of second convolution kernels to obtain a multi-level feature map.

[0061] In an embodiment of the present invention, the initial residual information includes key data such as the coordinates of the tobacco leaf contour and the size of the enclosed area output by the previous contour recognition module. These data clearly indicate the position and range of the tobacco leaves in the turnover box image. Using this information is like holding a "map" to accurately divide the part of the turnover box image with tobacco leaf residues from the entire image to form segmented images. For example, if the initial residual information marks the position and contour of a large piece of complete tobacco leaf in the corner of the turnover box, then based on this contour, the rectangular or irregular shaped area containing this tobacco leaf can be cropped out from the overall image of the turnover box to obtain a segmented image containing only this specific tobacco leaf, in preparation for the subsequent targeted feature extraction.

[0062] In an embodiment of the present invention, different tobacco leaf residues have different sizes and spatial states, and the initial residual information can reflect these characteristics. Large pieces of intact tobacco leaves have a larger residual area, broken tobacco leaf fragments are smaller and scattered, and compacted tobacco leaf piles are irregular in shape and have different thickness and other information. For larger-scale targets, such as large pieces of intact tobacco leaves, a larger-scale first convolution kernel is required, so that the convolution kernel can cover a larger image area in one operation, capture the overall characteristics of the large tobacco leaf, and avoid missing important information; and for small broken fragments, a small-scale convolution kernel is more suitable, which can focus more finely on the texture, edges and other subtle features of the small fragments. Therefore, according to the type and size of the tobacco leaf residues in the initial residual information, a first convolution kernel of appropriate scale is adapted for each segmented image.

[0063] In an embodiment of the present invention, for large intact tobacco leaves, due to their large area and relatively regular shape, a larger-scale first convolution kernel is often required, such as a 7×7 or 9×9 convolution kernel. This type of large-scale convolution kernel can cover a larger range of image areas in one convolution operation, and fully capture key information such as the overall outline of large tobacco leaves and the direction of the main texture, without missing important features. In the face of broken tobacco leaf fragments, due to their small size and fragmentation, they are more suitable for small-scale convolution kernels of 3×3 or 5×5. The small convolution kernel can focus on tiny parts of the fragments, accurately extract the unique texture and edge details of the fragments, and avoid averaging the subtle features of the fragments due to the convolution kernel being too large.

[0064] When the residue is at the bottom, sidewall or gap of the turnover box, its spatial distribution characteristics are different. The tobacco leaf fragments in the gap of the turnover box are not only small in size, but also often have a narrow and irregular shape. At this time, the convolution kernel with a very small scale and a special aspect ratio is more suitable, which can fit the shape of the gap to extract information; the compacted tobacco leaf pile at the bottom of the turnover box has a certain thickness and block aggregation characteristics. A slightly larger convolution kernel with an appropriate stride can help to include some key features of the pile at one time and speed up the feature extraction efficiency.

[0065] In an embodiment of the present invention, each segmented image is respectively fed into the first convolution kernel of the selected scale. During the convolution process, the convolution kernel is like a weighted sliding window, starting from the upper left corner of the image, and sliding rightward and downward pixel by pixel according to the set stride (such as 1 pixel). Every time it slides to a new position, the weight value in the convolution kernel is matrix multiplied with the pixel value of the corresponding image area, and then accumulated and summed to generate a new eigenvalue. This new value represents the feature intensity of the local image area after the convolution kernel filtering. By continuously sliding the convolution kernel to traverse the entire segmented image, a new feature map is obtained, which is the basic local feature. For example, for a large piece of complete tobacco leaf image segmented out, the first convolution kernel of the appropriate scale can enhance the sharpness of the leaf edge and highlight the texture characteristics of the main veins after sliding convolution. These unique local features are gathered into the feature map.

[0066] The basic local features retain the most basic and recognizable information in the segmented image. They reflect the original appearance of the tobacco residue, from the density of the texture, the degree of edge to the gradient of the color, laying a solid foundation for the subsequent deep feature extraction. Although these feature maps are in the early stages, they have filtered out a lot of redundant background information and focused on the tobacco residue itself.

[0067] In an embodiment of the present invention, the second convolution kernel of the first layer takes the basic local feature map as input, and applies the convolution operation again, selecting convolution kernels of smaller scale but larger number, such as a 3×3 convolution kernel array, to further explore the subtle changes hidden in the basic local features. These subtle changes may be finer vein bifurcations in texture and more subtle transition differences in color. As the levels progress, the second convolution kernels of subsequent layers continue to abstract and condense information based on the previously extracted features. With each layer of convolution, the features become more abstract and the semantic information becomes richer, gradually transitioning from simple visual representation features to deep features that can characterize the type and state of tobacco leaf residues.

[0068] In the embodiment of the present invention, the feature maps output by the second convolution kernels at different levels together constitute a multi-level feature map. The shallow feature map focuses on the detailed texture and precise edges of the image, while the deep map contains semantic judgment clues such as whether the tobacco leaf is intact, broken, or compacted. The maps at each level complement and cooperate with each other, outlining the full picture of the characteristics of the tobacco residue from the appearance to the inner essence, providing sufficient and hierarchical basis for subsequent accurate recognition and classification tasks.

[0069] In some embodiments, a multi-level feature map and initial residual information are input into a contour association feature extraction unit to determine spatial feature information, including: inputting the initial residual information into the contour association feature extraction unit to obtain contour inner features; inputting the multi-level feature map into the contour association feature extraction unit to obtain spatial association features; and determining spatial feature information based on contour inner features and spatial association features.

[0070] The various residues in the turnover box are not relatively isolated. For example, there may be some debris on the surface and below the large tobacco residues, and there may also be debris on the surface of the compacted tobacco pile. In addition, the tobacco pile and the residual leaves may block each other, which will make identification difficult. If these cannot be accurately identified, misjudgment may occur, which may lead to errors in the planning of cleaning paths and strength. For example, for tobacco leaf fragments, too much cleaning force will cause the fragments to fly around, and for tobacco leaf piles, too little force will not clean them thoroughly.

[0071] In an embodiment of the present invention, the contour-associated feature extraction unit focuses on the inside of each independent tobacco leaf contour. For the contour of a large piece of complete tobacco leaf, the coordinate information is used to lock the area range, and the unique texture feature distribution pattern inside the tobacco leaf is extracted, such as the thickness and direction of the main veins and the texture uniformity of the leaf flesh area; for broken tobacco leaf fragments, the more subtle texture fluctuations and color unevenness within the fragment contour are analyzed. Through targeted feature extraction algorithms, such as the grayscale co-occurrence matrix to calculate the contrast and correlation of the texture within the contour, and the local binary pattern to capture the texture pattern frequency, the contour features that only exist inside the contour and characterize the characteristics of the tobacco leaf itself are excavated.

[0072] The multi-level feature map is generated by processing multiple layers of convolution kernels in sequence, covering everything from shallow detail-oriented features to deep semantically rich features. After receiving the map, the contour association feature extraction unit first interprets the maps at each level, sorts out the progressive relationship between features at different levels, and understands the logic of feature evolution from representation to abstraction. Using the map information, the unit begins to capture the spatial association features between adjacent contours and between contours and the background of the turnover box. In the shallow map, based on the continuity of texture and edge features, it is judged whether adjacent fragments are superimposed or mixed, and the relationship between fragments is identified; with the assistance of the deep map, combined with semantic information, the distribution law between the contour sets at different positions (bottom, side wall, gap) of the turnover box is considered, such as the spatial density relationship between the compacted tobacco pile at the bottom and the scattered fragments on the side wall, and then the spatial association features reflecting the overall spatial layout are obtained.

[0073] In some embodiments, spatial feature information and a multi-level feature map are input into a fusion unit to obtain a residue type, a residue spatial state, and a residue position, including: determining a first feature based on the spatial feature information and the multi-level feature map; inputting the first feature into a fusion unit to obtain a residue type, a residue spatial state, and a residue position.

[0074] In the embodiment of the present invention, after obtaining the inner contour features and the spatial correlation features, the two are fused. The local precise information of the inner contour features and the overall layout information of the spatial correlation features are integrated to construct a comprehensive feature matrix. Specifically, the inner contour features and the spatial correlation features are integrated and spliced ​​to obtain the spatial feature information.

[0075] In some embodiments, the method further includes: acquiring dynamic images and the shooting area of ​​the camera in real time; inputting the residual type, residual spatial state and dynamic images of the residual tobacco leaves in the shooting area into a lightweight neural network to obtain residual correction information.

[0076] In the embodiment of the present invention, in order to track the cleaning process of the residue in real time and identify the residue in the blind spot before, a camera is installed on the cleaning robot to capture dynamic images and identify them in real time.

[0077] The camera installed on the cleaning robot starts working, continuously capturing the images inside the turnover box, and collecting dynamic images at a fixed frame rate (for example, 30 frames per second). The parameters of the camera need to be carefully adjusted in advance, including focal length, aperture, exposure time, etc., to ensure that clear and moderately bright images can be obtained under the complex and changeable lighting conditions of the cleaning environment. The focal length determines the field of view and the size of the image, and should be set according to the size of the turnover box and the details to be observed; the aperture and exposure time work together to control the amount of light entering to avoid overexposure or underexposure.

[0078] At the same time, the camera's built-in sensors and positioning algorithms are used to accurately determine the current shooting area of ​​the camera. On the one hand, the camera's own inertial sensors such as gyroscopes and accelerometers are used to sense its posture changes in three-dimensional space; on the other hand, the joint angle information of the robotic arm is combined with the kinematic model of the robotic arm to calculate the exact position and orientation of the camera relative to the turnover box, thereby defining the specific range of the shooting area in the turnover box, such as the upper left corner of the turnover box, the bottom center, and other specific positions.

[0079] The input data is sorted, and the corresponding data in the current shooting area is selected from the previously identified residue types (large intact tobacco residues, broken tobacco fragments, compacted tobacco piles) and residue spatial states (tobacco residues at the bottom of the turnover box, tobacco residues on the side walls, tobacco residues in the gaps, etc.). These data are preprocessed together with the real-time dynamic images. The dynamic images are first converted into a unified tensor format and the pixel values ​​are normalized; the residue type and spatial state information are encoded and converted into a vector form suitable for neural network input, such as using one-hot encoding to represent different types and states.

[0080] Select lightweight neural networks, such as MobileNet and ShuffleNet, which are designed for mobile terminals or resource-constrained scenarios. Input the preprocessed dynamic image, the encoded residual type, and the spatial state vector into the network in sequence. The image data enters the convolution layer to extract real-time visual features; the type and state vectors are connected to the fully connected layer to supplement the prior knowledge for subsequent comprehensive judgment.

[0081] Inside the network, the visual features extracted from the image are fused with the features contained in the type and state vectors through feature pyramid networks or skip connections, so that the network can comprehensively consider the previous recognition information and the current real-time picture. After that, after multiple convolutional layers, pooling layers, and fully connected layers, the network uses the fused features to make predictions and output the results of the residual correction information.

[0082] The residue correction information covers many aspects. On the one hand, it corrects the type and spatial status of the residues that were previously misidentified. For example, what was originally thought to be a large piece of intact tobacco leaves was actually determined to be multiple small broken pieces after real-time images and multi-angle observations; on the other hand, it supplements the residues that were not identified due to blind spots in the previous shooting, marks the newly discovered residues in the deep gaps or hidden corners, clarifies their types and states, and also includes updated information on the location of each residue, providing detailed and accurate guidance for the precise operation of the cleaning robot.

[0083] In some embodiments, the method further includes: optimizing the lightweight neural network according to a dynamic IOU post-processing algorithm.

[0084] In an embodiment of the present invention, the dynamic IOU (Intersection over Union) post-processing algorithm is mainly used to improve the accuracy of target detection and recognition. When the lightweight neural network is identifying residual tobacco leaves, the output includes the predicted residual tobacco leaf position information (presented in the form of a border) and the corresponding confidence score. The position and size of these prediction boxes are the basis for the subsequent calculation of IOU, and the confidence score reflects the reliability of the model for this prediction, which can provide a key reference for the adjustment of the dynamic IOU threshold. Lightweight networks often use multi-scale feature fusion technology to enhance detection capabilities, and feature maps of different scales correspond to receptive fields of different sizes. In the output stage, it is ensured that the fused features can capture the overall outline of large pieces of tobacco leaves, and take into account the fine position of small broken fragments, so that the final output prediction box is more accurate, the initial error is reduced, and a higher quality input is provided for subsequent dynamic IOU optimization.

[0085] According to the confidence scores output by the network, all prediction results are divided into different levels. For example, confidence scores above 0.8 are classified as high confidence layers, those between 0.5 and 0.8 are medium confidence layers, and those below 0.5 are low confidence layers. For different levels, different initial IOU threshold ranges are set. The high confidence layer can be set to [0.3, 0.5], the medium confidence layer to [0.5, 0.7], and the low confidence layer to [0.7, 0.9].

[0086] Combined with target characteristics: Consider the type and spatial state of the residual tobacco leaves to further fine-tune the threshold. For large intact tobacco leaves, because of their regular shape and easy identification, the threshold can be appropriately lowered at the same confidence level; for broken tobacco leaf fragments in the gap, the threshold is increased accordingly due to the difficulty of identification.

[0087] When training lightweight neural networks, in addition to the commonly used classification loss (such as cross entropy loss) and positioning loss (such as L1 or L2 loss), a new IOU-based loss function is added. The weight of the IOU loss is dynamically assigned according to different confidence levels and target characteristics. For high-confidence and easily identifiable targets, the IOU loss weight is reduced so that the model does not overfit these simple samples; for low-confidence and difficult-to-identify targets, the IOU loss weight is increased to encourage the model to focus on optimizing this key detection effect.

[0088] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0089] Figure 2 Schematic diagram of the structure of the residual tobacco leaf identification device based on machine vision in the turnover box provided by the embodiment of the present invention. Figure 2As shown, the residual tobacco leaf identification device for the turnover box based on machine vision includes:

[0090] An image acquisition module 210 is used to acquire an image of a turnover box;

[0091] The first recognition module 220 is used to input the turnover box image into the contour recognition module of the tobacco leaf recognition model to obtain initial residual information;

[0092] The second recognition module 230 is used to input the turnover box image and the initial residue information into the type recognition module of the tobacco leaf recognition model to obtain the residue type, the residue space state and the residue position.

[0093] Optionally, the type recognition module includes an image feature extraction unit, a contour association feature extraction unit and a fusion unit; the second recognition module 230 is used to input the initial residual information and the turnover box image into the image feature extraction unit to obtain a multi-level feature map; input the multi-level feature map and the initial residual information into the contour association feature extraction unit to determine the spatial feature information; input the spatial feature information and the multi-level feature map into the fusion unit to obtain the residual type, residual spatial state and residual position.

[0094] Optionally, the second recognition module 230 is used to segment the turnover box image according to the initial residual information to obtain a segmented image; select the scale of the first convolution kernel corresponding to each segmented image according to the initial residual information corresponding to each segmented image; input each segmented image into the corresponding first convolution kernel to obtain basic local features; input the basic local features into a multi-layer second convolution kernel to obtain a multi-level feature map.

[0095] Optionally, the second recognition module 230 is used to input the initial residual information into the contour association feature extraction unit to obtain the inner contour feature; input the multi-level feature map into the contour association feature extraction unit to obtain the spatial association feature; and determine the spatial feature information based on the inner contour feature and the spatial association feature.

[0096] Optionally, the second recognition module 230 is used to determine the first feature according to the spatial feature information and the multi-level feature image; input the first feature into the fusion unit to obtain the residue type, the residue spatial state and the residue position.

[0097] Optionally, the device also includes: a third recognition unit for acquiring dynamic images and the shooting area of ​​the camera in real time; inputting the residual type, residual spatial state and dynamic image of the residual tobacco leaves in the shooting area into a lightweight neural network to obtain residual correction information.

[0098] Optionally, the third recognition unit is used to optimize the lightweight neural network according to a dynamic IOU post-processing algorithm.

[0099] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for identifying residual tobacco leaves in a turnover box based on machine vision, characterized in that: include: Get the turnover box image; Inputting the turnover box image into a contour recognition module of a tobacco leaf recognition model to obtain initial residual information; The turnover box image and the initial residue information are input into a type recognition module of a tobacco leaf recognition model to obtain the residue type, residue spatial state and residue position.

2. The method for identifying residual tobacco leaves in a turnover box based on machine vision according to claim 1, characterized in that: The type recognition module includes an image feature extraction unit, a contour association feature extraction unit and a fusion unit; the turnover box image and the initial residue information are input into the type recognition module of the tobacco leaf recognition model to obtain the residue type, residue space state and residue position, including: Inputting the initial residual information and the turnover box image into the image feature extraction unit to obtain a multi-level feature map; Inputting the multi-level feature map and the initial residual information into the contour-associated feature extraction unit to determine spatial feature information; The spatial feature information and the multi-level feature map are input into a fusion unit to obtain a residue type, a residue spatial state and a residue position.

3. The method for identifying residual tobacco leaves in a turnover box based on machine vision according to claim 2, characterized in that: The initial residual information and the turnover box image are input into the image feature extraction unit to obtain a multi-level feature map, including: Segmenting the turnover box image according to the initial residual information to obtain a segmented image; According to the initial residual information corresponding to each segmented image, the scale of the first convolution kernel corresponding to each segmented image is selected; Input each segmented image into the corresponding first convolution kernel to obtain basic local features; The basic local features are input into the multi-layer second convolution kernel to obtain a multi-level feature map.

4. The method for identifying residual tobacco leaves in a turnover box based on machine vision according to claim 2, characterized in that: Inputting the multi-level feature map and the initial residual information into the contour-related feature extraction unit to determine spatial feature information includes: Inputting the initial residual information into the contour-associated feature extraction unit to obtain contour internal features; Inputting the multi-level feature map into the contour association feature extraction unit to obtain spatial association features; The spatial feature information is determined according to the features within the contour and the spatial association features.

5. The method for identifying residual tobacco leaves in a turnover box based on machine vision according to claim 2, characterized in that: The spatial feature information and the multi-level feature map are input into a fusion unit to obtain a residue type, a residue spatial state and a residue position, including: Determining a first feature according to the spatial feature information and the multi-level feature image; The first feature is input into the fusion unit to obtain the residue type, the residue space state and the residue position.

6. The method for identifying residual tobacco leaves in a turnover box based on machine vision according to claim 1, characterized in that: The method further comprises: Get dynamic images and camera shooting area in real time; The residual type, residual spatial state and the dynamic image of the residual tobacco leaves in the shooting area are input into the lightweight neural network to obtain residual correction information.

7. The method for identifying residual tobacco leaves in a turnover box based on machine vision according to claim 6, characterized in that: The method further comprises: The lightweight neural network is optimized according to the dynamic IOU post-processing algorithm.

8. A device for identifying residual tobacco leaves in a turnover box based on machine vision, characterized in that: include: An image acquisition module, used to acquire images of turnover boxes; A first recognition module, used for inputting the turnover box image into a contour recognition module of a tobacco leaf recognition model to obtain initial residual information; The second recognition module is used to input the turnover box image and the initial residue information into the type recognition module of the tobacco leaf recognition model to obtain the residue type, residue spatial state and residue position.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for identifying residual tobacco leaves in a turnover box based on machine vision as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for identifying residual tobacco leaves in a turnover box based on machine vision as described in any one of claims 1 to 7 are implemented.

Images