Method, device, storage medium and computer equipment for evaluating raw smoke grade

Abstract

The present invention discloses a method, device, storage medium and computer equipment for evaluating the grade of raw tobacco, comprising: obtaining a multimodal image and origin attribute data of a target raw tobacco in response to a grade evaluation signal; obtaining a preset grade evaluation model, wherein the preset grade evaluation model comprises a feature extraction network, a difference perception fusion network and a self-attention network, the feature extraction network comprises a plurality of feature extraction layers, the difference perception fusion network comprises a feature fusion layer, and the self-attention network comprises a grade evaluation layer; inputting each image and the origin attribute data into the preset grade evaluation model, performing feature extraction on each image through the feature extraction network to obtain image features corresponding to each feature extraction layer, performing fusion on each image feature through the difference perception fusion network to obtain fusion features corresponding to the feature fusion layer, performing grade evaluation on the origin attribute data and the fusion features through the self-attention network, and determining the tobacco leaf grade based on the grade evaluation result.

Description

Technical Field

[0001] The present invention relates to the technical field of tobacco processing, and in particular to a method, device, storage medium and computer equipment for evaluating the grade of raw tobacco. Background Art

[0002] In order to make rational use of tobacco leaves and avoid waste of resources, tobacco leaves need to be graded.

[0003] At present, tobacco leaves are usually graded manually. However, this tobacco leaf grading method is greatly affected by human subjective factors, resulting in low grading accuracy. At the same time, the manual grading method also leads to low tobacco leaf grading efficiency. Summary of the invention

[0004] The present invention provides a method, device, storage medium and computer equipment for evaluating the original smoke grade, which are mainly capable of improving the evaluation efficiency and evaluation accuracy of the original smoke grade.

[0005] According to a first aspect of the present invention, there is provided a method for evaluating the original smoke grade, comprising:

[0006] In response to a grade evaluation signal of target raw tobacco, acquiring a multimodal image and origin attribute data of the target raw tobacco;

[0007] Obtaining a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, wherein the feature extraction network includes a plurality of feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set;

[0008] Input each image and the origin attribute data into the preset grade evaluation model, perform feature extraction on each image through the feature extraction network, obtain image features under the corresponding image corresponding to each feature extraction layer, fuse each image feature through the difference perception fusion network, obtain fusion features corresponding to the feature fusion layer, perform grade evaluation on the origin attribute data and the fusion features through the self-attention network, and obtain a grade evaluation result corresponding to the grade evaluation layer;

[0009] Based on the grade evaluation result, the tobacco grade of the target raw tobacco is determined.

[0010] Optionally, the multimodal image includes: a reflected light image, a transmitted light image, and a reflected light partial image, and the multiple feature extraction layers include: a reflection feature extraction layer corresponding to the reflected light image, a transmission feature extraction layer corresponding to the transmitted light image, and a reflection partial feature extraction layer corresponding to the reflected light partial image;

[0011] The feature extraction network is used to extract features from each image to obtain image features corresponding to each feature extraction layer, including:

[0012] The reflected light image is input into the feature extraction network for feature extraction, and the output of the reflection feature extraction layer is obtained as the reflection image feature. The transmitted light image is input into the feature extraction network for feature extraction, and the output of the transmission feature extraction layer is obtained as the transmission image feature. The reflected light local image is input into the feature extraction network for feature extraction, and the output of the reflection local feature extraction layer is obtained as the reflection local image feature.

[0013] Optionally, the self-attention network further includes an encoding layer and a feature concatenation layer;

[0014] The step of performing grade evaluation on the origin attribute data and the fusion feature through the self-attention network to obtain a grade evaluation result corresponding to the grade evaluation layer includes:

[0015] Inputting the origin attribute data into the coding layer for feature coding to obtain origin attribute features;

[0016] Inputting the origin attribute feature and the fusion feature into the feature stitching layer for feature stitching to obtain a stitching feature;

[0017] The splicing features are input into the grade evaluation layer for grade evaluation to obtain the grade evaluation result of the target original cigarette.

[0018] Optionally, before obtaining the preset grade evaluation model, the method further includes:

[0019] Constructing a preset initial grade assessment model;

[0020] Acquire a sample original tobacco grade evaluation data set, wherein the sample original tobacco grade evaluation data set includes multiple groups of sample images with annotation information, each group of the sample images includes multiple types of sample original tobacco images, and the annotation information includes the tobacco leaf grade of the sample original tobacco;

[0021] The sample original smoke level assessment data set is divided into a training set and a test set, the training set is used to train the preset initial level assessment model, and the test set is used to test the trained preset initial level assessment model, and the trained preset initial level assessment model that meets the test conditions is used as the preset level assessment model.

[0022] Optionally, obtaining a multimodal image of the target original smoke includes:

[0023] The target original smoke is illuminated by a uniform linear array light source emitted by a uniform linear array light source emitting device, the target original smoke is illuminated by a multi-angle multi-focus surface light source emitted by a combined light source emitting device, and the target original smoke is transmitted by a light source of preset brightness emitted by a water-cooled high-brightness linear array light source emitting device;

[0024] A reflected light image acquisition device is used to acquire reflected light images of the target original smoke after being irradiated by the uniform linear array light source and by the multi-angle multi-focus surface light source. A transmitted light image acquisition device is used to acquire transmitted light images of the target original smoke after being irradiated by the light source of preset brightness. A reflected light local image acquisition device is used to acquire reflected light local images of local areas of the target original smoke after being irradiated by the uniform linear array light source and by the multi-angle multi-focus surface light source.

[0025] Optionally, the reflected light local image acquisition device includes a plurality of sub-image acquisition devices;

[0026] Before using the reflected light local image acquisition device to acquire the reflected light local image of the target original smoke local area after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source, the method further includes:

[0027] Determining tobacco leaf position information of the target raw tobacco based on the reflected light image and the transmitted light image;

[0028] Based on the tobacco leaf position information, determining in each sub-image acquisition device a target sub-image acquisition device for acquiring an image of a local area of ​​the target original tobacco leaf;

[0029] The reflected light local image acquisition device is used to acquire the reflected light local image of the target original smoke local area after being illuminated by the uniform linear array light source and the multi-angle multi-focus surface light source, including:

[0030] The target sub-image acquisition device is used to acquire a local image of reflected light from a local area of ​​the target original smoke after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source.

[0031] Optionally, before using the reflected light local image acquisition device to acquire the reflected light local image of the target original smoke local area after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source, the method further includes:

[0032] Extracting feature points with significant geometric structures from the reflected light image and the transmitted light image respectively, performing cross-image matching on the feature points in the reflected light image and the transmitted light image, and determining feature point pairs with similar geometric structures in the reflected light image and the transmitted light image based on the matching results;

[0033] Based on the feature point pairs, determining image space coordinate transformation coefficients for spatially transforming the reflected light image into the transmitted light image, and based on the image space coordinate transformation coefficients, performing spatial transformation on the reflected light image to obtain a registered reflected light image consistent with the spatial distribution information of the transmitted light image;

[0034] Using the registered reflected light image and the transmitted light image as the bottommost pyramid layer images of the corresponding image pyramid to be constructed;

[0035] Each pyramid layer except the bottom pyramid layer in the image pyramid to be constructed is respectively used as a target pyramid layer, and based on the number of rows and columns of the image in the upper pyramid layer corresponding to the target pyramid layer, Gaussian blur processing and downsampling processing are sequentially performed on the image in the upper pyramid layer to obtain the target pyramid layer image, and based on the bottom pyramid layer image and each of the target pyramid layer images, image pyramids corresponding to each of the images are constructed, wherein the image pyramids corresponding to each of the images include: a reflection image pyramid corresponding to the registered reflected light image and a transmission image pyramid corresponding to the transmitted light image;

[0036] Taking any pyramid layer in each of the image pyramids as a current pyramid layer, performing upsampling processing and Gaussian convolution processing on the image in the current pyramid layer in sequence to obtain a predicted image of the current pyramid layer, subtracting the image of the current pyramid layer from the predicted image to obtain a current Laplacian pyramid layer image in the Laplacian pyramid to be constructed, and constructing a Laplacian pyramid corresponding to each of the images from each current Laplacian pyramid layer image, wherein the number of pyramid layers of the Laplacian pyramid corresponding to the registered reflected light image is the same as the number of pyramid layers of the Laplacian pyramid corresponding to the transmitted light image, and the resolutions of the images in the same Laplacian pyramid layer are the same;

[0037] Determine a low-resolution Laplacian pyramid layer having an image resolution less than a preset resolution threshold in the Laplacian pyramid corresponding to the registered reflected light image, and determine a mapped low-resolution Laplacian pyramid layer having the same number of layers as the low-resolution Laplacian pyramid layer in the Laplacian pyramid corresponding to the transmitted light image;

[0038] Performing weighted averaging fusion of the image in the low-resolution Laplacian pyramid layer and the image in the corresponding mapped low-resolution Laplacian pyramid layer to obtain a fused image corresponding to the low-resolution fusion layer;

[0039] Remaining layers of the Laplacian pyramid corresponding to the registered reflected light image after removing the low-resolution Laplacian pyramid layer are used as high-resolution Laplacian pyramid layers, and a mapped high-resolution Laplacian pyramid layer having the same number of layers as the high-resolution Laplacian pyramid layer is determined in the Laplacian pyramid corresponding to the transmitted light image;

[0040] determining a maximum pixel value at a same pixel position in the high-resolution Laplacian pyramid layer image and the mapped high-resolution Laplacian pyramid layer image, using the maximum pixel value at the same pixel position as a fused pixel value at a corresponding pixel position in a high-resolution fusion layer, and obtaining a fused image corresponding to the high-resolution fusion layer based on each fused pixel value;

[0041] A fused image pyramid is formed by the fused images corresponding to each low-resolution fusion layer and the fused images corresponding to each high-resolution fusion layer;

[0042] Reconstructing images layer by layer starting from the top layer of the fused image pyramid, and using the image reconstructed to the lowest layer of the fused image pyramid as the original smoke fused image corresponding to the reflected light image and the transmitted light image;

[0043] Based on the original smoke fusion image, the target original smoke is divided into a plurality of sub-regions, and a local target original smoke region with significant features is determined in each of the sub-regions.

[0044] According to a second aspect of the present invention, there is provided an original smoke grade evaluation device, comprising:

[0045] A first acquisition unit, configured to acquire a multimodal image and origin attribute data of the target raw tobacco in response to a grade evaluation signal of the target raw tobacco;

[0046] A second acquisition unit is used to acquire a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, wherein the feature extraction network includes a plurality of feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set;

[0047] A grade evaluation unit, used for inputting each image and the origin attribute data into the preset grade evaluation model, performing feature extraction on each image through the feature extraction network, obtaining image features under the corresponding image corresponding to each feature extraction layer, fusing each image feature through the difference perception fusion network, obtaining fusion features corresponding to the feature fusion layer, performing grade evaluation on the origin attribute data and the fusion features through the self-attention network, and obtaining a grade evaluation result corresponding to the grade evaluation layer;

[0048] A determination unit is used to determine the tobacco grade of the target raw tobacco based on the grade evaluation result.

[0049] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which implements the above raw smoke grade evaluation method when executed by a processor.

[0050] According to a fourth aspect of the present invention, there is provided a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above original smoke grade evaluation method when executing the program.

[0051] According to a method, device, storage medium and computer equipment for evaluating the grade of raw tobacco provided by the present invention, compared with the current method of manually grading tobacco leaves, the present invention adopts a special model architecture, including a feature extraction network of multiple feature extraction layers, a difference perception fusion network of a feature fusion layer, and a self-attention network of a grade evaluation layer, and uses a sample raw tobacco grade evaluation data set for training, thereby realizing a rapid and accurate evaluation of the grade of raw tobacco. The present invention uses a feature fusion mechanism to enable the model to simultaneously capture detail information and global information in different images, thereby more accurately evaluating the grade of raw tobacco. The sample raw tobacco grade evaluation data set designed for the tobacco leaf grade evaluation scenario is used for training, so that the model is more adapted to the raw tobacco grade evaluation environment and the robustness of the model is enhanced. The present application collects multimodal images and origin attribute data of the target raw tobacco, and conducts a comprehensive analysis of the multimodal images and origin attribute data to evaluate the grade of raw tobacco, which can further improve the evaluation accuracy of the grade of raw tobacco. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0053] Figure 1 A flow chart of a method for evaluating the original smoke grade provided by an embodiment of the present invention is shown;

[0054] Figure 2 A flow chart of another method for evaluating the original smoke grade provided by an embodiment of the present invention is shown;

[0055] Figure 3 A schematic diagram of the installation structure of an image acquisition device and a light source emitting device provided by an embodiment of the present invention is shown;

[0056] Figure 4 A schematic structural diagram of a device for evaluating the grade of raw smoke provided by an embodiment of the present invention is shown;

[0057] Figure 5 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0059] At present, the manual grading of tobacco leaves is greatly affected by human subjective factors, which will lead to low grading accuracy. At the same time, the manual grading method will also lead to low grading efficiency of tobacco leaves.

[0060] In order to solve the above problems, the embodiment of the present invention provides a method for evaluating the original smoke level, such as Figure 1 As shown, the method includes:

[0061] 101. In response to a grade evaluation signal of target raw tobacco, obtain a multimodal image and origin attribute data of the target raw tobacco.

[0062] Among them, multimodal images refer to images of target raw tobacco collected by different image acquisition devices; the origin attribute data include: climate environment data (such as ambient temperature, ambient humidity, rainfall, etc.) of the origin of the target raw tobacco when cultivating the target raw tobacco, soil condition data (such as soil type, soil texture, soil structure, plow layer thickness, soil nutrients, etc.), etc.

[0063] In the embodiments of the present invention, a plurality of different image acquisition devices are used to acquire multimodal images of the target raw tobacco, such as using radioactive light imaging technology to acquire reflected light images of the target raw tobacco, using projected light imaging technology to acquire projected light images of the target raw tobacco, and using reflected light local high-definition imaging technology to acquire reflected light local images of the target raw tobacco. Since the color, texture and other information of tobacco leaves can be acquired through reflected light images, the thickness and other aspects of tobacco leaves can be acquired through projected light images, and the local detail feature information of tobacco leaves can be acquired through reflected light local images, the color, texture, thickness, local detail feature information of tobacco leaves, climate environment data and soil condition data of the production area of ​​the target raw tobacco when cultivating the target raw tobacco can be comprehensively analyzed to evaluate the grade of the target raw tobacco, thereby improving the accuracy of the grade evaluation of the target raw tobacco.

[0064] 102. Obtain a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, the feature extraction network includes multiple feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set.

[0065] For the embodiment of the present invention, a sample original smoke grade assessment data set is used in advance to train and construct a preset grade assessment model. By optimizing the model structure and using a specific data set for training, the preset grade assessment model's ability to assess the grade of the original smoke is significantly improved. The preset grade assessment model used consists of three parts: a feature extraction network, a difference perception fusion network, and a self-attention network. Feature extraction network: multiple feature extraction layers are used. These feature extraction layers can effectively extract the features of multimodal images in parallel at the same time. Since there are significant visual differences between reflected light images and transmitted light images, as well as reflected light local high-definition images, a multi-branch feature extraction layer is used. The features of reflected light images, transmitted light images, and reflected light local high-definition images can be extracted simultaneously. The structure of multiple branch feature extraction layers is symmetrical, which ensures that the features extracted from three different input images have consistent sizes and prepares for subsequent feature fusion. Difference perception fusion network: It includes a feature fusion layer, which is responsible for exchanging features extracted by different feature extraction layers, using channel weighting to fully integrate the fusion of multimodal image features of global context, fusing their complementary features, and automatically or explicitly combining different features to generate new feature combinations. These combined features may contain complex nonlinear relationships between original features, so that the model can capture more detailed and rich information in the image, that is, it can make full use of the relationship between various images, extract more implicit features, and make the subsequent grade evaluation results more accurate. Self-attention network: It includes an encoding layer and a grade evaluation layer, where the encoding layer is a fully connected layer, which is used to encode the origin attribute data to extract the origin attribute features in the origin attribute data, and the grade evaluation layer is used to evaluate the origin attribute features and image fusion features, ensuring that all factors affecting the tobacco leaf grade are comprehensively analyzed, thereby improving the evaluation accuracy of the tobacco leaf grade. Dataset used for training the model: The sample original tobacco grade evaluation dataset is used for training. This dataset is designed specifically for the original smoke level assessment scenario and contains a large number of original smoke images with complex backgrounds and multiple modalities, which helps the model learn how to effectively assess the original smoke level in multimodal and complex environments.

[0066] 103. Input each image and origin attribute data into a preset grade evaluation model, perform feature extraction on each image through the feature extraction network, obtain image features under the corresponding image corresponding to each feature extraction layer, fuse each image feature through the difference perception fusion network, obtain fusion features corresponding to the feature fusion layer, perform grade evaluation on the origin attribute data and the fusion features through the self-attention network, and obtain grade evaluation results corresponding to the grade evaluation layer.

[0067] For the embodiment of the present invention, after obtaining the preset grade evaluation model, each image and origin attribute data are input into the preset grade evaluation model, that is, each image is first input into the feature extraction network for feature extraction, and different images correspond to different feature extraction layers. Different feature extraction layers in the feature extraction network are used to extract features of the corresponding images to obtain image features corresponding to different images, and then each image feature is input into the feature fusion layer in the difference perception fusion network, and each image feature is fused through the feature fusion layer to obtain fusion features. At the same time, the origin attribute data is input into the encoding layer for encoding processing to obtain the origin attribute feature, and the origin attribute feature and the fusion feature are spliced ​​to obtain the splicing feature, and finally the splicing feature is input into the grade evaluation layer for grade evaluation, and the grade evaluation result of the target original tobacco can be output through the grade evaluation layer. Thus, the grade of the original tobacco is evaluated by the preset grade evaluation model, which can improve the grade evaluation efficiency and grade evaluation accuracy of tobacco leaves.

[0068] 104. Based on the grade assessment results, determine the tobacco grade of the target raw tobacco.

[0069] In the embodiment of the present invention, after the grade of the target raw tobacco is evaluated by using a preset grade evaluation model to obtain a grade evaluation result, the tobacco leaf grade of the target raw tobacco is finally determined according to the grade evaluation result. Therefore, the embodiment of the present invention collects multimodal images and origin attribute data of the target raw tobacco, and comprehensively analyzes the multimodal images and origin attribute data to evaluate the raw tobacco grade, which can improve the evaluation accuracy of the raw tobacco grade.

[0070] According to a method for evaluating the grade of raw tobacco provided by the present invention, compared with the current method of manually grading tobacco leaves, the present invention adopts a special model architecture, including a feature extraction network of multiple feature extraction layers, a difference perception fusion network of a feature fusion layer, and a self-attention network of a grade evaluation layer, and uses a sample raw tobacco grade evaluation data set for training, thereby achieving a fast and accurate evaluation of the grade of raw tobacco. The present invention uses a feature fusion mechanism to enable the model to simultaneously capture detail information and global information in different images, thereby more accurately evaluating the grade of raw tobacco. The model is trained with a sample raw tobacco grade evaluation data set designed for the tobacco leaf grade evaluation scenario, so that the model is more adapted to the raw tobacco grade evaluation environment and the robustness of the model is enhanced. The present application collects multimodal images and origin attribute data of the target raw tobacco, and conducts a comprehensive analysis of the multimodal images and origin attribute data to evaluate the raw tobacco grade, which can further improve the evaluation accuracy of the raw tobacco grade.

[0071] Further, in order to better illustrate the process of evaluating the original smoke level, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another method for evaluating the original smoke level, such as Figure 2 As shown, the method includes:

[0072] 201. In response to the target original smoke level assessment signal, the target original smoke is illuminated by a uniform linear array light source emitted by a uniform linear array light source emitting device, the target original smoke is illuminated by a multi-angle and multi-focus surface light source emitted by a combined light source emitting device, and the target original smoke is transmitted by a light source of preset brightness emitted by a water-cooled high-brightness linear array light source emitting device.

[0073] 202. Use a reflected light image acquisition device to acquire reflected light images of the target original smoke after being illuminated by a uniform linear array light source and by a multi-angle multi-focus surface light source. Use a transmitted light image acquisition device to acquire transmitted light images of the target original smoke after being illuminated by a light source of preset brightness. Use a reflected light local image acquisition device to acquire reflected light local images of local areas of the target original smoke after being illuminated by a uniform linear array light source and by a multi-angle multi-focus surface light source.

[0074] Among them, the reflected light image acquisition device is such as a 4K 32-bit true color line array camera, the transmitted light image acquisition device is such as a 2K 32-bit true color line array camera, and the reflected light local image acquisition device is such as a 4K 32-bit true color line array camera.

[0075] In the embodiment of the present invention, in order to solve the problem of uneven image brightness when the tobacco leaves are in a wrinkled and curled high-speed motion state, and to make the collected image clearer and the features extracted later more accurate, it is necessary to use different light sources to illuminate the target original tobacco, such as Figure 3As shown, one of the various light source installation structures is illustrated. In the embodiment of the present invention, if the reflected light image, the projected light image, and the reflected light partial image of the target raw cigarette are to be collected, for example, the reflected light image: a reflected light image collection device such as a 4K 32-bit true color linear array camera, a uniform linear array light source emitting device such as an LED high-brightness high-uniform linear array light source, etc. can be used. The light source and the camera are located on the same side above the sorting conveyor line. At the same time, in order to solve the problem of uneven image brightness when the tobacco leaves are wrinkled and curled up at high speed, a combined light source emitting device is added. The combined light source emitting device can emit a multi-angle and multi-focus surface combined light source. By combining light sources of different angles and different focusing surfaces, the collected tobacco leaf image has stable color and uniform brightness. The uniform linear array light source emitted by the uniform linear array light source emitting device is used to illuminate the target raw cigarette. The multi-angle and multi-focus surface light sources emitted by the combined light source emitting device are used to illuminate the target raw cigarette. Then, a reflected light image collection device such as a 4K 32-bit true color linear array camera is used to collect images of the raw cigarette after being illuminated by the light source. Transmitted light imaging: using transmitted light image acquisition devices such as 2K 32-bit true color linear array cameras, water-cooled high-brightness linear array light sources emitted by water-cooled high-brightness linear array light source emitting devices (light source illumination reaches more than 5 million LX), the light source is located below the sorting conveyor line, and the camera is located above the sorting conveyor line. In order to enhance the penetration effect of transmitted light, two groups of water-cooled LED combinations can be used, and symmetrical installation is adopted. It is required that the focal lines of the two groups of light sources completely overlap. At the same time, the object distance of the camera is as small as possible, and a short focal length lens is selected. Then, the target raw cigarette is transmitted and illuminated by a light source with a preset brightness emitted by a water-cooled high-brightness linear array light source emitting device, and the image of the raw cigarette after the light source is illuminated is collected by a transmitted light image acquisition device. Reflected light local image: The main difference from the reflected light image is that the reflected light local image does not belong to the complete tobacco leaf area, but only performs high-definition imaging of a part of the tobacco leaf. For example, if the magnification is greater than 0.2, it has an effect similar to that of a low-power microscope, which can highlight the local detail features of the tobacco leaf surface. The reflected light local image acquisition device is used to collect reflected light local images of the local area of ​​the target original tobacco after irradiation by a uniform linear array light source and a multi-angle and multi-focus surface light source.

[0076] Furthermore, in order to enable the reflected light local image acquisition to capture local areas of tobacco leaves with fine detail features, it is necessary to set multiple sub-image acquisition devices for the reflected light local image acquisition device. In order to determine a target sub-image acquisition device that can accurately capture local images among the multiple sub-image acquisition devices, the method includes: determining the tobacco leaf position information of the target original tobacco based on the reflected light image and the transmitted light image; based on the tobacco leaf position information, determining a target sub-image acquisition device in each sub-image acquisition device to perform image acquisition on the local area of ​​the target original tobacco; using the reflected light local image acquisition device to capture the reflected light local image of the local area of ​​the target original tobacco after being irradiated by the uniform linear array light source and by the multi-angle multi-focus surface light source, including: using the target sub-image acquisition device to capture the reflected light local image of the local area of ​​the target original tobacco after being irradiated by the uniform linear array light source and by the multi-angle multi-focus surface light source.

[0077] Specifically, since the width of the sorting conveyor line is greater than the width of the tobacco leaves, and since the incoming tobacco leaves are random and the position of the tobacco leaves cannot be predicted, a combination of multiple area array cameras (multiple sub-image acquisition devices in the reflected light local image acquisition device) can be used. The combined area array camera is installed after the reflected light image acquisition device and the transmitted light image acquisition device. The position of the target original tobacco leaves is identified according to the reflected light image acquired by the reflected light image acquisition device and the transmitted light image acquired by the transmitted light image acquisition device, so as to determine the triggering of the target sub-image acquisition device in the combined area array camera to perform image acquisition of the local area of ​​the tobacco leaves, thereby reducing the image transmission volume and processing power requirements. For example, if the width of the sorting conveyor line is 350mm, a combination of four 5-megapixel area array cameras can be used, and the field of view covered by each camera is about 90mm.

[0078] Furthermore, in order to find a local area containing important detail features in the target original smoke, before collecting the local area image of the target original smoke, the method also includes: extracting feature points with significant geometric structures in the reflected light image and the transmitted light image respectively, and performing cross-image matching on the feature points in the reflected light image and the transmitted light image, and based on the matching results, determining feature point pairs with similar geometric structures in the reflected light image and the transmitted light image; based on the feature point pairs, determining image space coordinate transformation coefficients for spatially transforming the reflected light image into the transmitted light image, and based on the image space coordinate transformation coefficients, performing spatial transformation on the reflected light image to obtain to a registered reflected light image consistent with the spatial distribution information of the transmitted light image; the registered reflected light image and the transmitted light image are respectively used as the bottommost pyramid layer images of the image pyramid to be constructed; each pyramid layer except the bottommost pyramid layer in the image pyramid to be constructed is respectively used as a target pyramid layer, based on the number of rows and columns of the image in the previous pyramid layer corresponding to the target pyramid layer, Gaussian blur processing and downsampling processing are sequentially performed on the image in the previous pyramid layer to obtain the target pyramid layer image, and based on the bottommost pyramid layer image and each of the target pyramid layer images, image pyramids corresponding to each of the images are respectively constructed, wherein each The image pyramids corresponding to the images include: a reflection image pyramid corresponding to the registered reflected light image and a transmission image pyramid corresponding to the transmitted light image; any pyramid layer in each of the image pyramids is used as a current pyramid layer, and the image in the current pyramid layer is sequentially upsampled and Gaussian convolution is performed to obtain a predicted image of the current pyramid layer, and the image of the current pyramid layer is subtracted from the predicted image to obtain a current Laplacian pyramid layer image in the Laplacian pyramid to be constructed, and the Laplacian pyramids corresponding to the images are constructed from each current Laplacian pyramid layer image, wherein the Laplacian corresponding to the registered reflected light image is The number of pyramid layers of the pyramid is the same as the number of pyramid layers of the Laplace pyramid corresponding to the transmitted light image, and the resolutions of images in the same Laplace pyramid layer are the same; a low-resolution Laplace pyramid layer having an image resolution less than a preset resolution threshold is determined in the Laplace pyramid corresponding to the registered reflected light image, and a mapped low-resolution Laplace pyramid layer having the same number of layers as the low-resolution Laplace pyramid layer is determined in the Laplace pyramid corresponding to the transmitted light image; the image in the low-resolution Laplace pyramid layer is weighted averaged and fused with the image in the corresponding mapped low-resolution Laplace pyramid layer to obtain a fused image corresponding to the low-resolution fused layer;The remaining layers of the low-resolution Laplacian pyramid layer are removed from the Laplacian pyramid corresponding to the registered reflected light image as the high-resolution Laplacian pyramid layer, and a mapped high-resolution Laplacian pyramid layer with the same number of layers as the high-resolution Laplacian pyramid layer is determined in the Laplacian pyramid corresponding to the transmitted light image; the maximum pixel value at the same pixel position is determined in the high-resolution Laplacian pyramid layer image and the mapped high-resolution Laplacian pyramid layer image, and the maximum pixel value at the same pixel position is used as the fused pixel value at the corresponding pixel position in the high-resolution fusion layer, and based on each fused pixel value, a fused image corresponding to the high-resolution fusion layer is obtained; the fused images corresponding to each low-resolution fusion layer and the fused images corresponding to each high-resolution fusion layer form a fused image pyramid; the image is reconstructed layer by layer from the top layer of the fused image pyramid, and the image reconstructed to the lowest layer of the fused image pyramid is used as the original smoke fusion image corresponding to the reflected light image and the transmitted light image; based on the original smoke fusion image, the target original smoke is divided into a plurality of sub-regions, and a local target original smoke region with significant features is determined in each of the sub-regions. ;

[0079] Among them, the preset resolution threshold is set according to actual needs. Specifically, the reflected light image and the transmitted light image are first registered, and the registration method is: first, feature points with significant geometric structures in the reflected light image and the transmitted light image are extracted respectively, such as corner points, edge points, high curvature points and other points with unique properties, and the feature vectors corresponding to the feature points are determined, and then based on the feature vectors, the similarity between each feature point is calculated across images based on the feature vectors, and then based on the feature vectors of each feature point, the similarity between each feature point is calculated across images. For example, based on the feature vector corresponding to the feature point a in the reflected light image, the Euclidean distance calculation method can be used to calculate the similarity between the feature point a and each feature point in the transmitted light image, and the feature points with the largest similarity in the transmitted light image are determined as similar feature points of the feature point a, and the feature point a and its corresponding similar feature point constitute a feature point pair. Furthermore, after obtaining the matching feature point pairs, it is necessary to calculate the transformation matrix to establish the spatial correspondence between the two images, that is, the image space coordinate transformation coefficients, where the transformation matrix can be an affine transformation matrix, a projection transformation matrix, etc. Finally, one of the images is transformed using the calculated transformation matrix to align it with the other image. The transformed image is the registered image. The registered image in the embodiment of the present invention is a registered reflected light image.

[0080] Furthermore, after registering the two images, it is necessary to construct image pyramids corresponding to the two images respectively. The specific method of constructing the image pyramid is as follows: taking the transmitted light image as an example, firstly, the transmitted light image is used as the lowest level image of the image pyramid, and then the upper pyramid level image is Gaussian blurred and downsampled in turn to obtain each pyramid level image. When the image resolution reaches the preset requirement, the downsampling is terminated, so that the image pyramid is formed by each pyramid level image. Similarly, the image pyramid corresponding to the registered reflected light image can be constructed. Furthermore, on the basis of the image pyramid, taking the image pyramid corresponding to the transmitted light image as an example, the images in all the image pyramid levels are upsampled and Gaussian convolved to obtain a predicted image, and then the image in the original image pyramid level is subtracted from the upsampled image with the same resolution to obtain a predicted image, and each level difference image is obtained. Finally, a Laplacian pyramid is constructed from each level difference image. Similarly, a Laplacian pyramid corresponding to the registered reflected light image can be constructed. Furthermore, in the Laplacian pyramid corresponding to the registered reflected light image, the image of the low-resolution pyramid layer is weighted averaged and fused with the image of the same layer in the Laplacian pyramid corresponding to the transmitted light image, thereby obtaining a fused image corresponding to each low-resolution pyramid layer. For the remaining high-resolution pyramid layers in the reflected light image and the transmitted light image, the maximum pixel value is determined at the same pixel position of the high-resolution pyramid layer images of the same layer corresponding to the two images, and the pixel value is determined as the fused pixel value at the pixel position, thereby determining the fused pixel value at each pixel position, and finally forming a fused image of the high-resolution pyramid layer by each fused pixel value, and forming a fused image pyramid by the fused images corresponding to each low-resolution fused layer and the fused images corresponding to each high-resolution fused layer. Furthermore, for each layer (from the top to the bottom) in the fused image pyramid: the current layer is upsampled so that its resolution matches that of the next layer (finer scale), the upsampled image is added to the residual image of the next layer to obtain a reconstructed image of the current scale, and the reconstructed image of the current scale is used as the initial image of the next layer iteration, and image reconstruction is performed in the above manner. When the bottom layer of the fused image pyramid is reached, the reconstruction process ends, and the reconstructed image at this time is the original smoke fusion image obtained by fusing the reflected light image and the transmitted light image. The embodiment of the present invention performs image fusion by constructing an image pyramid. Due to the hierarchical structure of the image pyramid, the image can be fully represented at different scales. In the fusion process, the information of these different scales can be fully utilized to achieve a finer fusion effect. By fusing at different resolutions, more detail information can be captured, making the fused image clearer and more natural, thereby improving the performance of the fused image in the area detection task.

[0081] Furthermore, for the original smoke fusion image, it is possible to identify the texture, color, shape and other features of the target original smoke, as well as factors such as noise and lighting conditions in the image. Based on the above information, the target original smoke can be divided into multiple sub-regions using a region-based segmentation algorithm, and then the local region of the target original smoke with significant features is determined in each sub-region. The significant features can be the color features, texture features, shape features, etc. of the target original smoke. The embodiment of the present invention identifies the local region of the target original smoke with significant features by constructing a fused image. Since image fusion can remove artifacts and noise in the original image and combine the advantages of multiple images at the same time, the clarity of the image is improved and the details are enhanced, thereby improving the accuracy of the subsequent determination of the local region of the original smoke.

[0082] 203. Obtain a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, the feature extraction network includes multiple feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set.

[0083] For the embodiments of the present invention, in order to improve the prediction accuracy of the preset grade evaluation model, it is first necessary to train and construct the preset grade evaluation model, based on this, the method includes: constructing a preset initial grade evaluation model; obtaining a sample original smoke grade evaluation data set, wherein the sample original smoke grade evaluation data set includes multiple groups of sample images with annotation information, each group of the sample images contains multiple types of sample images of sample original smoke, and the annotation information includes the tobacco leaf grade of the sample original smoke; dividing the sample original smoke grade evaluation data set into a training set and a test set, using the training set to train the preset initial grade evaluation model, and using the test set to test the trained preset initial grade evaluation model, and using the trained preset initial grade evaluation model that meets the test conditions as the preset grade evaluation model.

[0084] Specifically, first build a preset initial level evaluation model, and the model structure can be as described above. Secondly, download the data set from the official website or the specified data source. Make sure that the data set contains all necessary files, including image files and corresponding annotation information. Convert the annotation file to a format that the preset initial level evaluation model can understand. Finally, train and test the model. Specifically, you can first divide the data set: use random or specific strategies (such as stratified sampling) to divide the data set into a training set and a test set. Then use the training set to train the model, monitor the loss value, mAP and other indicators during the training process to evaluate the model performance. Adjust the training parameters as needed, such as learning rate, optimizer, regularization, etc., to optimize the training effect. Finally, test the model: use the test set to test the trained model and evaluate its performance on unseen data. Calculate and record indicators such as mAP, precision, and recall on the test set. If the model performance does not meet the requirements, you can return to the training stage for more iterations or adjustments. Thus, a preset level evaluation model with high processing accuracy can be constructed in the above manner. In another embodiment of the present invention, during the model construction and training process, for example, the training parameters of the model are: input batch 64, training iterations are performed based on fine-tuning of the model parameters, the weight parameters of the feature extraction network are initialized before model training, the parameters are optimized using the Adam method, the initial learning rate is 0.005, and the learning rate is adjusted using a cosine decay method. Before training, the learning rate is preheated and iterated for 4 epochs (the period in which the entire data set is used once during the training process).

[0085] Furthermore, after constructing the preset grade evaluation model, the multimodal images, including reflected light images, transmitted light images, reflected light local images and origin attribute data, are input into the preset grade evaluation model, and the preset grade evaluation model can output the tobacco leaf grade evaluation results of the target raw tobacco.

[0086] 204. Input each image and origin attribute data into a preset grade evaluation model, perform feature extraction on each image through a feature extraction network, obtain image features under the corresponding image corresponding to each feature extraction layer, fuse each image feature through a difference perception fusion network, obtain fusion features corresponding to the feature fusion layer, perform grade evaluation on the origin attribute data and the fusion features through a self-attention network, and obtain grade evaluation results corresponding to the grade evaluation layer.

[0087] Among them, each image includes a reflected light image, a transmitted light image, and a reflected light local image; the multiple feature extraction layers include: a reflection feature extraction layer corresponding to the reflected light image, a transmission feature extraction layer corresponding to the transmitted light image, and a reflection local feature extraction layer corresponding to the reflected light local image.

[0088] For the embodiment of the present invention, after obtaining the reflected light image, transmitted light image, and reflected light partial image of the target original cigarette, the reflected light image, the transmitted light image, the reflected light partial image, and the origin attribute data of the target original cigarette are input into a preset grade evaluation model. First, it is necessary to use the feature extraction network in the preset grade evaluation model to perform parallel image feature extraction. Based on this, the method includes: inputting the reflected light image into the feature extraction network for feature extraction, obtaining the output of the reflection feature extraction layer as the reflection image feature, inputting the transmitted light image into the feature extraction network for feature extraction, obtaining the output of the transmission feature extraction layer as the transmission image feature, inputting the reflected light partial image into the feature extraction network for feature extraction, and obtaining the output of the reflection local feature extraction layer as the reflection local image feature.

[0089] Specifically, the reflected light image, the transmitted light image, and the reflected light partial image are respectively input into the corresponding feature extraction layers for parallel feature extraction, thereby obtaining the reflected light image features corresponding to the reflected light image, the transmitted light image features corresponding to the transmitted light image, and the reflected light partial image features corresponding to the reflected light partial image. The embodiment of the present invention can save the extraction time of multiple image features by setting multiple feature extraction layers to extract multiple image features in parallel, and can also avoid the problem of damage to the feature extraction network due to large calculation amount when using one feature extraction layer to extract features from multiple images.

[0090] Furthermore, the image features extracted by each feature extraction layer are input into the feature fusion layer in the difference perception fusion network, and each image feature is fused through the feature fusion layer. The specific fusion process can be: performing feature-level fusion processing on each image to obtain a feature fusion vector; performing element-level fusion processing on each image feature to obtain an element fusion feature; performing low-order fusion processing on each image feature to obtain a low-order fusion feature; using a preset transformation function to transform the feature fusion vector, the element fusion feature, and the low-order fusion feature to obtain a fusion feature. The embodiment of the present invention can automatically or explicitly combine different features by fusing each image feature to generate a new feature combination. These combined features may contain complex nonlinear relationships between the original features, so that the model can capture more detailed and rich information in the image, that is, it can make full use of the relationship between various digital images, extract more implicit features, and take into account both high-order and low-order processing, so that the image information is more fully utilized, and the grade evaluation result obtained later is more accurate, meeting the needs of actual application scenarios.

[0091] Furthermore, after fusing the image features of each image to obtain the fused features, the fused features and the origin attribute data need to be input into the self-attention network for processing to obtain the tobacco grade of the target raw tobacco. Based on this, the method includes: inputting the origin attribute data into the encoding layer for feature encoding to obtain the origin attribute features; inputting the origin attribute features and the fused features into the feature splicing layer for feature splicing to obtain the splicing features; inputting the splicing features into the grade evaluation layer for grade evaluation to obtain the grade evaluation result of the target raw tobacco.

[0092] Among them, the self-attention network includes an encoding layer, a feature splicing layer and a grade evaluation layer. Specifically, the origin attribute data of the target raw tobacco is first input into the encoding layer (position encoding layer) for encoding processing to obtain the origin attribute features, and then the origin attribute features and fusion features are input into the feature splicing layer for splicing processing to obtain splicing features, and finally the splicing features are input into the grade evaluation layer for grade evaluation to obtain the grade evaluation result of the target raw tobacco. Therefore, in the process of evaluating the tobacco leaf grade of the target raw tobacco, the multimodal image features and origin attribute features of the tobacco leaves are comprehensively analyzed, thereby improving the evaluation accuracy of the tobacco leaf grade.

[0093] 205. Based on the grade assessment results, determine the tobacco grade of the target raw tobacco.

[0094] Specifically, the preset grade evaluation model can predict the tobacco leaf grade evaluation result of the target raw tobacco. The grade evaluation result can be a tobacco leaf score, such as 95 points. Then, the score interval corresponding to the score is determined in the preset grade configuration table, and the tobacco leaf grade corresponding to the score interval is determined, such as grade A.

[0095] According to another method for evaluating the grade of raw tobacco provided by the present invention, compared with the current method of manually grading tobacco leaves, the present invention adopts a special model architecture, including a feature extraction network of multiple feature extraction layers, a difference perception fusion network of a feature fusion layer, and a self-attention network of a grade evaluation layer, and uses a sample raw tobacco grade evaluation data set for training, thereby achieving a fast and accurate evaluation of the grade of raw tobacco. The present invention uses a feature fusion mechanism to enable the model to simultaneously capture detail information and global information in different images, thereby more accurately evaluating the grade of raw tobacco. The sample raw tobacco grade evaluation data set designed for the tobacco leaf grade evaluation scenario is trained to make the model more adaptable to the raw tobacco grade evaluation environment and enhance the robustness of the model. The present application collects multimodal images and origin attribute data of the target raw tobacco, and conducts a comprehensive analysis of the multimodal images and origin attribute data to evaluate the grade of raw tobacco, which can further improve the evaluation accuracy of the grade of raw tobacco.

[0096] Further, as Figure 1In a specific implementation, an embodiment of the present invention provides an evaluation device for the original smoke level, such as Figure 4 As shown, the device includes: a first acquisition unit 31, a second acquisition unit 32, a level evaluation unit 33, and a determination unit 34.

[0097] The first acquisition unit 31 may be configured to acquire a multimodal image and origin attribute data of the target raw tobacco in response to a grade evaluation signal of the target raw tobacco.

[0098] The second acquisition unit 32 can be used to acquire a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, the feature extraction network includes multiple feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set.

[0099] The grade assessment unit 33 can be used to input each image and the origin attribute data into the preset grade assessment model, perform feature extraction on each image through the feature extraction network, obtain image features under the corresponding image corresponding to each feature extraction layer, fuse each image feature through the difference perception fusion network, obtain fusion features corresponding to the feature fusion layer, perform grade assessment on the origin attribute data and the fusion features through the self-attention network, and obtain grade assessment results corresponding to the grade assessment layer.

[0100] The determination unit 34 may be configured to determine the tobacco grade of the target raw tobacco based on the grade evaluation result.

[0101] In a specific application scenario, the multimodal image includes: a reflected light image, a transmitted light image, and a reflected light local image, and the multiple feature extraction layers include: a reflection feature extraction layer corresponding to the reflected light image, a transmission feature extraction layer corresponding to the transmitted light image, and a reflection local feature extraction layer corresponding to the reflected light local image; in order to perform feature extraction on each image, the level evaluation unit 33 can be specifically used to input the reflected light image into the feature extraction network for feature extraction, obtain the output of the reflection feature extraction layer as a reflected image feature, input the transmitted light image into the feature extraction network for feature extraction, obtain the output of the transmission feature extraction layer as a transmitted image feature, input the reflected light local image into the feature extraction network for feature extraction, and obtain the output of the reflection local feature extraction layer as a reflected local image feature.

[0102] In a specific application scenario, the self-attention network further includes a coding layer and a feature splicing layer; in order to perform grade evaluation on the target original smoke, the grade evaluation unit 33 includes a coding module 331, a feature splicing module 332, and a grade evaluation module 333.

[0103] The encoding module 331 can be used to input the origin attribute data into the encoding layer for feature encoding to obtain origin attribute features.

[0104] The feature stitching module 332 can be used to input the origin attribute feature and the fusion feature into the feature stitching layer to perform feature stitching to obtain a stitching feature.

[0105] The grade evaluation module 333 may be used to input the splicing features into the grade evaluation layer for grade evaluation to obtain a grade evaluation result of the target original cigarette.

[0106] In a specific scenario, in order to construct a preset grade evaluation model, the device further includes a construction unit 35 .

[0107] The construction unit 35 can be used to construct a preset initial grade evaluation model; obtain a sample original smoke grade evaluation data set, wherein the sample original smoke grade evaluation data set includes multiple groups of sample images with annotation information, each group of sample images contains multiple types of sample images of sample original smoke, and the annotation information includes the tobacco leaf grade of the sample original smoke; divide the sample original smoke grade evaluation data set into a training set and a test set, use the training set to train the preset initial grade evaluation model, and use the test set to test the trained preset initial grade evaluation model, and use the trained preset initial grade evaluation model that meets the test conditions as the preset grade evaluation model.

[0108] In a specific application scenario, in order to obtain a multimodal image of the target original smoke, the first acquisition unit 31 includes an illumination module 311 and an image acquisition module 312 .

[0109] The illumination module 311 can be used to illuminate the target original smoke using a uniform linear array light source emitted by a uniform linear array light source emitting device, to illuminate the target original smoke using a multi-angle multi-focus surface light source emitted by a combined light source emitting device, and to transmitly illuminate the target original smoke using a light source of preset brightness emitted by a water-cooled high-brightness linear array light source emitting device.

[0110] The image acquisition module 312 can be used to use a reflected light image acquisition device to acquire a reflected light image of the target original smoke after being illuminated by the uniform linear array light source and the multi-angle multi-focus surface light source, use a transmitted light image acquisition device to acquire a transmitted light image of the target original smoke after being illuminated by the light source of the preset brightness, and use a reflected light local image acquisition device to acquire a reflected light local image of a local area of ​​the target original smoke after being illuminated by the uniform linear array light source and the multi-angle multi-focus surface light source.

[0111] In a specific application scenario, the reflected light local image acquisition device includes a plurality of sub-image acquisition devices; in order to acquire the reflected light local image, the first acquisition unit 31 further includes a determination module 313 .

[0112] The determination module 313 may be used to determine tobacco leaf position information of the target original tobacco based on the reflected light image and the transmitted light image.

[0113] The determination module 313 may be specifically used to determine, in each sub-image acquisition device, a target sub-image acquisition device for acquiring an image of a local area of ​​the target original tobacco based on the tobacco leaf position information.

[0114] The image acquisition module 312 can be specifically used to use the target sub-image acquisition device to acquire a local image of reflected light from a local area of ​​the target original smoke after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source.

[0115] In a specific application scenario, in order to determine the local area of ​​the target original smoke, the first acquisition unit 31 also includes: a matching module 314, a space transformation module 315, a construction module 316, an image fusion module 317, an image reconstruction module 318, and a division module 319.

[0116] The matching module 314 can be used to extract feature points with significant geometric structures in the reflected light image and the transmitted light image, respectively, and perform cross-image matching on the feature points in the reflected light image and the transmitted light image, and based on the matching results, determine feature point pairs with similar geometric structures in the reflected light image and the transmitted light image.

[0117] The spatial transformation module 315 can be used to determine the image space coordinate transformation coefficients for spatially transforming the reflected light image into the transmitted light image based on the feature point pairs, and to spatially transform the reflected light image based on the image space coordinate transformation coefficients to obtain a registered reflected light image that is consistent with the spatial distribution information of the transmitted light image.

[0118] The determination module 313 may also be configured to use the registered reflected light image and the transmitted light image as the bottommost pyramid layer images of the corresponding image pyramid to be constructed.

[0119] The construction module 316 may be configured to use each pyramid layer except the bottom pyramid layer in the image pyramid to be constructed as a target pyramid layer, perform Gaussian blur processing and downsampling processing on the image in the upper pyramid layer in sequence based on the number of rows and columns of the image in the upper pyramid layer corresponding to the target pyramid layer to obtain the target pyramid layer image, and construct image pyramids corresponding to each of the images based on the bottom pyramid layer image and each of the target pyramid layer images, wherein the image pyramids corresponding to each of the images include: a reflection image pyramid corresponding to the registered reflected light image and a transmission image pyramid corresponding to the transmitted light image.

[0120] The construction module 316 may also be configured to use any pyramid layer in each of the image pyramids as a current pyramid layer, perform upsampling processing and Gaussian convolution processing on the image in the current pyramid layer in sequence to obtain a predicted image of the current pyramid layer, subtract the image of the current pyramid layer from the predicted image to obtain a current Laplacian pyramid layer image in the Laplacian pyramid to be constructed, and construct a Laplacian pyramid corresponding to each of the images from each current Laplacian pyramid layer image, wherein the number of pyramid layers of the Laplacian pyramid corresponding to the registered reflected light image is the same as the number of pyramid layers of the Laplacian pyramid corresponding to the transmitted light image, and the resolutions of the images in the same Laplacian pyramid layer are the same.

[0121] The determining module 313 may also be configured to determine, in the Laplacian pyramid corresponding to the registered reflected light image, a low-resolution Laplacian pyramid layer having an image resolution less than a preset resolution threshold, and determine, in the Laplacian pyramid corresponding to the transmitted light image, a mapped low-resolution Laplacian pyramid layer having the same number of layers as the low-resolution Laplacian pyramid layer.

[0122] The image fusion module 317 may be used to perform weighted averaging fusion of the image in the low-resolution Laplacian pyramid layer and the image in the corresponding mapped low-resolution Laplacian pyramid layer to obtain a fused image corresponding to the low-resolution fusion layer.

[0123] The determining module 313 may also be configured to use remaining layers of the Laplacian pyramid corresponding to the registered reflected light image, after removing the low-resolution Laplacian pyramid layer, as high-resolution Laplacian pyramid layers, and determine, in the Laplacian pyramid corresponding to the transmitted light image, a mapped high-resolution Laplacian pyramid layer having the same number of layers as the high-resolution Laplacian pyramid layer.

[0124] The determining module 313 may also be configured to determine a maximum pixel value at the same pixel position in the high-resolution Laplacian pyramid layer image and the mapped high-resolution Laplacian pyramid layer image, use the maximum pixel value at the same pixel position as a fused pixel value at a corresponding pixel position in the high-resolution fusion layer, and obtain a fused image corresponding to the high-resolution fusion layer based on the fused pixel values.

[0125] The construction module 316 may be used to construct a fused image pyramid using fused images corresponding to each low-resolution fusion layer and fused images corresponding to each high-resolution fusion layer.

[0126] The image reconstruction module 318 may be configured to reconstruct the image layer by layer starting from the top layer of the fused image pyramid, and use the image reconstructed to the lowest layer of the fused image pyramid as the original smoke fused image corresponding to both the reflected light image and the transmitted light image.

[0127] The division module 319 may be used to divide the target original smoke into a plurality of sub-regions based on the original smoke fusion image, and determine a local region of the target original smoke having significant features in each of the sub-regions.

[0128] It should be noted that for other corresponding descriptions of the functional modules involved in the original smoke grade evaluation device provided in the embodiment of the present invention, reference can be made to Figure 1 The corresponding description of the method shown will not be repeated here.

[0129] Based on the above Figure 1The method shown, accordingly, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: in response to a grade evaluation signal of the target original cigarette, a multimodal image and origin attribute data of the target original cigarette are obtained; a preset grade evaluation model is obtained, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, wherein the feature extraction network includes multiple feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; The preset grade evaluation model is pre-trained based on a sample raw tobacco grade evaluation data set; each image and the origin attribute data are input into the preset grade evaluation model, and features are extracted from each image through the feature extraction network to obtain image features under the corresponding image corresponding to each feature extraction layer; each image feature is fused through the difference perception fusion network to obtain fusion features corresponding to the feature fusion layer; the origin attribute data and the fusion features are graded through the self-attention network to obtain a grade evaluation result corresponding to the grade evaluation layer; based on the grade evaluation result, the tobacco leaf grade of the target raw tobacco is determined.

[0130] Based on the above Figure 1 The method shown and Figure 4 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 5As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 implements the following steps when executing the program: in response to a grade evaluation signal of the target raw cigarette, obtaining a multimodal image and origin attribute data of the target raw cigarette; obtaining a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, wherein the feature extraction network includes a plurality of feature extraction layers, and the difference perception fusion network includes a feature fusion layer, The self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample raw tobacco grade evaluation data set; each image and the origin attribute data are input into the preset grade evaluation model, and the feature extraction network is used to extract features from each image respectively to obtain image features under the corresponding image of each feature extraction layer, and each image feature is fused through the difference perception fusion network to obtain a fusion feature corresponding to the feature fusion layer, and the origin attribute data and the fusion feature are graded through the self-attention network to obtain a grade evaluation result corresponding to the grade evaluation layer; based on the grade evaluation result, the tobacco leaf grade of the target raw tobacco is determined.

[0131] Through the technical solution of the present invention, the present invention realizes rapid and accurate evaluation of the grade of raw tobacco by adopting a special model architecture, including a feature extraction network of multiple feature extraction layers, a difference perception fusion network of the feature fusion layer, and a self-attention network of the grade evaluation layer, and using a sample raw tobacco grade evaluation data set for training. The present invention uses a feature fusion mechanism to enable the model to simultaneously capture detail information and global information in different images, thereby more accurately evaluating the grade of raw tobacco. Training is performed on a sample raw tobacco grade evaluation data set designed for the tobacco leaf grade evaluation scenario, so that the model is more adapted to the raw tobacco grade evaluation environment and enhances the robustness of the model. The present application collects multimodal images and origin attribute data of the target raw tobacco, and conducts a comprehensive analysis of the multimodal images and origin attribute data to evaluate the raw tobacco grade, which can further improve the evaluation accuracy of the raw tobacco grade.

[0132] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating the original smoke grade, characterized in that: include: In response to a grade evaluation signal of target raw tobacco, acquiring a multimodal image and origin attribute data of the target raw tobacco; Obtaining a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, wherein the feature extraction network includes a plurality of feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set; Input each image and the origin attribute data into the preset grade evaluation model, perform feature extraction on each image through the feature extraction network, obtain image features under the corresponding image corresponding to each feature extraction layer, fuse each image feature through the difference perception fusion network, obtain fusion features corresponding to the feature fusion layer, perform grade evaluation on the origin attribute data and the fusion features through the self-attention network, and obtain a grade evaluation result corresponding to the grade evaluation layer; Based on the grade evaluation result, the tobacco grade of the target raw tobacco is determined.

2. The method according to claim 1, characterized in that The multimodal image includes: a reflected light image, a transmitted light image, and a reflected light partial image, and the plurality of feature extraction layers include: a reflection feature extraction layer corresponding to the reflected light image, a transmission feature extraction layer corresponding to the transmitted light image, and a reflection partial feature extraction layer corresponding to the reflected light partial image; The feature extraction network is used to extract features from each image to obtain image features corresponding to each feature extraction layer, including: The reflected light image is input into the feature extraction network for feature extraction, and the output of the reflection feature extraction layer is obtained as the reflection image feature. The transmitted light image is input into the feature extraction network for feature extraction, and the output of the transmission feature extraction layer is obtained as the transmission image feature. The reflected light local image is input into the feature extraction network for feature extraction, and the output of the reflection local feature extraction layer is obtained as the reflection local image feature.

3. The method according to claim 1, characterized in that The self-attention network also includes an encoding layer and a feature concatenation layer; The step of performing grade evaluation on the origin attribute data and the fusion feature through the self-attention network to obtain a grade evaluation result corresponding to the grade evaluation layer includes: Inputting the origin attribute data into the coding layer for feature coding to obtain origin attribute features; Inputting the origin attribute feature and the fusion feature into the feature stitching layer for feature stitching to obtain a stitching feature; The splicing features are input into the grade evaluation layer for grade evaluation to obtain the grade evaluation result of the target original cigarette.

4. The method according to claim 1, characterized in that: Before obtaining the preset grade evaluation model, the method further includes: Constructing a preset initial grade assessment model; Acquire a sample original tobacco grade evaluation data set, wherein the sample original tobacco grade evaluation data set includes multiple groups of sample images with annotation information, each group of the sample images includes multiple types of sample original tobacco images, and the annotation information includes the tobacco leaf grade of the sample original tobacco; The sample original smoke level assessment data set is divided into a training set and a test set, the training set is used to train the preset initial level assessment model, and the test set is used to test the trained preset initial level assessment model, and the trained preset initial level assessment model that meets the test conditions is used as the preset level assessment model.

5. The method according to claim 1, characterized in that Acquiring a multimodal image of the target original smoke, including: The target original smoke is illuminated by a uniform linear array light source emitted by a uniform linear array light source emitting device, the target original smoke is illuminated by a multi-angle multi-focus surface light source emitted by a combined light source emitting device, and the target original smoke is transmitted by a light source of preset brightness emitted by a water-cooled high-brightness linear array light source emitting device; A reflected light image acquisition device is used to acquire reflected light images of the target original smoke after being irradiated by the uniform linear array light source and by the multi-angle multi-focus surface light source. A transmitted light image acquisition device is used to acquire transmitted light images of the target original smoke after being irradiated by the light source of preset brightness. A reflected light local image acquisition device is used to acquire reflected light local images of local areas of the target original smoke after being irradiated by the uniform linear array light source and by the multi-angle multi-focus surface light source.

6. The method according to claim 5, characterized in that The reflected light local image acquisition device includes a plurality of sub-image acquisition devices; Before using the reflected light local image acquisition device to acquire the reflected light local image of the target original smoke local area after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source, the method further includes: Determining tobacco leaf position information of the target raw tobacco based on the reflected light image and the transmitted light image; Based on the tobacco leaf position information, determining in each sub-image acquisition device a target sub-image acquisition device for acquiring an image of a local area of ​​the target original tobacco leaf; The reflected light local image acquisition device is used to acquire the reflected light local image of the target original smoke local area after being illuminated by the uniform linear array light source and the multi-angle multi-focus surface light source, including: The target sub-image acquisition device is used to acquire a local image of reflected light from a local area of ​​the target original smoke after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source.

7. The method according to claim 5, characterized in that Before using the reflected light local image acquisition device to acquire the reflected light local image of the target original smoke local area after being illuminated by the uniform linear array light source and by the multi-angle multi-focus surface light source, the method further includes: Extracting feature points with significant geometric structures from the reflected light image and the transmitted light image respectively, performing cross-image matching on the feature points in the reflected light image and the transmitted light image, and determining feature point pairs with similar geometric structures in the reflected light image and the transmitted light image based on the matching results; Based on the feature point pairs, determining image space coordinate transformation coefficients for spatially transforming the reflected light image into the transmitted light image, and based on the image space coordinate transformation coefficients, performing spatial transformation on the reflected light image to obtain a registered reflected light image consistent with the spatial distribution information of the transmitted light image; Using the registered reflected light image and the transmitted light image as the bottommost pyramid layer images of the corresponding image pyramid to be constructed; Each pyramid layer except the bottom pyramid layer in the image pyramid to be constructed is respectively used as a target pyramid layer, and based on the number of rows and columns of the image in the upper pyramid layer corresponding to the target pyramid layer, Gaussian blur processing and downsampling processing are sequentially performed on the image in the upper pyramid layer to obtain the target pyramid layer image, and based on the bottom pyramid layer image and each of the target pyramid layer images, image pyramids corresponding to each of the images are constructed, wherein the image pyramids corresponding to each of the images include: a reflection image pyramid corresponding to the registered reflected light image and a transmission image pyramid corresponding to the transmitted light image; Taking any pyramid layer in each of the image pyramids as a current pyramid layer, performing upsampling processing and Gaussian convolution processing on the image in the current pyramid layer in sequence to obtain a predicted image of the current pyramid layer, subtracting the image of the current pyramid layer from the predicted image to obtain a current Laplacian pyramid layer image in the Laplacian pyramid to be constructed, and constructing a Laplacian pyramid corresponding to each of the images from each current Laplacian pyramid layer image, wherein the number of pyramid layers of the Laplacian pyramid corresponding to the registered reflected light image is the same as the number of pyramid layers of the Laplacian pyramid corresponding to the transmitted light image, and the resolutions of the images in the same Laplacian pyramid layer are the same; Determine a low-resolution Laplacian pyramid layer having an image resolution less than a preset resolution threshold in the Laplacian pyramid corresponding to the registered reflected light image, and determine a mapped low-resolution Laplacian pyramid layer having the same number of layers as the low-resolution Laplacian pyramid layer in the Laplacian pyramid corresponding to the transmitted light image; Performing weighted averaging fusion of the image in the low-resolution Laplacian pyramid layer and the image in the corresponding mapped low-resolution Laplacian pyramid layer to obtain a fused image corresponding to the low-resolution fusion layer; Remaining layers of the Laplacian pyramid corresponding to the registered reflected light image after removing the low-resolution Laplacian pyramid layer are used as high-resolution Laplacian pyramid layers, and a mapped high-resolution Laplacian pyramid layer having the same number of layers as the high-resolution Laplacian pyramid layer is determined in the Laplacian pyramid corresponding to the transmitted light image; determining a maximum pixel value at a same pixel position in the high-resolution Laplacian pyramid layer image and the mapped high-resolution Laplacian pyramid layer image, using the maximum pixel value at the same pixel position as a fused pixel value at a corresponding pixel position in a high-resolution fusion layer, and obtaining a fused image corresponding to the high-resolution fusion layer based on each fused pixel value; A fused image pyramid is formed by the fused images corresponding to each low-resolution fusion layer and the fused images corresponding to each high-resolution fusion layer; Reconstructing images layer by layer starting from the top layer of the fused image pyramid, and using the image reconstructed to the lowest layer of the fused image pyramid as the original smoke fused image corresponding to the reflected light image and the transmitted light image; Based on the original smoke fusion image, the target original smoke is divided into a plurality of sub-regions, and a local target original smoke region with significant features is determined in each of the sub-regions.

8. A device for evaluating the original smoke level, characterized in that: include: A first acquisition unit, configured to acquire a multimodal image and origin attribute data of the target raw tobacco in response to a grade evaluation signal of the target raw tobacco; A second acquisition unit is used to acquire a preset grade evaluation model, wherein the preset grade evaluation model includes a feature extraction network for extracting image features, a difference perception fusion network for fusing image features, and a self-attention network for grade evaluation, wherein the feature extraction network includes a plurality of feature extraction layers, the difference perception fusion network includes a feature fusion layer, and the self-attention network includes a grade evaluation layer; the preset grade evaluation model is pre-trained based on a sample original smoke grade evaluation data set; A grade evaluation unit, used for inputting each image and the origin attribute data into the preset grade evaluation model, performing feature extraction on each image through the feature extraction network, obtaining image features under the corresponding image corresponding to each feature extraction layer, fusing each image feature through the difference perception fusion network, obtaining fusion features corresponding to the feature fusion layer, performing grade evaluation on the origin attribute data and the fusion features through the self-attention network, and obtaining a grade evaluation result corresponding to the grade evaluation layer; A determination unit is used to determine the tobacco grade of the target raw tobacco based on the grade evaluation result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images