Method and device for detecting abnormal straw images

Abstract

This application provides a method and device for detecting abnormal straw images. The method comprises: inputting multiple straw images to be detected into a pre-trained neural network model; and determining abnormal straw images among the multiple straw images to be detected based on the classification results and similarity results output by the neural network model, wherein the classification results are used to indicate the category of the straw images to be detected, and the similarity results are used to indicate the similarity between the straw images to be detected and a template straw image. This technical solution can determine abnormal straw images among the straw images to be detected based on the output results of the model, thereby reducing the number of manual image reviews when selecting erroneous data.

Description

Technical Field

[0001] The present application relates to the field of agriculture, and more specifically, to a method and a detection device for detecting abnormal straw images. Background Art

[0002] In the agricultural sector, it is often necessary to collect photos of work plots to obtain some information about the work plots. For example, by training a neural network model using the collected photos of the work plots, the trained neural network model can be used to directly identify the type of straw in the work plot, allowing agricultural machinery to perform operations selectively, thereby improving the degree of agricultural automation. However, during specific operations, when collecting photos of the work plots, they will be affected by factors such as occlusion, light, shooting angle, and various interferences. This will cause the captured images to deviate significantly from the training set, which will affect the straw classification results of the work plots.

[0003] In addition, when using neural networks to classify the straw types in the work plots, there will be a certain accuracy rate. For example, the accuracy rate is 90%. Then, for the 10% recognition errors, manual review is required to determine which pictures are wrong. When the amount of data is large, the workload of manual review is large, which will consume a lot of energy and time. Summary of the Invention

[0004] The embodiments of the present application provide a method and a detection device for detecting abnormal straw images. This technical solution can determine the abnormal straw images in the straw images to be detected based on the degree of difference between the straw images to be detected and the template straw images, thereby reducing the workload of users in manually reviewing and identifying erroneous data.

[0005] In a first aspect, a method for detecting abnormal straw pictures is provided, the method comprising: inputting multiple straw pictures to be detected into a pre-trained neural network model; determining abnormal straw pictures among the multiple straw pictures to be detected based on classification results and similarity results output by the neural network model, wherein the classification result is used to represent the category of the straw picture to be detected, and the similarity result is used to represent the similarity between the straw picture to be detected and the template straw picture.

[0006] In the embodiment of the present application, abnormal straw pictures among the straw pictures to be detected can be determined based on the degree of difference between the straw pictures to be detected and the template straw pictures, thereby reducing the workload of the user in manually reviewing and identifying erroneous data.

[0007] In combination with the first aspect, in an implementation method of the first aspect, the abnormal straw picture among the multiple straw pictures to be detected is determined based on the classification results and similarity results output by the neural network model, including: when the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, determining that the straw picture to be detected is an abnormal straw picture.

[0008] In combination with the first aspect, in an implementation method of the first aspect, the template straw picture includes N types, and when the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, the straw picture to be detected is determined to be an abnormal straw picture, including: when the maximum similarity among the N similarities between the straw picture to be detected and the N types in the template straw picture is less than a first preset value, the straw picture to be detected is determined to be an abnormal straw picture; wherein N is an integer greater than or equal to 2.

[0009] Based on the embodiment of the present application, whether the straw picture to be detected is abnormal can be determined by the similarity between the straw picture to be detected and the template, which is helpful to reduce the workload of the review.

[0010] In combination with the first aspect, in an implementation method of the first aspect, determining the abnormal straw picture among the multiple straw pictures to be detected based on the classification result and similarity result output by the neural network model includes: when the classification result of the straw picture to be detected is inconsistent with the similarity result, determining that the straw picture to be detected is an abnormal straw picture.

[0011] Based on the embodiment of the present application, when the classification result of the straw picture to be detected is inconsistent with the similarity result, it can be determined that the straw picture to be detected is an abnormal straw picture.

[0012] In combination with the first aspect, in an implementation method of the first aspect, the template straw picture includes N types, and when the classification result of the straw picture to be detected is inconsistent with the similarity result, the straw picture to be detected is determined to be an abnormal straw picture, including: when the similarity between the straw picture to be detected and the type corresponding to the classification result of the straw picture to be detected is not the maximum value among the N similarities, determining that the straw picture to be detected is an abnormal straw picture; wherein N is an integer greater than or equal to 2.

[0013] In conjunction with the first aspect, in an implementation of the first aspect, before inputting the plurality of straw images to be detected into the trained neural network model, the method further includes:

[0014] The training data set and the template data are input into the neural network model for training, or the features of the training data set and the template data are input into the neural network model for training.

[0015] In the embodiment of the present application, by adding template data or features of template data to train the model, some references can be provided for the training data set, which is helpful for determining the degree of association between the training data set and different templates.

[0016] In combination with the first aspect, in an implementation of the first aspect, the inputting the features of the training data set and the template data into the neural network model for training includes: in the neural network model, performing self-attention learning on the first feature and the second feature of the training data set, wherein the second feature is a feature obtained by random initialization; and performing cross-attention learning on the third feature obtained after self-attention learning of the second feature and the first feature and the fourth feature of the template data.

[0017] Based on the embodiment of the present application, the first feature and the second token of the training data set are subjected to self-attention learning. Through the self-attention mechanism learning, the network model can learn the relationship between different image nodes in the work plot, thereby obtaining the global expression of the work plot; through this cross-attention learning, the correlation features between the feature representation of the work plot and the features of the template can be unified, thereby simulating the process of the human brain understanding and judging through prior knowledge when recognizing unfamiliar things.

[0018] In conjunction with the first aspect, in an implementation of the first aspect, the method further includes:

[0019] The abnormal straw picture is stored in a first collection.

[0020] This technical solution is helpful for finding straw pictures with model recognition errors from the first set.

[0021] In combination with the first aspect, in an implementation of the first aspect, the loss function of the neural network model is:

[0022] L total =α·L cls +β·L dis ;

[0023] in,

[0024] α represents the weight of the classification loss function, β represents the weight of the similarity loss function; w i Indicates the score of the category, c i Indicates the predicted category; A i The feature vector representing the template straw image features, B i The eigenvector representing the standard data, L dis Calculates the cosine distance between two vectors.

[0025] In the embodiment of the present application, a similarity loss function is added so that the similarity between the straw image to be detected and the template straw image can be calculated to identify abnormal straw images.

[0026] In combination with the first aspect, in an implementation of the first aspect, the template straw images include at least two of the following types: interference, no straw, individual images with straw, a small amount of continuous straw, a large amount of continuous straw, and full amount of straw.

[0027] In a second aspect, a detection device is provided, comprising: an input module for inputting multiple straw pictures to be detected into a pre-trained neural network model; a processing module for determining abnormal straw pictures among the multiple straw pictures to be detected based on the classification results and similarity results output by the neural network model; wherein the classification result is used to represent the category of the straw picture to be detected, and the similarity result is used to represent the similarity between the straw picture to be detected and the template straw picture.

[0028] In combination with the second aspect, in an implementation of the second aspect, the processing module is specifically used to: when the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, determine that the straw picture to be detected is an abnormal straw picture.

[0029] In combination with the second aspect, in an implementation method of the second aspect, the template straw picture includes N types, and the processing module is specifically used to: when the similarity between the straw picture to be detected and the N similarity results of the N types in the template straw picture is less than a first preset value, determine that the straw picture to be detected is an abnormal straw picture; wherein N is an integer greater than or equal to 2.

[0030] In combination with the second aspect, in an implementation of the second aspect, the processing module is specifically configured to: when the classification result of the straw picture to be detected is inconsistent with the similarity result, determine that the straw picture to be detected is an abnormal straw picture.

[0031] In combination with the second aspect, in an implementation method of the second aspect, the template straw picture includes N types, and the processing module is specifically used to: when the similarity between the straw picture to be detected and the type corresponding to the classification result of the straw picture to be detected is not the maximum value among the N similarities, determine that the straw picture to be detected is an abnormal straw picture; wherein N is an integer greater than or equal to 2.

[0032] In combination with the second aspect, in an implementation of the second aspect, before the input module inputs multiple straw pictures to be detected into the trained neural network model, the device also includes: a training module, which is used to input the training data set and template data into the neural network model for training, or, input the features of the training data set and template data into the neural network model for training.

[0033] In combination with the second aspect, in an implementation of the second aspect, the training module is specifically used to: in the neural network model, perform self-attention learning on the first feature and the second feature of the training data set, wherein the second feature is a feature obtained by random initialization; and perform cross-attention learning on the third feature obtained after self-attention learning of the second feature and the first feature and the fourth feature of the template data.

[0034] In combination with the second aspect, in an implementation of the second aspect, the processing module is further used to: store the abnormal straw picture in a first set.

[0035] In conjunction with the second aspect, in an implementation of the second aspect, the loss function of the neural network model is:

[0036] L total =α·L cls +β·L dis ;

[0037] in,

[0038] α represents the weight of the classification loss function, β represents the weight of the similarity loss function; w i Indicates the score of the category, c i Indicates the predicted category; A i The feature vector representing the template straw image features, B i The eigenvector representing the standard data, L dis Calculates the cosine distance between two vectors.

[0039] In combination with the second aspect, in an implementation of the second aspect, the template straw images include at least two of the following types: interference, no straw, individual images with straw, a small amount of continuous straw, a large amount of continuous straw, and the entire amount of straw.

[0040] In a third aspect, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed, the method for detecting abnormal straw images as described in the first aspect and any possible implementation thereof is executed.

[0041] In a fourth aspect, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when run, enables the method for detecting abnormal straw images as described in the first aspect and any possible implementation thereof to be executed.

[0042] In a fifth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the method for detecting abnormal straw images as described in the first aspect and any possible implementation thereof is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the straw image classification provided in the embodiment of the present application.

[0044] Figure 2 This is a schematic diagram of improving the generalization ability of the neural network model in this application.

[0045] Figure 3 This is a schematic diagram of a neural network model provided in an embodiment of the present application.

[0046] Figure 4 This is a schematic flow chart of a method for detecting abnormal straw images provided in an embodiment of the present application.

[0047] Figure 5 It is a schematic diagram of Experiment 1 provided in the examples of this application.

[0048] Figure 6 This is a schematic diagram of Experiment 2 provided in the examples of this application.

[0049] Figure 7 This is a schematic diagram of Experiment 3 provided in the examples of this application.

[0050] Figure 8 It is a schematic block diagram of a detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0052] In the agricultural sector, it is often necessary to collect photos of work plots to obtain some information about them. For example, by training a neural network model using photos of the work plots, the trained neural network model can then be used to directly identify the type of straw in the work plots, allowing agricultural machinery to perform operations selectively, thereby improving the level of agricultural automation. However, during specific operations, when collecting photos of the work plots, they are affected by factors such as occlusion, lighting, shooting angles, and various interferences. This can result in significant deviations between the captured images and the training set, affecting the classification results of the work plots.

[0053] In addition, when using neural networks to classify the straw types in the work plots, there will be a certain accuracy rate. For example, the accuracy rate is 90%. Then, for the 10% recognition errors, manual review is required to determine which pictures are wrong. When the amount of data is large, the workload of manual review is large, which will consume a lot of energy and time.

[0054] In view of this, an embodiment of the present application provides a method and a detection device for detecting abnormal straw pictures. This technical solution can determine the abnormal straw pictures among the straw pictures to be detected based on the output results of the model, thereby reducing the number of manual review pictures when selecting erroneous data.

[0055] The following will be combined Figures 1 to 8 Introduce the technical solutions in the embodiments of this application.

[0056] During the operation of agricultural machinery, the amount of straw contained in different plots is different. In order to accurately distinguish the amount of straw in the plots, it is necessary to classify the amount of straw in the plots. In this regard, some classifications are made for the collected straw images in the embodiment of the present application.

[0057] Figure 1 This is a schematic diagram of the straw image classification provided in the embodiment of this application. Figure 1 As shown, the straw images are classified according to the amount of straw in the collected working plots.

[0058] In some embodiments, straw images are divided into six categories: interference, no straw, individual images with straw, a small amount of continuous straw, a large amount of continuous straw, and full amount of straw.

[0059] It should be understood that pictures classified as "interference" can be considered to be collected pictures that are not the plot where the straw is located. For example, the content included in the picture is a machine in operation, which will cause some interference to the judgment of the straw amount, so it is classified as "interference".

[0060] The pictures classified as no straw, individual pictures with straw, continuous small amounts of straw, continuous large amounts of straw, and full amounts of straw include working plots, which are differentiated according to the amount of straw included.

[0061] In deep learning, the quality of the data set will affect the accuracy of the model. In the embodiment of the present application, the preparation of the data set may include the following steps:

[0062] Step 1: Get a picture of the work area.

[0063] In step 1, electronic devices such as mobile phones, cameras, and vehicle-mounted cameras can be used to take pictures of different work plots to obtain a large number of straw pictures. For example, a total of 3,000 different straw pictures can be obtained.

[0064] It should be understood that for the same working plot, multiple straw pictures of the working plot can be obtained by taking pictures from multiple angles, or multiple straw pictures of the working plot can be obtained by taking pictures in different weather conditions and at different time periods.

[0065] Step 2: Classify the images according to their characteristics.

[0066] For example, the pictures obtained in step 1 are divided into six categories according to the amount of straw in the pictures: interference, no straw, individual pictures with straw, a small amount of continuous straw, a large amount of continuous straw, and full amount of straw.

[0067] It should be understood that in some embodiments, pictures may be divided into more or fewer categories according to the amount of straw in the pictures, and this embodiment itself is not limited to this.

[0068] Step 3: Manually label the images of each category to obtain the data set.

[0069] For example, if the images are divided into six categories in step 2, then the images in each category can be labeled with a number in step 3. For example, images of the category "interference" can be labeled as 1, images of the category "no straw" can be labeled as 2, images of the category "some images with straw" can be labeled as 3, images of the category "small amounts of straw in a row" can be labeled as 4, images of the category "large amounts of straw in a row" can be labeled as 5, and images of the category "full amount of straw" can be labeled as 6.

[0070] Step 4: Initially input the model into the network for training to verify the rationality of the data set.

[0071] In step 4, the data set can be input into the network model for training to verify the rationality of the data set.

[0072] In an embodiment of the present application, the data set can be trained using a neural network model, and templates of various categories can be obtained through a series of parameter optimizations.

[0073] In the embodiment of the present application, for each type, ten straw pictures can be selected as templates for that type. These ten straw pictures serve as template straw pictures, and the templates of each category constitute a standard data set. In subsequent processing, the standard data set can be used to determine the difference between the straw picture to be detected and the standard data set. When the difference is large, the straw picture to be detected can be considered an abnormal picture, and then the abnormal picture can be automatically selected, thereby reducing the number of manually reviewed pictures when selecting erroneous data.

[0074] It should be understood that the number of pictures in each category can also be other values, for example, 15, 20, etc., which is not limited in the embodiment of the present application.

[0075] The loss function involved in the embodiment of this application is introduced below:

[0076] The loss function of this application mainly consists of two parts: classification loss function L cls And the similarity loss function L dis .

[0077] L total =α·L cls +β·L dis (1)

[0078] Among them, α represents the weight of the classification loss function and β represents the weight of the similarity loss function.

[0079] Classification loss function L cls is the cross entropy loss function, which is defined as follows:

[0080]

[0081] Among them, w i Indicates the score of the category, c i Represents the predicted category. The cross entropy loss function is a loss function that effectively finds the minimum target.

[0082] Similarity loss function L dis is defined as follows:

[0083]

[0084] Among them, A i The feature vector representing the template token, B i The eigenvector representing the standard data, L dis Calculates the cosine distance between two vectors.

[0085] Figure 2 This is a schematic diagram of improving the generalization ability of the neural network model in this application. Figure 2As shown, in order to increase the robustness of the neural network model in the embodiment of the present application, data enhancement processing can be performed on pictures of each category.

[0086] In some embodiments, some unimportant areas in the picture may be erased. For example, in the first group of pictures, image information of some areas (such as area A) in the picture may be erased.

[0087] Alternatively, the straw image can be rotated to obtain images of straw at different angles. Alternatively, the straw image can be cropped. Alternatively, noise, blur, color conversion, and other processing can be added to the straw image to achieve data enhancement.

[0088] It should be understood that the specific method of data enhancement in the embodiments of the present application is not limited.

[0089] Figure 3 This is a schematic diagram of a neural network model provided in an embodiment of the present application. Figure 3 As shown in FIG, the neural network model can be composed of six stages. For the sake of simplicity, only stage 1 and stage 6 are shown in the figure. It should be understood that the structure of the omitted stages 2 to 5 can be the same as stage 1.

[0090] Each stage can include two deep self-attention transformer blocks (TBs). In each transformer block, the first feature (token) and the second token of the straw image to be detected are subjected to self-attention learning. This self-attention learning mechanism enables the network model to learn the relationship between different image nodes in the work area, thereby obtaining a global representation of the work area.

[0091] It should be understood that the second token of the straw image to be detected can be randomly generated or preset, and this embodiment of the present application is not limited thereto.

[0092] For example, the straw image to be detected is divided into nine small patches, and the nine small patches are the first token of the image to be detected. The second token is the size of a small patch. The second token performs self-attention learning with each small patch of the image to be detected, so that the second token can learn the characteristics of the entire straw image to be detected.

[0093] While the first and second tokens of the straw image to be detected are learned through a self-attention mechanism, the third token obtained through self-attention learning is then used with the fourth token of the template straw image to learn through a cross-attention mechanism. This cross-attention learning unifies the correlation between the feature representation of the work plot and the features of the template, simulating the process by which the human brain uses prior knowledge to understand and judge unfamiliar objects.

[0094] It should be understood that the fourth token is generated in a similar manner to the second token. For example, the template straw image is divided into nine small blocks, and the initialization token of the template straw image and the above nine small blocks are subjected to self-attention learning to obtain the fourth token.

[0095] Figure 4 This is a schematic flow chart of a method for detecting abnormal straw images provided in an embodiment of the present application. Figure 4 As shown, the method may include steps 410 to 420.

[0096] 410 , multiple straw images to be detected are input into a pre-trained neural network model.

[0097] It should be understood that the multiple straw images to be detected can be straw images of different working plots taken using electronic devices such as mobile phones, cameras, and vehicle-mounted cameras.

[0098] 420. Determine abnormal straw pictures among the multiple straw pictures to be detected based on the classification results and similarity results output by the neural network model, wherein the classification results are used to represent the category of the straw pictures to be detected, and the similarity results are used to represent the similarity between the straw pictures to be detected and the template straw pictures.

[0099] In this step, the neural network model can output both the classification of multiple straw images to be tested and the similarity between the straw image to be tested and the template straw image. This helps determine whether the straw image to be tested is abnormal and reduces the workload of subsequent manual review.

[0100] In some embodiments, determining abnormal straw images among the plurality of straw images to be detected based on the classification results and similarity results output by the neural network model includes:

[0101] When the similarity result between the straw picture to be detected and the template straw picture is less than a first preset value, it is determined that the straw picture to be detected is an abnormal straw picture.

[0102] Optionally, the template straw picture includes N types, and when the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, determining that the straw picture to be detected is an abnormal straw picture includes:

[0103] When the maximum similarity between the N similarities between the straw picture to be detected and the template straw picture is less than a first preset value, determining that the straw picture to be detected is an abnormal straw picture;

[0104] Wherein, N is an integer greater than or equal to 2.

[0105] For example, when the template straw images are divided into six categories, for each of the multiple straw images to be detected, the neural network model will calculate the similarity between the image and each of the six categories of template straw images, and obtain the similarity between the image and the category. For example, for straw image A to be detected, six similarity values ​​will be obtained through calculation, namely similarity A, similarity B, similarity C, similarity D, similarity E, and similarity F. When all six similarity values ​​are less than a first preset value, it can be determined that the similarity between the straw image A to be detected and the six categories of template straw images is small, and the straw image to be detected can be determined to be an abnormal straw image.

[0106] It should be understood that the first preset value may be pre-set, for example, it may be 0.5, 0.7, etc. The embodiment of the present application does not limit the specific value of the first preset value.

[0107] In other embodiments, determining abnormal straw images among the plurality of straw images to be detected based on the classification results and similarity results output by the neural network model includes:

[0108] When the classification result of the straw picture to be detected is inconsistent with the similarity result, it is determined that the straw picture to be detected is an abnormal straw picture.

[0109] In this embodiment, whether the straw picture to be detected is an abnormal straw picture can be determined by combining the classification result and the similarity result of the straw picture to be detected.

[0110] Optionally, the template straw picture includes N types, and when the classification result of the straw picture to be detected is inconsistent with the similarity result, determining that the straw picture to be detected is an abnormal straw picture includes:

[0111] When the similarity between the image of straw to be detected and the template type to which the classification result of the image of straw to be detected belongs is not the maximum value among N similarities, the image of straw to be detected is determined to be abnormal straw. When manually reviewing the image, refer to the result of abnormal elimination to locate the image where the model is most likely to make an error. If the classification accuracy after removing the abnormal image can meet the business requirements, there is no need to review the image again, and only consider abnormal elimination.

[0112] Stalk pictures;

[0113] Wherein, N is an integer greater than or equal to 2.

[0114] For example, N=6, that is, the template straw images are divided into six categories. The classification result of the straw image A to be detected is category A, but the similarity result between the straw image A to be detected and category A of the template straw type is not the maximum value among the six similarity results. In this case, the straw image to be detected can be determined to be an abnormal straw image. For example, the category of the straw image A to be detected is category A, and its similarity result with category A in the template straw image is 0.55, its similarity with category B in the template straw image is 0.61, its similarity with category C in the template straw image is 0.44, its similarity with category D in the template straw image is 0.43, its similarity with category E in the template straw image is 0.46, and its similarity with category F in the template straw image is 0.41. In this case, it can be determined that the straw image A to be detected has the greatest similarity with category B in the template straw image, but its classification result is category A. In this case, the straw image A to be detected can be determined to be an abnormal straw image.

[0115] In some embodiments, before inputting the plurality of straw images to be detected into the trained neural network model, the method further comprises:

[0116] The training data set and the template data are input into the neural network model for training, or the features of the training data set and the template data are input into the neural network model for training.

[0117] It should be understood that the template data may be the template straw picture mentioned above.

[0118] For example, if straw pictures are divided into six categories according to the classification method mentioned above, the training data set and the six categories of template straw pictures can be input into the neural network model together, so that the template straw pictures can provide a reference for the training data set, which is conducive to the subsequent calculation of the similarity between the training data set and the template straw pictures.

[0119] In another example, features of the training dataset and template data are input into a trained neural network model.

[0120] It should be understood that the feature of the template data can be the token of the template straw picture (such as the fourth token in the previous article). In this step, the training data set can be converted into a feature vector after (convolutional neural networks, CNN) mapping (projection), and then the token of the template straw picture is read and input into the neural network model for training.

[0121] The token of the template straw image may be pre-generated, for example, one token is generated for each type of template straw image.

[0122] In the embodiment of the present application, by adding template data or features of template data to train the model, some references can be provided for the training data set, which is helpful for determining the degree of association between the training data set and different templates.

[0123] In some embodiments, inputting the features of the training dataset and template data into the neural network model for training includes:

[0124] In the neural network model, self-attention learning is performed on a first feature and a second feature of the training data set, wherein the second feature is a feature obtained by random initialization;

[0125] A third feature obtained by self-attention learning of the second feature and the first feature is subjected to cross-attention learning with the fourth feature of the template data.

[0126] For example, see Figure 3 The first feature can be the small patches (patches) arranged in sequence into which the straw images in the training dataset are divided, and the second feature (token) can be randomly initialized, and then the second token is self-attention-learned with the first feature; thereafter, the second token is updated to the third feature (token), and then cross-attention-learned with the fourth token of the template data.

[0127] Based on the embodiment of the present application, the first feature and the second token of the training data set are subjected to self-attention learning. Through the self-attention mechanism learning, the network model can learn the relationship between different image nodes in the work plot, thereby obtaining the global expression of the work plot; through this cross-attention learning, the correlation features between the feature representation of the work plot and the features of the template can be unified, thereby simulating the process of the human brain understanding and judging through prior knowledge when recognizing unfamiliar things.

[0128] In some embodiments, the method further comprises:

[0129] The abnormal straw picture is stored in a first collection.

[0130] It should be understood that the first set may be located in a folder of the electronic device, or in a target path.

[0131] In an embodiment of the present application, the identified abnormal straw pictures can be stored together, which is convenient for users to view.

[0132] In some embodiments, the loss function of the neural network model is:

[0133] L total =α·L cls +β·L dis ;

[0134] in,

[0135] α represents the weight of the classification loss function, β represents the weight of the similarity loss function; w i Indicates the score of the category, c i Indicates the predicted category; A i The feature vector representing the template straw image features, B i The eigenvector representing the standard data, L dis Calculates the cosine distance between two vectors.

[0136] In some embodiments, the template straw images include at least two types of the following: interference, no straw, individual images with straw, a small amount of continuous straw, a large amount of continuous straw, and a full amount of straw.

[0137] It should be understood that the types included in the template straw image can also be divided in other ways, which is not limited in the embodiment of the present application.

[0138] The following will combine several experiments to verify the effect of the technical solution in the embodiment of this application.

[0139] Experiment 1: The template token is updated with parameters, but its loss function is not updated.

[0140] When training the neural network model, the tokens of the training dataset and the template tokens of the standard dataset (template data) are input into the neural network model for self-attention learning.

[0141] like Figure 5 As shown, Figure 5 This is a diagram of Experiment 1. The classification token from the training dataset and the template token from the standard dataset are fed into the neural network model for self-attention learning. The template token parameters are continuously updated during training. It should be understood that the classification token can be the second token mentioned above.

[0142] The classification token of the training data set is used to calculate the loss function loss1 (softmax) to classify the straw pictures included in the training data set. The classification token is then matmuled with the template token as the loss function loss2 to calculate the similarity between the straw picture to be detected and the template straw picture. The loss2 part is not updated.

[0143] After the neural network model is trained, the test set is used to verify the model.

[0144] The results of Experiment 1 are shown in Table 1.

[0145] Table 1 Results of Experiment 1

[0146] Total amount of straw images to be tested 813 Number of abnormal images detected 267 Original classification accuracy 86％ Classification accuracy after removing abnormal images 91％ Abnormal exclusion ratio 33％

[0147] As shown in Table 1, the total number of straw pictures to be detected is 813, and the original classification accuracy without abnormality detection on the straw pictures to be detected is 86%; when the technical solution of the present application is used to detect abnormal straw pictures on the 813 straw pictures to be detected, the number of abnormal straw pictures detected is 267, accounting for 33% of the total straw pictures to be detected (abnormality exclusion ratio); at the same time, the classification accuracy is improved to 91%.

[0148] In this technical solution, the original classification accuracy was 86%, while the recognition error rate was 14%. To identify the data that was misidentified, users might need to manually review 813 images to find the 14% of misidentified data. When the detection algorithm was used to detect abnormal straw images, the classification accuracy increased to 91%, and the model's recognition error rate was 9%. This improved classification accuracy means that users only need to search for the 9% of misidentified data among the remaining 546 images, reducing the amount of data required for manual review.

[0149] Experiment 2: Update the template data.

[0150] When training the neural network model, the template data (template straw image) and the training data set are input into the neural network model together.

[0151] like Figure 6 As shown, Figure 6 This is a diagram of Experiment 2. The classification tokens in the training dataset are used to calculate loss function loss1, which is used to classify the straw images in the training dataset. The template data is updated during training. The classification tokens are dot-multiplied (matmul) with the template data to form loss function loss2, which is used to calculate the similarity between the straw images in the training dataset and the template data.

[0152] It should be understood that the classification token can be the second token mentioned above.

[0153] After the neural network model is trained, the test set is used to verify the model.

[0154] The results of Experiment 2 are shown in Table 2.

[0155] Table 2 Results of Experiment 2

[0156] Total amount of straw images to be tested 813 Number of abnormal images detected 224 Original classification accuracy 86％ Classification accuracy after removing abnormal images 92％ Abnormal exclusion ratio 28％

[0157] As shown in Table 2, the total number of straw pictures to be detected is 813, and the original classification accuracy without abnormality detection on the straw pictures to be detected is 86%; when the technical solution of the present application is used to detect abnormal straw pictures on the 813 straw pictures to be detected, the number of abnormal straw pictures detected is 224, accounting for 28% of the total straw pictures to be detected (abnormality exclusion ratio); at the same time, the classification accuracy is increased to 92%.

[0158] In this technical solution, the original classification accuracy was 86%, while the recognition error rate was 14%. To identify the data that was misidentified, users might need to manually review 813 images to find the 14% of misidentified data. When the detection algorithm was used to detect abnormal straw images, the classification accuracy increased to 92%, and the model's recognition error rate was 8%. This improved classification accuracy means that users only need to search for the 8% of misidentified data from the remaining 589 images, reducing the amount of data required for manual review.

[0159] Experiment 3: Template token is used to update parameters and a cross attention layer is added.

[0160] like Figure 7 As shown, Figure 7 This is a diagram of Experiment 3. When training the neural network model, the training dataset tokens and initialization tokens are self-attentioned, and the classification tokens obtained after updating the initialization tokens are cross-attentioned with the template tokens of the template data. The template token parameters are continuously updated during training.

[0161] The classification token obtained after cross-attention learning is used to calculate the loss function loss1, and the template token is multiplied by the template data as the loss function loss2, which is used to train the similarity between the straw images in the dataset and the template data.

[0162] After the neural network model is trained, the test set is used to verify the model.

[0163] The results of Experiment 3 are shown in Table 3.

[0164] Table 3 Results of Experiment 3

[0165] Total amount of straw images to be tested 813 Number of abnormal images detected 117 Original classification accuracy 86％ Classification accuracy after removing abnormal images 91％ Abnormal exclusion ratio 14％

[0166] As shown in Table 3, the total number of straw pictures to be detected is 813, and the original classification accuracy without abnormality detection on the straw pictures to be detected is 86%; when the technical solution of the present application is used to detect abnormal straw pictures on the 813 straw pictures to be detected, the number of abnormal straw pictures detected is 117, accounting for 14% of the total straw pictures to be detected (abnormality exclusion ratio); at the same time, the classification accuracy is improved to 91%.

[0167] In this technical solution, the original classification accuracy was 86%, while the recognition error rate was 14%. To identify the data that was misidentified, users might need to manually review 813 images to find the 14% of misidentified data. When the detection algorithm was used to detect abnormal straw images, the classification accuracy increased to 91%, and the model's recognition error rate was 9%. This improved classification accuracy means that users only need to search for the 9% of misidentified data among the remaining 696 images, reducing the amount of data required for manual review.

[0168] Figure 8 This is a schematic block diagram of a detection device provided in an embodiment of the present application. Figure 8 As shown, the detection device 500 may include an input module 510 and a processing module 520 .

[0169] The input module 510 is used to input multiple straw pictures to be detected and template straw pictures into a pre-trained neural network model, or to input features of multiple straw pictures to be detected and template straw pictures into a pre-trained neural network model;

[0170] The processing module 520 is used to determine the abnormal straw picture among the multiple straw pictures to be detected based on the classification results and similarity results output by the neural network model, wherein the classification result is used to represent the category of the straw picture to be detected, and the similarity result is used to represent the similarity between the straw picture to be detected and the template straw picture.

[0171] It should be understood that the input module 510 and the processing module 520 can also be used to execute corresponding actions in the method for detecting abnormal straw images as described in any of the above embodiments to achieve corresponding functions.

[0172] An embodiment of the present application also provides a chip, which includes a processor and a communication interface. The communication interface is used to receive a signal and transmit the signal to the processor. The processor processes the signal so that the method for detecting abnormal straw images as described in the above embodiment is executed.

[0173] An embodiment of the present application also provides a computer-readable storage medium having a computer program or instruction stored thereon, which, when executed, causes a computer to execute the method for detecting abnormal straw images as described in any one of the foregoing.

[0174] An embodiment of the present application also provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute the method for detecting abnormal straw images as described in any one of the foregoing.

[0175] An embodiment of the present application further provides a processing device, including a processor and an interface; the processor is used to execute the method in any of the above method embodiments.

[0176] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] The methods in the embodiments of the present application, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The storage medium includes at least: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0178] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for detecting abnormal straw images, characterized in that: include: Input multiple straw images to be detected into the pre-trained neural network model; Determining abnormal straw images among the plurality of straw images to be detected according to the classification results and similarity results output by the neural network model, including: When the classification result of the straw picture to be detected is inconsistent with the similarity result, determining that the straw picture to be detected is an abnormal straw picture; Wherein, the classification result is used to represent the category of the straw picture to be detected, and the similarity result is used to represent the similarity between the straw picture to be detected and the template straw picture; Before inputting the plurality of straw images to be detected into the trained neural network model, the features of the training data set and the template data are input into the neural network model for training, including: In the neural network model, self-attention learning is performed on a first feature and a second feature of the training data set, wherein the second feature is a feature obtained by random initialization; Performing cross-attention learning on a third feature obtained by performing self-attention learning on the second feature and the first feature and a fourth feature of the template data; The neural network model outputs the classification result and similarity result based on the result of the cross-attention learning.

2. The method according to claim 1, characterized in that The step of determining abnormal straw images among the plurality of straw images to be detected based on the classification results and similarity results output by the neural network model includes: When the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, the straw picture to be detected is determined to be an abnormal straw picture.

3. The method according to claim 2, characterized in that The template straw pictures include N types, and when the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, determining that the straw picture to be detected is an abnormal straw picture includes: When the maximum similarity between the N similarities between the straw picture to be detected and the template straw picture is less than a first preset value, determining that the straw picture to be detected is an abnormal straw picture; Wherein, N is an integer greater than or equal to 2.

4. The method according to claim 1, wherein The template straw pictures include N types. When the classification result of the straw picture to be detected is inconsistent with the similarity result, determining that the straw picture to be detected is an abnormal straw picture includes: When the similarity between the straw picture to be detected and the type corresponding to the classification result of the straw picture to be detected is not the maximum value among N similarities, determining that the straw picture to be detected is an abnormal straw picture; Wherein, N is an integer greater than or equal to 2.

5. The method according to claim 3 or 4, characterized in that The loss function of the neural network model is: L total =α·L cls +β·L dis ; in, α represents the weight of the classification loss function, β represents the weight of the similarity loss function; w i Indicates the score of the category, c i Indicates the predicted category; A i The feature vector representing the template straw image features, B i Represents the feature vector of the straw image to be detected, L dis Calculates the cosine distance between two vectors.

6. The method according to any one of claims 1 to 4, characterized in that The template straw images include at least two types of the following: interference, no straw, individual images with straw, a small amount of continuous straw, a large amount of continuous straw, and the entire amount of straw.

7. A device for detecting abnormal straw images, characterized in that: include: An input module is used to input multiple straw images to be detected into a pre-trained neural network model; A processing module is used to determine an abnormal straw picture among the multiple straw pictures to be detected based on the classification result and the similarity result output by the neural network model; the processing module is specifically used to: when the classification result of the straw picture to be detected is inconsistent with the similarity result, determine that the straw picture to be detected is an abnormal straw picture; wherein the classification result is used to indicate the category of the straw picture to be detected, and the similarity result is used to indicate the similarity between the straw picture to be detected and the template straw picture; The device further comprises: The training module is used to input the features of the training data set and template data into the neural network model for training; the training module is specifically used to: In the neural network model, self-attention learning is performed on a first feature and a second feature of the training data set, wherein the second feature is a feature obtained by random initialization; Performing cross-attention learning on a third feature obtained by performing self-attention learning on the second feature and the first feature and a fourth feature of the template data; The classification result and similarity result are output based on the result of the cross attention learning.

8. The device according to claim 7, characterized in that The processing module is specifically used for: When the similarity between the straw picture to be detected and the template straw picture is less than a first preset value, the straw picture to be detected is determined to be an abnormal straw picture.

9. The device according to claim 8, characterized in that The template straw images include N types, and the processing module is specifically used to: When the similarity between the straw picture to be detected and the N similarity results of N types of straw pictures in the template straw picture is less than a first preset value, determining that the straw picture to be detected is an abnormal straw picture; Wherein, N is an integer greater than or equal to 2.

10. The device according to claim 7, characterized in that The template straw images include N types, and the processing module is specifically used to: When the similarity between the straw picture to be detected and the type corresponding to the classification result of the straw picture to be detected is not the maximum value among N similarities, determining that the straw picture to be detected is an abnormal straw picture; Wherein, N is an integer greater than or equal to 2.

11. The device according to claim 9 or 10, characterized in that The loss function of the neural network model is: L total =α·L cls +β·L dis ; in, α represents the weight of the classification loss function, β represents the weight of the similarity loss function; w i Indicates the score of the category, c i Indicates the predicted category; A i The feature vector representing the template straw image features, B i Represents the feature vector of the straw image to be detected, L dis Calculates the cosine distance between two vectors.

12. The device according to any one of claims 7 to 10, characterized in that The template straw images include at least two types of the following: interference, no straw, individual images with straw, a small amount of continuous straw, a large amount of continuous straw, and the entire amount of straw.

13. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the method for detecting abnormal straw images according to any one of claims 1 to 6 is executed.

14. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the method for detecting abnormal straw images according to any one of claims 1 to 6 is performed.

Images