Grain pile surface pest detection model training method, detection method and equipment

Abstract

The invention provides a grain pile surface pest detection model training method, a detection method and equipment, and relates to the technical field of image recognition, and the training method comprises the steps: respectively generating different types of data samples based on grain pile surface image data marked with pest bounding box labels in a plurality of data fields in advance, respectively inputting the two model branches, so that the two model branches sequentially carry out backbone network feature extraction and backdoor adjustment on the data sample to obtain a target feature map and pest detection result data; and determining the causal feature alignment total loss corresponding to the current iteration round according to the target feature maps obtained by the two model branches so as to optimize the two model branches. According to the method, the domain generalization ability of the grain pile surface pest detection model and the adaptability to complex and changeable actual detection scenes can be effectively improved, the application convenience and effectiveness of the grain pile surface pest detection model can be improved, and then the accuracy and reliability of a grain pile surface pest detection result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a grain heap surface pest detection model training method, a detection method and equipment. BACKGROUND

[0002] The reproduction and invasion of stored grain pests is one of the key factors threatening the safe storage of grain. The insect damage caused by the reproduction and invasion of stored grain pests not only causes damage to the quantity and quality of grain, but also may cause food safety problems. Therefore, it is of great significance to realize real-time monitoring of pests. In view of the easy-to-observe characteristics of grain heap surface pests, detecting grain heap surface pests has become an extremely important supervision task.

[0003] At present, the existing grain heap surface pest detection method realizes automatic detection of grain heap surface pests by using a target detection model. However, the existing grain heap surface pest detection method does not consider the domain generalization problem of the model, which makes the model only effective in a specific scene. When facing a new detection scene, new data needs to be collected to train the data, and the original model needs to be fine-tuned. Therefore, multiple special models need to be maintained for different environments, which has a large deployment cost. SUMMARY

[0004] In view of this, the embodiments of the present application provide a grain heap surface pest detection model training method, a detection method and equipment to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present application provides a grain heap surface pest detection model training method, comprising:

[0006] In the current iteration round, different types of data samples respectively generated based on each grain heap surface image data labeled with a pest bounding box label in multiple data domains are respectively input into the first model branch and the second model branch, so that the first model branch and the second model branch respectively perform backbone network feature extraction and backdoor adjustment on different types of data samples to obtain target feature maps and corresponding pest detection result data; and a causal feature alignment total loss corresponding to the current iteration round is determined according to the target feature maps obtained by the first model branch and the second model branch respectively, so as to optimize the first model branch and the second model branch based on the causal feature alignment total loss.

[0007] If the current iteration round is the last iteration round, a grain heap surface pest detection model for outputting corresponding pest detection result data according to grain heap surface image data is generated according to the first model branch.

[0008] In some embodiments of the present application, each of the first model branch and the second model branch comprises, in sequence, a convolutional neural network, a causal intervention representation module, a contrastive causal feature alignment module, and a detection head; and weights are shared between the convolutional neural networks in the first model branch and the second model branch, and weights are shared between the detection heads in the first model branch and the second model branch.

[0009] The convolutional neural network is configured to perform feature extraction on an input data sample to output a corresponding original feature map.

[0010] The causal intervention representation module is configured to determine a mixed factor estimation value of a current iteration round based on the original feature map corresponding to the data sample, the pest bounding box label corresponding to the data sample, and a mixed factor estimation value obtained in a previous iteration round, and determine a target feature map after causal intervention representation corresponding to the original feature map based on the mixed factor estimation value of the current iteration round and the original feature map based on a cross-attention mechanism.

[0011] The contrastive causal feature alignment module is configured to transmit the target feature map after causal intervention representation to the detection head, and is further configured to obtain positive samples and difficult negative samples from the target feature map after causal intervention representation, to calculate a contrastive loss corresponding to each positive sample based on the positive samples and difficult negative samples obtained by each of the two contrastive causal feature alignment modules, and to determine a causal feature alignment total loss corresponding to the current iteration round based on the contrastive loss corresponding to each positive sample.

[0012] The detection head is configured to output a pest detection result data of the data sample based on the target feature map after causal intervention representation.

[0013] In some embodiments of the present application, the causal intervention representation module comprises a mixed factor estimation unit and a causal intervention cross-attention unit.

[0014] The mixed factor estimation unit is configured to obtain a feature vector group composed of feature vectors falling within the pest bounding box label from the original feature map corresponding to the data sample based on the pest bounding box label corresponding to the data sample, to obtain a pest target prediction box corresponding to the data sample based on each of the feature vectors as a positive sample corresponding to the data sample, to calculate an intersection over union value of the pest target prediction box corresponding to the data sample and the pest bounding box label, and to determine a weighted average value of each of the positive samples in the feature vector group as a weight, to update the weighted average value of the positive samples in an exponential moving average manner based on a mixed factor estimation value obtained in a previous iteration round to obtain a mixed factor estimation value of a current iteration round.

[0015] The causal intervention cross-attention unit is configured to sequentially perform channel number adjustment and dimension conversion on the original feature map to obtain a corresponding adjusted feature vector; perform cross-attention-based backdoor adjustment on the adjusted feature vector and the mixed factor estimation value of the current iteration round respectively to obtain a backdoor-adjusted feature vector; restore the dimension of the backdoor-adjusted feature vector to the same dimension as that of the original feature map to obtain a corresponding dimension-restored feature map; splice the dimension-restored feature map and the original feature map to obtain a corresponding spliced feature map; and reduce the dimension of the spliced feature map to the same dimension as that of the original feature map to obtain a corresponding target feature map represented after causal intervention.

[0016] In some embodiments of the present application, the contrast causal feature alignment module comprises a sample mining unit; and the contrast causal feature alignment module corresponding to each of the first model branch and the second model branch shares a same causal feature alignment unit;

[0017] The sample mining unit is configured to take other feature vectors in the target feature map represented after causal intervention except the positive samples as negative samples respectively; sort each negative sample in descending order of confidence, and select a plurality of negative samples in the sorted negative samples in front-to-back order, the number of which is the same as that of the positive samples, as difficult negative samples respectively;

[0018] The causal feature alignment unit is configured to obtain each positive sample and each difficult negative sample respectively acquired by the sample mining unit in the first model branch and the sample mining unit in the second model branch, calculate a contrast loss corresponding to each positive sample as a target of reducing feature distance between the positive samples and increasing feature distance between the positive samples and the difficult negative samples, and perform weighted average calculation on the contrast loss corresponding to each positive sample to obtain a causal feature alignment total loss corresponding to the current iteration round.

[0019] In some embodiments of the present application, before the different types of data samples respectively generated in advance based on the grain pile surface image data labeled with pest bounding boxes in a plurality of data domains are obtained, the grain pile surface pest detection model training method further comprises:

[0020] Obtain an image data group corresponding to each of a plurality of preset data domains, each of the image data groups containing a plurality of grain pile surface image data labeled with pest bounding boxes;

[0021] randomizing the frequency domain and the spatial domain of each of the grain pile surface image data respectively to obtain an enhanced image data corresponding to each of the grain pile surface image data;

[0022] generating different types of data samples according to each of the grain pile surface image data and the enhanced image data corresponding to each of the grain pile surface image data respectively.

[0023] In some embodiments of the present application, the generating different types of data samples according to each of the grain pile surface image data and the enhanced image data corresponding to each of the grain pile surface image data respectively includes:

[0024] merging each of the image data groups to obtain a first training data set, and taking each of the grain pile surface image data in the first training set as a type of data sample for training the first model branch;

[0025] and taking the enhanced image data corresponding to each of the grain pile surface image data as another type of data sample for training the second model branch.

[0026] In some embodiments of the present application, the generating different types of data samples according to each of the grain pile surface image data and the enhanced image data corresponding to each of the grain pile surface image data respectively includes:

[0027] dividing each of the image data groups into a second training data set and a third training data set, so that the second training data set and the third training data set each contain the grain pile surface image data of different data domains;

[0028] adding the enhanced image data corresponding to each of the grain pile surface image data in the second training data set to the second training data set, and taking each of the grain pile surface image data and the enhanced image data in the second training set as a type of data sample for training the first model branch;

[0029] and adding the enhanced image data corresponding to each of the grain pile surface image data in the third training data set to the third training data set, and taking each of the grain pile surface image data and the enhanced image data in the third training set as another type of data sample for training the second model branch.

[0030] In some embodiments of the present application, the generating a grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data according to the first model branch includes:

[0031] The model composed of the convolutional neural network in the first model branch and the detection head is taken as a grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data, wherein the convolutional neural network in the grain pile surface pest detection model is connected with the detection head.

[0032] The second aspect of the present application provides a grain pile surface pest detection method, comprising:

[0033] Collecting target grain pile surface image data in a grain storage space;

[0034] Inputting the target grain pile surface image data into a grain pile surface pest detection model to make the grain pile surface pest detection model output pest detection result data corresponding to the target grain pile surface image data; wherein the grain pile surface pest detection model is trained by the grain pile surface pest detection model training method of the first aspect.

[0035] The third aspect of the present application provides a grain pile surface pest detection model training device, comprising:

[0036] The iteration training module is configured to, in a current iteration round, input different types of data samples respectively generated based on each of the grain pile surface image data labeled with a pest bounding box label in multiple data domains into the first model branch and the second model branch respectively, so that the first model branch and the second model branch respectively perform backbone network feature extraction and backdoor adjustment on different types of the data samples to obtain target feature maps and corresponding pest detection result data; and determine a causal feature alignment total loss corresponding to the current iteration round according to the target feature maps obtained by the first model branch and the second model branch respectively, to optimize the first model branch and the second model branch based on the causal feature alignment total loss.

[0037] The application model output module is configured to, if the current iteration round is the last iteration round, generate a grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data according to the first model branch.

[0038] The fourth aspect of the present application provides a grain pile surface pest detection device, comprising:

[0039] The image data collection module is configured to collect target grain pile surface image data in a grain storage space.

[0040] The model detection module inputs the target grain pile surface image data into the grain pile surface pest detection model, so that the grain pile surface pest detection model outputs pest detection result data corresponding to the target grain pile surface image data; wherein the grain pile surface pest detection model is obtained by training using the grain pile surface pest detection model training method of the first aspect.

[0041] The fifth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the grain pile surface pest detection model training method, and / or implement the grain pile surface pest detection method.

[0042] The sixth aspect of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the grain pile surface pest detection model training method, and / or implement the grain pile surface pest detection method.

[0043] The seventh aspect of the present application provides a computer program product, which contains a computer program, wherein the computer program is executed by a processor to implement the grain pile surface pest detection model training method, and / or implement the grain pile surface pest detection method.

[0044] The grain pile surface pest detection model training method provided in the application, in the current iteration round, different types of data samples respectively generated based on each grain pile surface image data labeled with a pest bounding box label in multiple data domains are respectively input into a first model branch and a second model branch, so that the first model branch and the second model branch respectively perform backbone network feature extraction and backdoor adjustment on different types of data samples to obtain target feature maps and corresponding pest detection result data; and the causal feature alignment total loss corresponding to the current iteration round is determined according to the target feature maps obtained by the first model branch and the second model branch respectively, so as to optimize the first model branch and the second model branch based on the causal feature alignment total loss; if the current iteration round is the last iteration round, a grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data is generated according to the first model branch. By using each grain pile surface image data labeled with a pest bounding box label in multiple data domains as data samples of a training model, and using a double-branch domain generalization framework to respectively predict different types of data samples and perform backdoor adjustment on the features of the data samples during model training, the domain generalization ability of the grain pile surface pest detection model obtained by training can be effectively improved, and the adaptability of the grain pile surface pest detection model obtained by training to complex and variable actual detection scenarios can be improved. Without maintaining multiple special-purpose models for detection image data in different data domains, the application convenience and effectiveness of the grain pile surface pest detection model can be improved, thereby the accuracy and reliability of the grain pile surface pest detection result can be improved, and fast and accurate automatic detection result data for grain pest control is provided.

[0045] Additional advantages, objects, and features of the application will be set forth in part in the description which follows, and will in part be apparent to those skilled in the art upon examination of the following or can be learned from practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings.

[0046] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. The components in the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. For purposes of clarity and a consistent approach, portions of the drawings may be exaggerated relative to the other portions, so as to highlight some parts of the application, which can have been otherwise obscured or rendered difficult to discern. In the drawings:

[0048] Figure 1 A first flowchart of a grain surface pest detection model training method in an embodiment of the present application.

[0049] Figure 2 A first architecture diagram of the first model branch and the second model branch in an embodiment of the present application.

[0050] Figure 3 A schematic diagram of a grain surface pest detection model training structure based on a comparative causal intervention representation in an embodiment of the present application.

[0051] Figure 4 A second architecture diagram of the first model branch and the second model branch in an embodiment of the present application.

[0052] Figure 5 A flowchart of the process performed by the mixed factor estimation unit in an embodiment of the present application.

[0053] FIG. 6(a) is a data execution flowchart of the causal intervention representation module in an embodiment of the present application.

[0054] FIG. 6(b) is a data execution flowchart of the causal intervention cross-attention unit in an embodiment of the present application.

[0055] Figure 7 A third architecture diagram of the first model branch and the second model branch in an embodiment of the present application.

[0056] Figure 8 A data execution flowchart framework of the comparative causal feature alignment module in an embodiment of the present application.

[0057] Figure 9 A construction flowchart of the multi-domain grain surface pest image dataset in an embodiment of the present application.

[0058] Figure 10 A collection scene diagram of the pest image data in an embodiment of the present application.

[0059] Figure 11 A schematic diagram of the role principle of the random domain generation module in an embodiment of the present application.

[0060] Figure 12 FIG. 13(a) is a partial schematic diagram of model training in a single-domain mode according to an embodiment of the present application.

[0061] FIG. 13(a) is a partial schematic diagram of model training in a single-domain mode according to an embodiment of the present application.

[0062] FIG. 13(b) is a partial schematic diagram of model training in a multi-domain mode according to an embodiment of the present application.

[0063] Figure 14 FIG. 13(b) is a partial schematic diagram of model training in a multi-domain mode according to an embodiment of the present application.

[0064] Figure 15 FIG. 13(b) is a partial schematic diagram of model training in a multi-domain mode according to an embodiment of the present application.

[0065] Figure 16 FIG. 13(b) is a partial schematic diagram of model training in a multi-domain mode according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but not to limit the present application.

[0067] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0068] It should be emphasized that the term “comprises / comprising” as used herein indicates the presence of the stated features, elements, steps or components, but does not exclude one or more additional features, elements, steps or components.

[0069] It should be noted that, if not specifically stated, the term “connected” as used herein can not only mean direct connection, but also indirect connection with an intermediate object.

[0070] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0071] At present, many granaries are equipped with video monitoring devices such as network spherical cameras, mainly for local grain pile surface pest detection work. However, the traditional pest monitoring method for the surface of the grain pile mainly relies on manual observation of pests in the camera-captured image, which is time-consuming, labor-intensive, slow in detection speed and prone to errors.

[0072] In recent years, a large number of pest detection methods based on deep learning have been proposed, representing the development trend of automatic pest detection. For example, a granary pest detection system based on convolutional neural network, by adopting a lightweight VGG16 (Visual Geometry Group 16) model, a convolutional neural network, and introducing a weighted value for classification and regression tasks in the loss function to balance positive and negative samples, accurately identifies the area where pests exist, effectively shortening the training and detection time. Its prediction frame mechanism further improves the detection capability with the help of multi-scale feature extraction. However, this method is mainly suitable for pest detection in simple scenes such as laboratory glass culture dishes or white sticky insect plates, and when used to detect pests on the surface of grain piles, it is easy to misjudge the impurities, grain gaps and other similar pest targets on the surface of grain piles as pests.

[0073] For example, a pest detection method based on the YOLOv7 model in the target detection model, which integrates the ACmix adaptive convolution separation module and the CBAM (Convolutional Block Attention Module) convolution block attention module, ACmix is a new hybrid model that combines convolutional neural network CNN and self-attention mechanism, thereby improving the sensitivity to small-scale targets, reducing the influence of noise, and improving the detection accuracy in local occlusion, motion blur, and impurity adhesion. However, the training data used by this method are all from a single scene, without considering the situation that the actual detection data and the training data of the target model may not be consistent in the collection scene. This situation may cause data distribution differences (referred to as domain shift in the field of computer data), which may lead to a decline in detection performance.

[0074] For example, a deep learning-based rice pest detection and identification method and system, for pest image data from two different background sources under the pest forecasting lamp and in the field, respectively uses different models for analysis and identification. Although two backgrounds are distinguished, the data set is still limited to a specific scene and cannot cover more environmental changes, each model is trained only for a single background, and lacks cross-domain data or design for model generalization ability.

[0075] And a grain storage pest detection method based on a YOLOv5s algorithm in a target detection model, which adopts rotation, flipping, mirroring, scaling, translation and mosaic enhancement and other data enhancement methods to expand the data set. Moreover, the application also integrates the visual Transformer model (Swin Transformer) structure into the detection head to improve the feature extraction capability. However, this method is essentially only a modification of the existing general target detection framework, and does not consider the domain generalization problem that needs to be solved in actual pest detection.

[0076] That is, most of the current deep learning-based grain pile surface pest monitoring methods do not fully consider the data domain difference between the actual pest detection scene of the model and the scene collected by the training data. When facing a new detection scene, new data needs to be collected for training, and the original model needs to be fine-tuned, so multiple special models need to be maintained for different environments, which has a large deployment cost; and is usually limited to improvement of existing benchmark deep learning frameworks (such as YOLO, SSD, etc.), or only uses simple data enhancement strategies, and does not optimize model performance through domain generalization methods, resulting in limited adaptability to diversified detection scenes. That is, the existing grain pile surface pest detection method does not consider the domain generalization problem of the model, which makes the model only effective in a specific scene. When facing a new detection scene, new data needs to be collected for training, and the original model needs to be fine-tuned. Therefore, multiple special models need to be maintained for different environments, which has a large deployment cost.

[0077] Therefore, in order to solve the problem that the existing grain pile surface pest detection method does not consider the limited domain generalization ability of the model, the embodiments of the present application respectively provide a grain pile surface pest detection model training method, a grain pile surface pest detection method, a grain pile surface pest detection model training device for executing the grain pile surface pest detection model training method, a grain pile surface pest detection device for executing the grain pile surface pest detection method, an electronic device, a computer readable storage medium and a computer program product, which can effectively improve the domain generalization ability of the grain pile surface pest detection model and the adaptability to complex and variable actual detection scenes.

[0078] The embodiments are specifically described as follows.

[0079] Based on this, the embodiments of the present application provide a grain pile surface pest detection model training method which can be implemented by a grain pile surface pest detection model training device, see Figure 1 , the grain pile surface pest detection model training method specifically includes the following contents:

[0080] Step 100: in the current iteration round, different types of data samples respectively generated based on the grain heap surface image data labeled with pest bounding box labels in multiple data domains are respectively input into the first model branch and the second model branch, so that the first model branch and the second model branch respectively perform backbone network feature extraction and backdoor adjustment on different types of data samples to obtain target feature maps and corresponding pest detection result data; and a causal feature alignment total loss corresponding to the current iteration round is determined according to the target feature maps obtained by the first model branch and the second model branch respectively, so as to optimize the first model branch and the second model branch based on the causal feature alignment total loss.

[0081] In one or more embodiments of the present application, the data domain can be referred to as a domain, which refers to a set of pest image data with different characteristic differences. Such differences are caused by differences in data collection conditions (such as different lighting and different shooting devices). The set of image data collected under different conditions corresponds to different data domains. The multiple grain heap surface image data used for training the model are labeled with pest bounding box labels, which can be referred to as labels. The labels are used to identify the pest bounding box in the grain heap surface image data, i.e. the real box annotated by humans. The grain heap surface image data labeled with pest bounding box labels in multiple data domains means that multiple grain heap surface image data correspond to the first data domain, and multiple grain heap surface image data correspond to the second data domain, and so on, so as to consider the domain generalization ability of the model in the training data selection stage.

[0082] In step 100, the target feature map can include a first target feature map and a second target feature map; and the pest detection result data can include a first pest detection result data and a second pest detection result data. Correspondingly, the first model branch is used to sequentially perform backbone network feature extraction and backdoor adjustment on one type of data sample to obtain a first target feature map, and output the first pest detection result data corresponding to the first target feature map; the second model branch is used to sequentially perform backbone network feature extraction and backdoor adjustment on another type of data sample to obtain a second target feature map, and output the first pest detection result data corresponding to the first target feature map; and then a causal feature alignment total loss corresponding to the current iteration round is determined according to the first target feature map and the second target feature map, so as to optimize the first model branch and the second model branch based on the causal feature alignment total loss.

[0083] It can be understood that the backbone network corresponding to the backbone network feature extraction can adopt a convolutional neural network (CNN), and the backdoor adjustment is an important method for processing confounders in causal inference, which is used to estimate the true causal effect between variables in observational data. When there are confounders (variables that affect both cause variables and result variables), directly calculating the correlation of cause variables to result variables will lead to wrong conclusions due to confounding bias (such as Simpson's paradox). Backdoor adjustment separates the pure causal effect of cause variables on result variables by blocking the "backdoor path" (i.e. non-causal path composed of confounders) from cause variables to result variables, so as to eliminate the influence of confounders on the pest detection performance of the model, and further improve the generalization ability of the model.

[0084] It should be further pointed out that the causal feature alignment total loss is a composite loss function designed in a multi-modal or causal learning framework, which aims to improve the explainability and generalization ability of the model by jointly optimizing the alignment degree of the model decision mechanism and the real causal path and the feature semantic consistency.

[0085] That is, at least one iteration needs to be performed in the training process of the grain pile surface pest detection model, and step 100 is performed in each iteration round. Then it is judged whether the model composed of the first model branch (which can be referred to as the first branch) and the second model branch (which can be referred to as the second branch) converges, or whether the current iteration round is the last preset iteration round. If yes, step 200 is performed, and if no, the optimized first model branch and second model branch are returned to perform step 100 for model training in the next iteration round.

[0086] Step 200: If the current iteration round is the last iteration round, a grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data is generated according to the first model branch.

[0087] As can be known from the above description, the grain pile surface pest detection model training method provided by the embodiments of the present application can effectively improve the domain generalization capability of the grain pile surface pest detection model obtained by training, and can improve the adaptability of the grain pile surface pest detection model obtained by training to complex and variable actual detection scenes, without the need to maintain multiple special models for detection image data of different data domains, and can improve the application convenience and effectiveness of the grain pile surface pest detection model, thereby improving the accuracy and reliability of the grain pile surface pest detection result, and providing fast and accurate automatic detection result data for grain pest control.

[0088] In order to further improve the domain generalization capability of the grain pile surface pest detection model and improve the accuracy and reliability of the grain pile surface pest detection, in the grain pile surface pest detection model training method provided by the embodiments of the present application, referring to Figure 2 , the first model branch and the second model branch in the grain pile surface pest detection model training method specifically include: a convolutional neural network 1, a causal intervention representation module 2, a contrastive causal feature alignment module 3 and a detection head 4 connected in sequence; and the convolutional neural network 1 in the first model branch and the convolutional neural network 1 in the second model branch share weights, and the detection head 4 in the first model branch and the detection head 4 in the second model branch share weights.

[0089] The convolutional neural network 1 is used for feature extraction of the input data sample, to output the corresponding original feature map M.

[0090] The causal intervention representation module 2 is used for determining the mixed factor estimation value z t-1 of the current iteration round according to the original feature map M corresponding to the data sample, the pest bounding box label corresponding to the data sample and the mixed factor estimation value z t obtained in the last iteration round; and determining the target feature map M b after causal intervention representation corresponding to the original feature map according to the mixed factor estimation value z t of the current iteration round and the original feature map M based on the cross-attention mechanism. Wherein, the target feature map after causal intervention representation output by the causal intervention representation module 2 in the first model branch is the first target feature map; and the target feature map after causal intervention representation output by the causal intervention representation module 2 in the second model branch is the second target feature map.

[0091] It can be understood that the principle of the causal intervention representation module lies in using the "backdoor adjustment" operation of the causal inference theory to eliminate the false correlation between the input and output caused by confounding factors in the feature modeling process of the convolutional neural network (CNN). By blocking non-causal false connections, the model is prevented from relying on non-causal co-occurrence relationships in image data (such as background color and non-causal contextual features) for prediction. This improves the causal prediction ability of the model and enhances its robustness to domain shift phenomena. The specific implementation includes estimation of confounding factors and construction of backdoor adjustment based on causal intervention cross-attention.

[0092] The contrast causal feature alignment module 3 is configured to align the target feature map M b to the detection head 4, and is further configured to obtain positive samples and difficult negative samples from the causal intervention represented target feature map M b to calculate the contrast loss L corresponding to each positive sample xi based on the contrast loss L corresponding to each positive sample xi to determine the total causal feature alignment loss corresponding to the current iteration round

[0093] Specifically, the contrast causal feature alignment module 3 uses a contrast learning mechanism to achieve cross-domain causal feature alignment, aiming to further improve the model's ability to extract domain-invariant causal features. At the same time, by distinguishing targets and background noise that are similar in appearance to pests on the surface of the grain pile, the model's false detection phenomenon caused by such interference factors is reduced, thereby enhancing the accuracy and reliability of the model's detection. Its specific technical solution includes difficult negative sample mining and causal feature alignment.

[0094] The detection head 4 is configured to output pest detection result data of the data sample according to the causal intervention represented target feature map M b

[0095] Specifically, the grain pile surface stored grain pest detection domain generalization framework composed of the first model branch and the second model branch uses domain generalization technology based on contrast causal feature representation to train the pest detection model, which can alleviate the performance decline of the model in actual detection scenarios. The training architecture of the model uses a double-branch mode, and the specific structure is as follows Figure 3 ​As shown. During the training process, first, the data distribution diversity of the training data is enhanced through the random domain generation module. Then the original image data and the enhanced image data are input into the two branches respectively, and the basic representation and feature extraction operations are performed through the backbone network of the model (such as convolutional neural network). Then, the extracted original feature map M will be processed by the causal intervention representation module to block the false correlation introduced by the mixed factors. Finally, with the help of the contrast causal feature alignment module and combined with the difficult negative sample mining strategy, cross-domain feature alignment is realized. This process can further strengthen the ability of the model to extract domain-invariant causal features, while effectively reducing the interference of background noise on pest detection accuracy, and significantly improving the performance and reliability of the model in complex detection scenarios.

[0096] In order to further improve the application accuracy and reliability of causal intervention representation in the training process of the grain pile surface pest detection model, in the grain pile surface pest detection model training method provided in an embodiment of the present application, referring to Figure 4 , the causal intervention representation module 2 in the grain pile surface pest detection model training method specifically includes a mixed factor estimation unit 21 and a causal intervention cross-attention unit 22 connected to each other.

[0097] The mixed factor estimation unit 21 is configured to obtain a feature vector group composed of feature vectors falling within the pest bounding box label from the original feature map M corresponding to the data sample according to the pest bounding box label corresponding to the data sample; take each feature vector as a positive sample corresponding to the data sample to obtain a pest target prediction box corresponding to the data sample based on each positive sample; calculate the intersection over union value of the pest target prediction box corresponding to the data sample and the pest bounding box label, and take the intersection over union value as a weight to determine the weighted average value of each positive sample in the feature vector group; update the weighted average value of the positive sample in an exponential moving average manner based on the mixed factor estimation value obtained in the last iteration round to obtain the mixed factor estimation value in the current iteration round.

[0098] Specifically, referring to Figure 5 , the process executed by the mixed factor estimation unit includes:

[0099] During the model training process, the dimension of the data sample X input in each batch is Bx3xHxW, and after being processed by the backbone network, the original feature map M is output, and the dimension of the original feature map M is BxCxHxW. According to the pest bounding box label (i.e. real frame) labeled in the data set, each feature vector f1, f2,..., f N to form a feature vector group F = [f1, f2,..., f N] and each feature vector falling into the pest bounding box label is taken as a positive sample, which is responsible for the final pest target prediction. The specific prediction method is: the entire feature map containing positive samples is transmitted to the detection head through the causal intervention representation module and the contrast causal feature alignment module. The detection head outputs the prediction box coordinates and the confidence, wherein the prediction box coordinates are input into the confounding factor estimation unit to calculate the intersection over union (IoU) of the prediction box and the real label box. Wherein, B (Batch Size) represents the batch size, i.e. the number of data samples processed at a time; C (Channels) represents the number of channels, i.e. the number of color or feature channels of the data sample; H (Height) represents the height, i.e. the number of pixels in the vertical direction of the data sample; W (Width) represents the width, i.e. the number of pixels in the horizontal direction of the data sample.

[0100] Then, the intersection over union (IoU) value of the pest target prediction box predicted by the positive sample and the real box labeled in the data set is taken as the weight, and the weighted average value of each positive sample in the feature vector group is calculated as the estimated value z of the confounding factor. The specific calculation method is as follows:

[0101]

[0102] Wherein, f i represents the feature vector as a positive sample, N represents the total number of each positive sample in the feature vector group, and IoU i is the intersection over union (IoU) value between the pest target prediction box (i.e. the prediction box) of the i-th positive sample and the pest bounding box label (i.e. the real box).

[0103] Wherein, the static feature mean estimation (directly average the positive sample features) can also be used instead of the weighted average operation of the present application to achieve a certain causal intervention effect.

[0104] In the model training process, in order to ensure the stability and continuity of the confounding factor estimation, the exponential moving average (EMA) method is used to update the estimated value z of the confounding factor in each batch (i.e. the weighted average value of each positive sample in the feature vector group), as shown in Figure 6(a). The update formula is:

[0105]

[0106] Wherein, z t-1 is the confounding factor estimation value obtained in the last iteration round t-1, z t is the confounding factor estimation value in the current iteration round t; a represents the preset estimation value weight; f i t represents the feature vector of the positive sample in the current iteration round t; The intersection over union (IoU) value between the pest target prediction box (i.e., prediction box) representing the i-th positive sample in the current iteration round t and the pest bounding box label (i.e., real box).

[0107] The causal intervention cross-attention unit 22 is configured to sequentially perform channel number adjustment and dimension conversion on the original feature map M to obtain a corresponding adjusted feature vector, perform cross-attention-based backdoor adjustment on the adjusted feature vector and the mixed factor estimation value of the current iteration round respectively to obtain a backdoor-adjusted feature vector, restore the dimension of the backdoor-adjusted feature vector to the same as that of the original feature map M to obtain a corresponding dimension-restored feature map, splice the dimension-restored feature map and the original feature map M to obtain a corresponding spliced feature map, and reduce the dimension of the spliced feature map to the same as that of the original feature map M to obtain a corresponding target feature map represented after causal intervention.

[0108] Specifically, referring to FIG. 6(b), the process performed by the causal intervention cross-attention unit includes:

[0109] The original feature map M output by the backbone network has a dimension of BxCxHxW, and is first subjected to a 1x1 convolution operation for channel number adjustment. Subsequently, the feature map is converted into a feature vector (i.e., adjusted feature vector) with a dimension of BxHWxC through a reshape operation, where the reshape operation is a commonly used array or tensor shape transformation method in deep learning and scientific computing, and the core thereof is to adjust the dimension structure of data without changing the data content.

[0110] The adjusted feature vector is subjected to cross-attention-based backdoor adjustment operation with the estimated mixed factor z t The cross-attention-based backdoor adjustment operation is performed to realize causal intervention on the feature. The cross-attention formula is as follows:

[0111]

[0112] where f y (x,z t ) represents the backdoor adjustment operation, x represents the feature vector contained in the original feature map M, z t represents the mixed factor estimation value of the current iteration round t.

[0113] By introducing fully connected layers W q , W k and W v , linear transformation is performed on the query (Query), key (Key) and value (Value) respectively to obtain the backdoor-adjusted feature vector to enhance the expression ability of the model on the feature.

[0114] The feature vector adjusted through the backdoor is restored to the dimension BxCxHxW through a reshape operation again to obtain a dimension-restored feature map M'.

[0115] Subsequently, the dimension-restored feature map M' is spliced with the original feature map M to obtain a spliced feature map with a dimension of Bx2CxHxW.

[0116] Finally, the spliced feature map is processed through a 1x1 convolution to obtain a target feature map after causal intervention representation with a dimension of BxCxHxW.

[0117] Among them, by calculating the attention maps of the output features of the two branches, the effect of the present application can also be achieved by replacing the cross-attention mechanism method used in the present application in the form of reducing the difference between the attention maps.

[0118] In order to further improve the domain generalization ability of the grain pile surface pest detection model and improve the accuracy and reliability of the grain pile surface pest detection, in the grain pile surface pest detection model training method provided in the embodiment of the present application, referring to Figure 7 , the comparative causal feature alignment module 3 in the grain pile surface pest detection model training method specifically includes a sample mining unit 31; and the comparative causal feature alignment module 3 corresponding to the first model branch and the second model branch respectively shares the same causal feature alignment unit 32.

[0119] The sample mining unit 31 is configured to take other feature vectors in the target feature map after causal intervention representation except the positive samples as negative samples respectively; sort each negative sample in order from high to low confidence, and select a plurality of negative samples with the same number of positive samples in the sorted negative samples in order from front to back as difficult negative samples.

[0120] Among them, in the sample mining unit 31, the positive samples in the target feature map are the positive samples in the mixed factor estimation unit 21, that is, each feature vector falling within the pest bounding box label. In the sample mining unit 31, the positive samples determined in the mixed factor estimation unit 21 can be directly used, or each feature vector falling within the pest bounding box label can be determined as the positive sample in the same way as the mixed factor estimation unit 21.

[0121] Specifically, the data execution flow framework of the comparative causal feature alignment module is as shown in Figure 8The difficult negative sample mining is to improve the distinguishing ability of the model to the confusing features. The specific steps are as follows: first, the feature vector responsible for the final prediction of the pest target in the original feature map M output by the backbone network is defined as a positive sample, and the remaining feature vectors are defined as negative samples. Then, the confidence mean p of the prediction results (including the confidence and the bounding box coordinates) of all positive samples in the current batch is calculated. Assuming that the number of positive samples in the current batch is k, the first k negative samples with a confidence higher than p are selected from all negative samples as difficult negative samples. If the number of negative samples with a confidence higher than p is less than k, the first k negative samples with the highest confidence are selected as difficult negative samples. These difficult negative samples usually correspond to targets similar to the pest morphology or background noise in the image data, and are easily misdetected as pests due to their high similarity to the pest target and high confidence.

[0122] In addition, the feature distance between the negative sample feature vector and the positive sample feature (pest feature) vector is compared, and the negative samples with a feature distance close to the positive sample feature are selected as difficult negative samples, instead of the confidence-based method for screening negative samples.

[0123] Specifically, the feature alignment operation is performed on the positive samples and the difficult negative samples. The method is to reduce the feature distance between the positive samples and increase the feature distance between the positive samples and the difficult negative samples by optimizing algorithm under the contrast learning framework. For a single positive sample pair, the contrast loss L xi The calculation formula is as follows:

[0124]

[0125] Where x i and x j represent the i-th and j-th positive sample pair, represents a difficult negative sample, and τ is a temperature parameter. N pos and N neg represent the number of positive samples and difficult negative samples, respectively, L xi is the contrast loss of a single positive sample, and exp() is the natural exponential function.

[0126] The final causal feature alignment total loss is the weighted average of all positive sample contrast losses L xi , and the calculation method is as follows:

[0127]

[0128] Where IoU xiThe IoU value between the prediction box obtained by the positive sample and the real box. The reason for this design is that different positive samples have significant differences in the quality of representing the pest target. Specifically, the positive sample with a higher IoU value with the real box can more accurately represent the characteristics of the pest target. Therefore, using IoU as a weight factor can enhance the model's attention to high-quality positive samples.

[0129] In order to further improve the effectiveness and reliability of the training data basis of the grain surface pest detection model, and further improve the domain generalization ability of the grain surface pest detection model and improve the accuracy and reliability of the grain surface pest detection, in the grain surface pest detection model training method provided in the embodiment of the present application, before step 100 in the grain surface pest detection model training method, the following content is further included:

[0130] Step 010: Obtain each preset data domain corresponding image data group, each of the image data group contains a plurality of grain surface image data marked with pest bounding box label.

[0131] Step 020: Randomize the frequency domain and spatial domain of each of the grain surface image data to obtain each of the grain surface image data corresponding enhanced image data.

[0132] Step 030: Generate different types of data samples according to each of the grain surface image data and each of the grain surface image data corresponding enhanced image data.

[0133] Specifically, in one specific implementation example of step 010, referring to Figure 9 The construction process of the multi-domain grain surface pest image data set containing each preset data domain corresponding image data group is as follows:

[0134] In the actual grain storage environment, configure multiple image data acquisition conditions and implement artificial pest release; based on multiple image data acquisition conditions, collect multi-view image data of artificially released grain pests; perform data labeling processing on the collected pest image data, and implement multi-domain division based on the image data acquisition conditions. In addition, in order to further expand the data set, the present application can also simulate the actual grain storage environment under laboratory conditions, and collect pest image data in the same way as in the actual grain storage environment, thereby enhancing the flexibility of image data acquisition and the richness of the data set.

[0135] (1) The pest image data collection scene is shown in Figure 10 The specific implementation method is as follows:

[0136] The network spherical camera (ball machine) commonly used in the grain store as image data acquisition equipment. The ball machine is installed on the wall of the grain store through a fixed support, and the center of its lens is 1.2-1.8 meters away from the vertical height of the grain pile surface to achieve the best shooting angle and imaging resolution. The grain store can be a simulated indoor area using a simulated real grain store or directly using a real grain store, and in order to avoid wasting grain, the grain store is filled with grain that has exceeded the safe food consumption period and is no longer suitable for human consumption, and / or after obtaining the grain pile surface image data (which can be referred to as image data) for training, the grain store is quickly treated to remove pests without affecting the quality and safety of the grain.

[0137] Secondly, the image data acquisition area is determined, taking the grain pile surface directly below the center of the lens of the ball machine as the center point, and a semicircular image data acquisition area with a radius of 3-5 meters is marked along the grain pile surface. Within the semicircular image data acquisition area, 100-150 independent image data acquisition points are selected according to the equal interval distribution principle, and the area of each acquisition point is 500-600 square centimeters.

[0138] Then, at the acquisition points, place trays, grain membranes and other containers, and evenly lay the same type of grain on the containers as the grain pile surface to simulate the real grain pile surface and achieve pest isolation and control.

[0139] Subsequently, live body storage pests are evenly placed on the simulated grain pile surface to make the distribution and behavior of the pests conform to the natural invasion characteristics.

[0140] Finally, the ball machine is remotely controlled by the ball machine control software, and all image data acquisition points are sequentially photographed in the predetermined order, thereby obtaining the image data of the pests. The image data acquisition process in the laboratory environment is consistent with the operation in the actual grain store, covering the determination of the image data acquisition area, the selection of the image data acquisition points, the laying of the grain, and the placement of the pests, etc.

[0141] (2) The image data shooting conditions include the following settings:

[0142] ① Use ball machines of different models and service life to reflect the influence of ball machine resolution, lens aging degree, sensor performance attenuation and other factors on image data quality.

[0143] ②Lay different types of grain with different storage times at the image data shooting points, including new grain and old grain with shorter storage times. To reflect the influence of grain color change, blackening, mold and other reasons on the background information and context features of the pests target surrounding during the grain storage process.

[0144] ③ Considering the significant influence of light conditions on the visual features of image data, diversified light conditions are adopted for shooting. By simulating different light intensities, white balance settings and other conditions, the differences in visual features such as brightness, contrast, color saturation of pest image data under light changes are simulated.

[0145] (3) Data labeling and division: use image data labeling tools such as LabelImg to label pest image data. LabelImg is an open-source image data labeling tool that identifies and annotates the location coordinates of each pest target in the image data. Then, according to the set shooting conditions, the collected image data set is divided into data domains.

[0146] Specifically, in one specific implementation example of step 020, referring to Figure 11 , step 020 can be generated using a random domain generation module. The principle of the random domain generation module is to perform randomization operations on the frequency domain and spatial domain of the input image data to expand the coverage of the training data distribution. The core idea is to perturb the domain change features in the image data that are irrelevant to the pest features, such as background color, noise, etc.; while keeping the domain invariant features unchanged, i.e. the morphology of the pest target is unchanged, thereby achieving the enhancement of the diversity of the training data domain. The technical advantage of the method lies in: under the premise of maintaining the semantic integrity of the input image data, controllable perturbation is performed on the domain change features, generating new samples with domain differences, thereby improving the sample diversity of model training.

[0147] The specific implementation of step 200 performed by the random domain generation module is as follows:

[0148] (1) Frequency domain randomization step: First, use Discrete Cosine Transform (DCT) to convert the image data from spatial domain to frequency domain, and obtain the corresponding frequency domain spectrum. Then, the frequency domain spectrum is decomposed into low frequency components, medium frequency components and high frequency components by a pre-set band-pass filter set, where the low frequency components and high frequency components correspond to domain change features, and the medium frequency components correspond to domain invariant features. Then, random Gaussian noise is introduced to the low frequency and high frequency components. Finally, the frequency domain spectrum after noise injection is reconstructed by Inverse Discrete Cosine Transform (IDCT) to generate enhanced image data The process is described in detail as follows:

[0149]

[0150] where X represents the input data sample, DCT(·) and DCT'(·) represent Discrete Cosine Transform and Inverse Discrete Cosine Transform respectively. M(r) is a band-pass filter, and the passband is r(R L ,R H ), where R L and RH is the percentage range of the passband low cutoff frequency and high cutoff frequency. The function R G () is used to introduce Gaussian noise, described as represents a standard Gaussian distribution with mean 0 and variance 1.

[0151] In addition, the discrete cosine transform in the frequency domain randomization operation can be replaced by other methods such as discrete Fourier transform, discrete wavelet transform, etc.

[0152] (2) Spatial domain randomization step: the randomization of the spatial domain adopts a color dithering strategy, which randomly rearranges the RGB three color channels of the image data according to a predetermined probability. For example, the RGB three color channels are randomly arranged as GBR, BRG, etc. The non-destructive disturbance of the background color of the image data is realized through channel order reconstruction. The method is characterized in that: under the premise of maintaining the semantic integrity of the pest target, the domain diversity of the training data is enhanced through controllable color channel replacement. In addition, the operation of randomly exchanging the RGB three color channels can be modified to other color dithering methods, such as randomly adjusting the brightness and contrast of the image data.

[0153] In order to further improve the application convenience of the grain pile surface pest detection model and reduce the device resource occupancy rate, in the grain pile surface pest detection model training method provided in the embodiment of the present application, step 200 in the grain pile surface pest detection model training method further specifically contains the following content:

[0154] Step 210: If the current iteration round is the last iteration round, the model composed of the convolutional neural network in the first model branch and the detection head is used as a grain pile surface pest detection model for outputting corresponding pest detection result data according to the grain pile surface image data, wherein the convolutional neural network in the grain pile surface pest detection model is connected with the detection head.

[0155] In order to further enrich the distribution combination of the training data and enhance the adaptability of the model to different detection scenes, in the grain pile surface pest detection model training method provided in the embodiment of the present application, referring to Figure 12 , step 030 in the grain pile surface pest detection model training method specifically contains the following content:

[0156] Step 031: merging each of the image data groups to obtain a first training data set, and taking each of the grain pile surface image data in the first training set as a type of data sample for training the first model branch.

[0157] and, step 032: the enhanced image data corresponding to each of the grain pile surface image data is respectively added into the second training data set, and each of the grain pile surface image data and the enhanced image data in the second training data set is respectively used as a type of data sample for training the first model branch.

[0158] Referring to FIG. 13(a), step 100 can be further performed in a single-domain mode: all training data is merged and regarded as a single independent and identically distributed data domain. During training, the two model branches are respectively input with original image data and image data enhanced by a random domain generation (RDG) module, so as to improve the diversity of data.

[0159] In order to further enrich the distribution combination of training data and enhance the adaptability of the model to different detection scenes, in an embodiment of the grain pile surface pest detection model training method provided in the present application, referring to FIG. 13(b), step 030 can further include the following content: Figure 14

[0160] Step 033: each of the image data groups is divided into a second training data set and a third training data set, so that the second training data set and the third training data set each contain the grain pile surface image data of different data domains.

[0161] Step 034: the enhanced image data corresponding to each of the grain pile surface image data in the second training data set is also added into the second training data set, and each of the grain pile surface image data and the enhanced image data in the second training set is respectively used as a type of data sample for training the first model branch.

[0162] and, step 035: the enhanced image data corresponding to each of the grain pile surface image data in the third training data set is also added into the third training data set, and each of the grain pile surface image data and the enhanced image data in the third training set is respectively used as another type of data sample for training the second model branch.

[0163] Referring to FIG. 13(b), step 100 can be further performed in a multi-domain mode: the two branches randomly receive data of different domains and are both enhanced by a random domain generation module, so as to expand the domain coverage of data and improve the generalization ability of the model. This mode helps the model to adapt to multiple data domains and enhances its applicability in different environments, wherein H1, H2, G1 and G2, etc. are used to represent different data domains.

[0164] In summary, the grain pile surface pest detection model training method provided in the embodiments of the present application is used to solve the following technical problems existing in the prior art:

[0165] ​1) Existing deep learning-based pest detection methods mostly do not fully consider the difference between the actual pest detection scene of the model and the scene of the collected training data, and their generalization ability is weak. After the pest detection model is trained and deployed, its detection performance is significantly reduced compared to the training data.

[0166] 2) The training data used by existing deep learning-based pest detection methods mostly comes from limited pest detection scenarios, and is not reasonably divided according to the image data collection scenario, and the training data is independent and identically distributed. When the pest detection model is deployed to a new detection environment, the model's detection accuracy will significantly decrease due to the lack of similar data distribution. This makes the model only effective in specific scenarios, limiting the practical application of deep learning methods.

[0167] 3) Existing pest detection methods often need to collect new data for training data when facing new detection scenarios, and then fine-tune the original model. Therefore, multiple specialized models need to be maintained for different environments, which has a high deployment cost.

[0168] 4) Existing pest detection methods are usually limited to improvements on existing benchmark deep learning frameworks (such as YOLO, SSD, etc.), or only use simple data augmentation strategies, and do not optimize model performance through domain generalization methods, resulting in limited adaptability to diverse detection scenarios.

[0169] 5) Traditional deep learning-based pest detection methods are easily affected by mixed factors. For example, a certain type of pest often appears in image data with a yellow background (background information) or on a corn surface (contextual features). The model may associate these mixed factors with pest features and form false correlations. Once the background information or contextual features of the image data change, the detection performance of the model will significantly decrease.

[0170] 6) Existing methods do not fully consider how to suppress the impact of background noise on pest detection accuracy. For example, impurities on the surface of the grain pile, grain gaps, and other features similar to the shape of the pest target, the model is prone to false detection.

[0171] The present application belongs to the technical field of grain storage pest monitoring, and specifically relates to a domain generalization detection algorithm framework based on comparative causal intervention representation, especially suitable for visual detection of pests on the surface of grain piles during grain storage. The present application mainly aims to solve the problem of decreased pest detection performance caused by domain shift when a deep learning detection model is trained and deployed in an actual storage environment. By constructing an environment-robust domain generalization model, the present application solves the problem of insufficient model generalization caused by domain shift factors such as environmental differences and device parameter changes in the process of existing visual detection technology in grain storage, thereby affecting the accuracy of detection and solving the industry technical bottleneck.

[0172] That is, the grain pile surface pest detection model training method provided by the embodiments of the present application aims to solve the problem of detection performance decline caused by the inconsistency between the actual detection data and the training data distribution of the model, and improve the adaptability of the model to complex detection scenarios. The main content includes: ① Construct a multi-domain grain pile surface pest image dataset. ② Design a grain pile surface stored grain pest detection domain generalization framework, which includes: a random domain generation module, a causal intervention representation module, and a comparative causal feature alignment module. The model framework specifically includes the following content:

[0173] 1) Construct a multi-domain pest image dataset: collect pest image data from multiple stored grain scenes to simulate the domain shift phenomenon that may occur in actual pest detection scenarios, and provide data basis for model training and verification.

[0174] 2) Design a random domain generation strategy: enrich the diversity of training data during model training to enable the model to learn different domain features and improve its generalization ability.

[0175] 3) Design a causal intervention representation module: block the interference of mixed factors (such as image data background and context features) from the feature level to ensure that the model only relies on causal features (such as morphology and texture) of pests for detection, and improve the robustness of the model.

[0176] 4) Design a causal feature alignment module combined with difficult negative sample mining: improve the ability of the model to extract domain-invariant causal features while reducing the impact of background noise on pest detection accuracy, further improving the detection performance and reliability of the model.

[0177] Based on this, the grain pile surface pest detection model training method provided by the embodiments of the present application, through the construction method of the multi-domain stored grain pest image dataset; the double-branch design idea and module combination method in the stored grain pest detection domain generalization framework; the estimation method of mixed factors in the causal intervention representation module; the implementation method of backdoor adjustment based on causal intervention cross-attention in the causal intervention representation module; the mining mechanism of difficult negative samples in the comparative causal feature alignment module; the IoU weighting strategy and its comparative learning process in the comparative causal feature alignment module; the image data enhancement method combining frequency domain disturbance and spatial disturbance in the random domain generation module; the model training strategy, the training method of single-domain and multi-domain mode, so that the method has the following beneficial effects:

[0178] 1. Application of domain generalization technology:

[0179] The existing pest detection method will significantly decrease in performance when facing data distribution that the model has not seen during training. The application designs three domain generalization modules: causal intervention representation module, contrastive causal feature alignment module and random domain generation module, and integrates them into the pest detection framework. This innovative technical solution can effectively alleviate the performance decline of the pest detection model caused by data distribution differences in actual application, thereby significantly improving the model's adaptability to complex and variable actual detection scenarios and expanding the model's application range and practicality.

[0180] 2. Establishment method of multi-domain pest image dataset:

[0181] The application proposes a method for constructing a multi-domain pest image dataset. Unlike traditional pest detection methods, the application selects a strategy for shooting pest image data under actual warehouse and laboratory conditions, making full use of the diversity and complexity of the real warehouse environment. At the same time, the application designs a method for detailed domain division of collected image data by controlling shooting conditions, including but not limited to light intensity, background type, grain type, etc. This method can generate high-quality training data covering various actual grain storage scenarios, effectively simulating various complex situations faced by pest detection in the real world.

[0182] 3. Design of double-branch domain generalization framework:

[0183] Existing pest detection methods are mostly limited to simple improvements on existing benchmark deep learning frameworks, without considering domain generalization in actual detection. The application focuses on the model's adaptability to complex detection scenarios. By designing a double-branch domain generalization framework, the causal intervention representation, contrastive causal feature alignment and random domain generation modules are organically integrated. Based on this double-branch structure, the application proposes training strategies for single-domain and multi-domain modes.

[0184] 4. Random domain generation module for data enhancement to improve domain richness:

[0185] The random domain generation module designed by the application achieves effective enhancement of training data through randomization operations in the frequency domain and spatial domain. Unlike traditional data enhancement methods such as translation, flipping and cropping, these methods can only change the appearance of image data within a limited range, cannot effectively improve the domain richness of data, and may damage the original semantic information of image data. The random domain generation module of the application can perturb the domain-related features of image data without damaging the semantic information of the image data, thereby generating more diverse training data. This enhancement method not only effectively expands the domain coverage of training data, but also enables the model to encounter a wider range of feature variations during training, thereby significantly improving the model's adaptability and robustness to different detection scenarios.

[0186] 5. Introducing a causal intervention representation based on backdoor adjustment:

[0187] The present application introduces a causal intervention representation mechanism based on "backdoor adjustment" to eliminate the influence of confounding factors on pest detection performance, thereby improving the generalization ability of the model. Existing pest detection methods do not fully consider the influence of confounding factors (such as image data background color, background context information, etc.) on model performance during the training process, resulting in the model tending to rely on these false correlations for prediction. This makes the model perform significantly worse when facing new, unseen detection scenarios. To solve this problem, the present application uses a causal intervention cross-attention mechanism to block the interference of confounding factors on model prediction through a designed causal intervention representation module. This module enables the model to pay more attention to the causal features of the pest itself, such as pest morphology, thereby maintaining stable detection performance in new detection scenarios and improving the generalization ability of the model.

[0188] 6. Introducing a causal feature alignment strategy combined with difficult negative sample mining:

[0189] Existing pest detection methods are prone to false positives when dealing with similar non-pest targets or background noise on the surface of the grain pile (such as impurities on the surface of the grain pile, grain gaps, etc.). To address this problem, the present application automatically identifies those targets that are prone to false positives as negative samples during model training through difficult negative sample mining technology. Under the contrast learning framework, the model not only narrows the feature distance between positive samples, but also widens the feature distance between positive and negative samples. This strategy not only improves the model's ability to extract domain-invariant features, but also effectively alleviates the false positive phenomenon, enabling the model to more accurately identify pest targets in complex detection scenarios.

[0190] 7. Training strategy for single-domain and multi-domain modes:

[0191] Existing pest detection methods usually mix all training data together, assuming that the data is independently and identically distributed. This approach limits the way data distributions are combined, which can result in the model not adapting well to changes in data distribution in actual applications. In contrast, the training strategy of the present application can flexibly choose single-domain or multi-domain training modes depending on whether the training data has domain division characteristics. This flexibility not only enriches the distribution combination of training data, but also enhances the model's ability to adapt to different detection scenarios, enabling the model to better cope with diverse and complex data distribution situations in actual applications.

[0192] Based on the embodiments of the grain surface pest detection model training method provided in the preceding embodiments, the present application also provides an embodiment of a grain surface pest detection method, which specifically includes the following content:

[0193] Step 300: collect target grain pile surface image data in a grain storage space;

[0194] Step 400: input the target grain pile surface image data into the grain pile surface pest detection model, so that the grain pile surface pest detection model outputs the pest detection result data corresponding to the target grain pile surface image data; wherein the grain pile surface pest detection model is obtained by pre-training using the grain pile surface pest detection model training method.

[0195] In the embodiments of the grain pile surface pest detection method provided in the present application, the grain pile surface pest detection model training method can specifically adopt the processing flow of the embodiments of the grain pile surface pest detection model training method described above, and the functions thereof will not be repeated here. Please refer to the detailed description of the above-mentioned embodiments of the grain pile surface pest detection model training method.

[0196] From the software aspect, the present application also provides a grain pile surface pest detection model training device for executing all or part of the grain pile surface pest detection model training method, which is described in detail with reference to Figure 15 , and the grain pile surface pest detection model training device specifically includes the following contents:

[0197] The iterative training module 10 is configured to, in the current iteration round, input different types of data samples respectively generated based on the grain pile surface image data labeled with the pest bounding box label in multiple data domains into the first model branch and the second model branch, respectively, so that the first model branch and the second model branch respectively perform backbone network feature extraction and backdoor adjustment on different types of data samples to obtain target feature maps and corresponding pest detection result data; and determine a causal feature alignment total loss corresponding to the current iteration round according to the target feature maps obtained by the first model branch and the second model branch, respectively, to optimize the first model branch and the second model branch based on the causal feature alignment total loss.

[0198] The application model output module 20 is configured to, if the current iteration round is the last iteration round, generate a grain pile surface pest detection model for outputting corresponding pest detection result data according to the grain pile surface image data based on the first model branch.

[0199] The embodiments of the grain pile surface pest detection model training device provided in the present application can specifically be used to execute the processing flow of the embodiments of the grain pile surface pest detection model training method described above, and the functions thereof will not be repeated here. Please refer to the detailed description of the above-mentioned embodiments of the grain pile surface pest detection model training method.

[0200] The part of the grain bulk surface pest detection model training apparatus for performing grain bulk surface pest detection model training can be completed in a server or a client device. Specifically, the selection can be made according to the processing capability of the client device and the limitation of the user's use scenario, etc. The present application does not limit this. If all operations are completed in the client device, the client device can further include a processor for specific processing of grain bulk surface pest detection model training.

[0201] The above-mentioned client device can have a communication module (i.e., a communication unit) and can be communicatively connected with a remote server to realize data transmission with the server. The server can include a server of a task scheduling center side, and in other implementation scenarios, it can also include a server of an intermediate platform, such as a server of a third-party server platform communicatively connected with the server of the task scheduling center. The server can include a single computer device, or a server cluster composed of multiple servers, or a server structure of a distributed device.

[0202] The server and the client device can use any suitable network protocol for communication, including a network protocol that has not been developed at the filing date of the present application. The network protocol can include, for example, a TCP / IP protocol, a UDP / IP protocol, an HTTP protocol, an HTTPS protocol, etc. Of course, the network protocol can also include, for example, a RPC protocol (Remote Procedure Call Protocol) and a REST protocol (Representational State Transfer) used on the above-mentioned protocols.

[0203] As can be known from the above description, the grain bulk surface pest detection model training apparatus provided by the embodiments of the present application can effectively improve the domain generalization capability of the grain bulk surface pest detection model obtained by training, and can improve the adaptability of the grain bulk surface pest detection model obtained by training to complex and changeable actual detection scenarios, without the need to maintain multiple special models for detection image data of different data domains, and can improve the application convenience and effectiveness of the grain bulk surface pest detection model, thereby improving the accuracy and reliability of the grain bulk surface pest detection result, and providing a fast and accurate automatic detection result data for grain pest control.

[0204] From the software level, the application further provides a grain heap surface pest detection device for performing all or part of the grain heap surface pest detection method, referring to Figure 16 , the grain heap surface pest detection device specifically includes the following contents:

[0205] An image acquisition module 30 is configured to acquire target grain heap surface image data in a grain storage space.

[0206] A model detection module 40 is configured to input the target grain heap surface image data into a grain heap surface pest detection model, so that the grain heap surface pest detection model outputs pest detection result data corresponding to the target grain heap surface image data; wherein the grain heap surface pest detection model is obtained by training using the grain heap surface pest detection model training method of the first aspect.

[0207] The embodiments of the grain heap surface pest detection device provided by the application can be specifically used to execute the processing procedures of the embodiments of the grain heap surface pest detection method described above, and the functions thereof will not be repeated here, and the detailed description can be referred to the above detailed description of the grain heap surface pest detection method embodiments.

[0208] The grain heap surface pest detection model training method in the embodiments of the grain heap surface pest detection device provided by the application can specifically use the processing procedures of the embodiments of the grain heap surface pest detection model training method described above, and the functions thereof will not be repeated here, and the detailed description can be referred to the above detailed description of the grain heap surface pest detection model training method embodiments.

[0209] The part of the grain heap surface pest detection model training device for performing grain heap surface pest detection can be completed in a server or a client device. Specifically, it can be selected according to the processing capacity of the client device and the limitation of the user's use scenario. The application does not limit this. If all operations are completed in the client device, the client device can further include a processor for specific processing of grain heap surface pest detection.

[0210] The application further provides an electronic device, which can include a processor, a memory, a receiver and a transmitter, the processor being configured to execute the grain heap surface pest detection model training method and / or the grain heap surface pest detection method mentioned in the above embodiments, wherein the processor and the memory can be connected through a bus or other means, taking the bus connection as an example. The receiver can be connected with the processor and the memory through wired or wireless means.

[0211] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, or a combination thereof.

[0212] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules of the grain pile surface pest detection model training method and / or the grain pile surface pest detection method in the embodiments of the present application. The processor executes various functions and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implements the grain pile surface pest detection model training method and / or the grain pile surface pest detection method in the above method embodiments.

[0213] The memory can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created by the processor and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0214] The one or more modules are stored in the memory, and when executed by the processor, implement the grain pile surface pest detection model training method and / or the grain pile surface pest detection method in the embodiments.

[0215] In some embodiments of the present application, the user equipment can include a processor, a memory and a transceiver unit which can include a receiver and a transmitter, the processor, the memory, the receiver and the transmitter can be connected through a bus system, the memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to transceive signals.

[0216] As an implementation manner, the functions of the receiver and the transmitter in the present application can be implemented by a transceiver circuit or a transceiver dedicated chip, and the processor can be implemented by a dedicated processing chip, a processing circuit or a general-purpose chip.

[0217] As another implementation manner, the server provided by the embodiments of the present application can be implemented by using a general-purpose computer. That is, the program codes for implementing the functions of the processor, the receiver and the transmitter are stored in the memory, and the general-purpose processor implements the functions of the processor, the receiver and the transmitter by executing the codes in the memory.

[0218] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the aforementioned grain pile surface pest detection model training method and / or the grain pile surface pest detection method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0219] The embodiments of the present application further provide a computer program product, which includes a computer program. The computer program is executed by a processor to implement the steps of the aforementioned grain pile surface pest detection model training method and / or the grain pile surface pest detection method.

[0220] Those skilled in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. The actual implementation depends on the specific application and design constraints imposed on the overall system. Skilled persons can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave in a transmission medium or communication link.

[0221] It is to be expressly understood that the application is not limited to the described and illustrated particular configurations and processes. For the sake of clarity, detailed descriptions of known methods are omitted. In the above described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the application are not limited to the specific steps described and illustrated, and the skilled person can make various changes, modifications and additions, or change the order of the steps, after having understood the spirit of the application.

[0222] In this application, features described and / or illustrated with respect to one embodiment can be used in the same or similar manner in one or more other embodiments and / or combined with or substituted for features of other embodiments.

[0223] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a pest detection model on the surface of a grain pile, characterized in that, The method comprises the following steps: In the current iteration round, different types of data samples generated respectively based on the grain pile surface image data of each pest bounding box label in multiple data domains are input into the first model branch and the second model branch respectively, so that the first model branch and the second model branch respectively perform backbone network feature extraction and backdoor adjustment on different types of data samples to obtain target feature maps and corresponding pest detection result data; And the causal feature alignment total loss corresponding to the current iteration round is determined according to the target feature maps obtained by the first model branch and the second model branch respectively, so as to optimize the first model branch and the second model branch based on the causal feature alignment total loss. If the current iteration round is the last iteration round, the grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data is generated according to the first model branch.

2. The grain bulk surface pest detection model training method according to claim 1, characterized in that, The first model branch and the second model branch each comprise a convolutional neural network, a causal intervention representation module, a contrastive causal feature alignment module and a detection head connected in sequence; and the convolutional neural networks in the first model branch and the second model branch share weights, and the detection heads in the first model branch and the second model branch share weights. The convolutional neural network is used for feature extraction of the input data sample to output a corresponding original feature map. The causal intervention representation module is used for determining a mixed factor estimation value of the current iteration round according to the original feature map corresponding to the data sample, the pest bounding box label corresponding to the data sample and a mixed factor estimation value obtained in the last iteration round; and determining a target feature map after causal intervention representation of the original feature map based on a cross-attention mechanism according to the mixed factor estimation value of the current iteration round and the original feature map. The contrastive causal feature alignment module is used for transmitting the target feature map after causal intervention representation to the detection head, and is also used for obtaining positive samples and difficult negative samples from the target feature map after causal intervention representation, so as to calculate a contrastive loss corresponding to each positive sample according to the positive samples and difficult negative samples obtained by two contrastive causal feature alignment modules, and determine a causal feature alignment total loss corresponding to the current iteration round based on the contrastive loss corresponding to each positive sample. The detection head is used for outputting pest detection result data of the data sample according to the target feature map after causal intervention representation.

3. The grain bulk surface pest detection model training method according to claim 2, characterized in that, The causal intervention representation module comprises a mixed factor estimation unit and a causal intervention cross-attention unit. The hybrid factor estimation unit is configured to: obtain, according to the pest bounding box label corresponding to the data sample, a feature vector group composed of feature vectors falling within the pest bounding box label from the original feature map corresponding to the data sample; take each feature vector as a positive sample corresponding to the data sample respectively to obtain a pest target prediction box corresponding to the data sample based on each positive sample; calculate an intersection over union value of the pest target prediction box corresponding to the data sample and the pest bounding box label, and take the intersection over union value as a weight to determine a weighted average value of each positive sample in the feature vector group; update the weighted average value of the positive sample in an exponential moving average manner based on a hybrid factor estimation value obtained in a previous iteration round to obtain a hybrid factor estimation value in a current iteration round; The causal intervention cross-attention unit is configured to: sequentially perform channel number adjustment and dimension conversion on the original feature map to obtain an adjusted feature vector; perform cross-attention based backdoor adjustment on the adjusted feature vector and the hybrid factor estimation value in the current iteration round respectively to obtain a backdoor adjusted feature vector; restore the dimension of the backdoor adjusted feature vector to the same dimension as that of the original feature map to obtain a corresponding dimension restored feature map; splice the dimension restored feature map and the original feature map to obtain a corresponding spliced feature map; and reduce the dimension of the spliced feature map to the same dimension as that of the original feature map to obtain a corresponding target feature map represented after causal intervention.

4. The grain bulk surface pest detection model training method according to claim 3, characterized in that, The comparative causal feature alignment module comprises a sample mining unit; and the comparative causal feature alignment modules corresponding to the first model branch and the second model branch each share a same causal feature alignment unit; The sample mining unit is configured to take other feature vectors in the target feature map represented after causal intervention except the positive samples as negative samples respectively; Each negative sample is sorted in an order from high to low in terms of confidence, and a plurality of negative samples with the same number as that of the positive samples are selected in a front-to-back order from the sorted negative samples to be difficult negative samples respectively; The causal feature alignment unit is configured to: calculate a comparative loss corresponding to each positive sample based on each positive sample and each difficult negative sample obtained by the sample mining unit in the first model branch and the sample mining unit in the second model branch respectively, so as to reduce the feature distance between the positive samples and increase the feature distance between the positive samples and the difficult negative samples; and perform weighted average calculation on the comparative loss corresponding to each positive sample to obtain a causal feature alignment total loss corresponding to a current iteration round.

5. The grain bulk surface pest detection model training method according to claim 1, characterized in that, Before the different types of data samples generated based on the grain pile surface image data labeled with pest bounding boxes in multiple data domains are taken as the positive samples, the method further comprises: acquire image data sets corresponding to each of the preset data domains, each of the image data sets containing a plurality of grain pile surface image data labeled with a pest bounding box tag; perform randomization operations on the frequency domain and the spatial domain of each of the grain pile surface image data to obtain enhanced image data corresponding to each of the grain pile surface image data; generate different types of data samples according to each of the grain pile surface image data and the enhanced image data corresponding to each of the grain pile surface image data.

6. The grain bulk surface pest detection model training method according to claim 5, characterized in that, The generating of different types of data samples according to each of the grain pile surface image data and the enhanced image data corresponding to each of the grain pile surface image data includes: merging each of the image data sets to obtain a first training data set, and taking each of the grain pile surface image data in the first training set as a type of data sample for training the first model branch; and taking the enhanced image data corresponding to each of the grain pile surface image data as another type of data sample for training the second model branch.

7. The grain bulk surface pest detection model training method according to claim 5, characterized in that, The generating of different types of data samples according to each of the grain pile surface image data and the enhanced image data corresponding to each of the grain pile surface image data includes: dividing each of the image data sets into a second training data set and a third training data set, so that the second training data set and the third training data set each contain the grain pile surface image data of different data domains; adding the enhanced image data corresponding to each of the grain pile surface image data in the second training data set to the second training data set, and taking each of the grain pile surface image data and the enhanced image data in the second training set as a type of data sample for training the first model branch; and adding the enhanced image data corresponding to each of the grain pile surface image data in the third training data set to the third training data set, and taking each of the grain pile surface image data and the enhanced image data in the third training set as another type of data sample for training the second model branch.

8. The grain bulk surface pest detection model training method according to claim 2, characterized in that, The generating of the grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data according to the first model branch includes: taking the model composed of the convolutional neural network and the detection head in the first model branch as the grain pile surface pest detection model for outputting corresponding pest detection result data according to grain pile surface image data, wherein the convolutional neural network in the grain pile surface pest detection model is connected with the detection head.

9. A method for detecting a surface pest of a grain mass, characterized by, It includes: acquiring target grain pile surface image data in a grain storage space; input the target grain pile surface image data into the grain pile surface pest detection model, so that the grain pile surface pest detection model outputs the pest detection result data corresponding to the target grain pile surface image data; wherein the grain pile surface pest detection model is obtained by pre-training using the grain pile surface pest detection model training method in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the grain pile surface pest detection model training method in any one of claims 1 to 8 and / or the grain pile surface pest detection method in claim 9 when executing the computer program.

Images