Neural network model training method, food defect identification method and electronic equipment

Abstract

The invention relates to a neural network model training method, a food defect identification method and electronic equipment, and the method comprises the steps: carrying out the data enhancement of a preset real label sample image set, and determining a real label enhanced sample image set; according to a first preset neural network and a preset label-free sample image set, pseudo label generation and consistency calibration are carried out, a pseudo label sample image set is determined, and any pseudo label food sample image in the pseudo label sample image set has at least one target pseudo label; according to the real label enhanced sample image set and the pseudo label sample image set, semi-supervised iteration training is carried out on a second preset neural network, a target food defect recognition model is determined, the target food defect recognition model is used for carrying out food defect recognition on the to-be-recognized food image, and a food defect recognition result of the to-be-recognized food image is determined. The accuracy and generalization ability of the target food defect identification model can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a neural network model training method, a food defect recognition method, and an electronic device. Background Technology

[0002] Food safety and quality control are crucial aspects of the food industry. Identifying food defects (e.g., foreign objects, mold, rot, deformation, disease) is a vital step in ensuring food safety. Current technologies typically rely on manual inspection and image recognition methods for food defect identification. Manual inspection methods are subjective, inefficient, and costly. Meanwhile, commonly used image recognition methods often suffer from poor accuracy and robustness due to the limited amount of sample image data containing food defects. Summary of the Invention

[0003] In view of this, this disclosure proposes a technical solution for a neural network model training method, a food defect recognition method, and an electronic device.

[0004] According to one aspect of this disclosure, a neural network model training method is provided, comprising: performing data augmentation on a preset set of real-label sample images to determine a set of real-label-enhanced sample images, wherein any real-label-enhanced food sample image in the set of real-label-enhanced sample images is determined by local mixing based on at least two real-label food sample images in the preset set of real-label sample images, and any real-label-enhanced food sample image has at least one real label; performing pseudo-label generation and consistency calibration based on a first preset neural network and a preset set of unlabeled sample images to determine a set of pseudo-label sample images, wherein any pseudo-label food sample image in the set of pseudo-label sample images has at least one target pseudo-label; and performing semi-supervised iterative training on a second preset neural network based on the set of real-label-enhanced sample images and the set of pseudo-label sample images to determine a target food defect recognition model, wherein the target food defect recognition model is used to recognize food defects in a food image to be recognized, and to determine the food defect recognition result of the food image to be recognized.

[0005] In one possible implementation, the step of performing data augmentation on a preset set of real label sample images to determine a set of real label augmented sample images includes: for any real label food sample image in the preset set of real label sample images, determining a candidate real label food sample image corresponding to the real label food sample image based on the real label of the real label food sample image, wherein the candidate real label food sample image corresponding to any real label food sample image has at least one label of the same type as the real label food sample image; and for any real label food sample image in the preset set of real label sample images, performing local blending on the real label food sample image and the corresponding candidate real label food sample image to determine a real label augmented food sample image corresponding to the real label food sample image.

[0006] In one possible implementation, the step of locally blending any real-label food sample image from the preset real-label sample image set with a corresponding candidate real-label food sample image to determine the real-label enhanced food sample image corresponding to the real-label food sample image includes: for any real-label food sample image from the preset real-label sample image set, determining a target region in the real-label food sample image and a candidate region in the corresponding candidate real-label food sample image based on any label of the same type shared by the real-label food sample image and the corresponding candidate real-label food sample image; performing weighted fusion on the target region and the candidate region to determine the blending result; and replacing the target region with the blending result to determine the real-label enhanced food sample image corresponding to the real-label food sample image.

[0007] In one possible implementation, the step of generating pseudo-labels and performing consistency calibration based on a first preset neural network and a preset set of unlabeled sample images to determine the set of pseudo-labeled sample images includes: for any unlabeled food sample image in the preset set of unlabeled sample images, generating at least one initial pseudo-label for the unlabeled food sample image based on the first preset neural network to obtain a pseudo-labeled food sample image corresponding to the unlabeled food sample image; for any pseudo-labeled food sample image, filtering all initial pseudo-labels of the pseudo-labeled food sample image according to a preset confidence threshold to determine multiple first pseudo-labels of the pseudo-labeled food sample image; performing consistency calibration on the multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image; and determining the set of pseudo-labeled sample images based on each pseudo-labeled food sample image and its corresponding at least one target pseudo-label.

[0008] In one possible implementation, the step of performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: determining the target food type of the pseudo-labeled food sample image based on multiple pseudo-labels to be calibrated, wherein the pseudo-label to be calibrated is a first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing visual semantic consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image; and performing context consistency calibration on multiple pseudo-labels to be calibrated based on the target food type of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0009] In one possible implementation, the step of performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: determining multiple regional visual features of the pseudo-labeled food sample image based on multiple pseudo-labels to be calibrated, wherein the pseudo-labels to be calibrated are the first pseudo-labels of the pseudo-labeled food sample image, or pseudo-labels obtained after performing contextual consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image; dividing the multiple pseudo-labels to be calibrated of the pseudo-labeled food sample image into at least one group of pseudo-labels to be calibrated based on the similarity between any two regional visual features; and performing visual semantic consistency calibration on all pseudo-labels within any group of pseudo-labels to be calibrated to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0010] In one possible implementation, the step of performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: determining the overlap, intersection, and union ratio (IoU) of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image based on the multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image, wherein the pseudo-labels to be calibrated are the first pseudo-labels of the pseudo-labeled food sample image, or pseudo-labels obtained after performing contextual consistency calibration and / or visual semantic consistency calibration on the multiple first pseudo-labels of the pseudo-labeled food sample image; and performing spatial consistency calibration based on the IoU of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0011] In one possible implementation, the method further includes: during the semi-supervised iterative training of the second preset neural network, simultaneously iteratively updating the model parameters of the first preset neural network using exponential moving average and buffer weight updates based on the model parameters of the second preset neural network.

[0012] According to another aspect of this disclosure, a method for identifying food defects is provided, comprising: determining a food image to be identified; using a target food defect identification model to identify food defects in the food image to be identified, and determining the food defect identification result of the food image to be identified, wherein the target food defect identification model is trained according to the method described above.

[0013] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0014] In this embodiment, data augmentation is performed on a preset set of real-label sample images to construct a larger set of real-label augmented sample images with higher feature diversity. Furthermore, any real-label augmented food sample image in the set is determined by local mixing of at least two real-label food sample images from the preset set. Each real-label augmented food sample image has at least one real label, thereby reducing the impact of insufficient training data on subsequent deep learning model training and improving the accuracy and generalization ability of the trained target food defect recognition model. Based on the first preset neural network and the preset unlabeled sample image set, pseudo-label generation and consistency calibration can be performed to ensure the accuracy and reliability of the target pseudo-labels for each pseudo-labeled food sample image. This determines the pseudo-label sample image set, providing high-quality supervision signals for the semi-supervised training of the second preset neural network, further improving the accuracy of the trained target food defect recognition model. Based on the real label-enhanced sample image set and the pseudo-label sample image set, the second preset neural network can be semi-supervised iteratively trained to determine the target food defect recognition model, which can be used to identify food defects in the food image to be identified, and determine the food defect recognition result of the food image to be identified, thereby improving the accuracy and reliability of food defect recognition. It also has high adaptability to application scenarios with complex backgrounds, variable lighting, or complex defect morphologies.

[0015] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0017] Figure 1 A flowchart is shown for a neural network model training method according to an embodiment of the present disclosure;

[0018] Figure 2 This diagram illustrates a process for local hybrid data enhancement according to an embodiment of the present disclosure;

[0019] Figure 3 A schematic diagram illustrating a conformance calibration process according to an embodiment of the present disclosure is shown.

[0020] Figure 4 A flowchart illustrating a method for identifying food defects according to an embodiment of the present disclosure is shown.

[0021] Figure 5 A block diagram of a neural network model training apparatus according to an embodiment of the present disclosure is shown;

[0022] Figure 6 A block diagram of a food defect identification device according to an embodiment of the present disclosure is shown;

[0023] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0024] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0025] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0026] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0027] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0028] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0031] Food safety and quality control are crucial aspects of the food industry. Among these, the identification of food defects (such as foreign objects, mold, rot, deformation, and disease) is an important step in ensuring food safety.

[0032] In existing technologies, food defect recognition typically relies on manual inspection methods and image recognition methods. Among them, manual inspection methods mainly depend on human visual inspection to check for defects in food or food images. This method is highly subjective, has low recognition efficiency, and is costly. In addition, when facing real-world application scenarios with complex backgrounds, variable lighting, or complex defect morphologies, the accuracy and robustness of food defect recognition are also poor.

[0033] Image recognition methods can include traditional image processing methods or machine learning-based image processing methods. Traditional image processing techniques, such as thresholding, color histograms, or filtering, also suffer from the problems common to manual detection methods. Machine learning-based image processing methods can detect food blemishes by manually designing features (e.g., texture, color, shape features) and combining them with classifiers such as Support Vector Machines (SVMs). While this method outperforms traditional image processing methods in terms of food blemish recognition performance, its feature design process relies heavily on expert experience, resulting in limited generalization ability and difficulty adapting to the diverse types of food blemishes encountered in real-world applications.

[0034] With the development of deep learning models, represented by Convolutional Neural Networks (CNNs), object detection methods based on deep learning models have been applied to the detection of food defects. However, existing deep learning models typically rely on private datasets collected in controlled laboratory environments. The food sample images often have uniform backgrounds and lighting, creating a significant domain gap compared to complex real-world scenarios such as planting, transportation, and sales. Furthermore, acquiring large-scale, finely labeled food sample images (typically including both the location and type of food defects) is costly, resulting in insufficient training data for deep learning models.

[0035] Furthermore, with limited training data, deep learning models are prone to overfitting. They may learn spurious associations between food defects and irrelevant features of the food itself (e.g., the shape, color, or size of a fruit), failing to accurately learn the intrinsic characteristics of the defects. For example, a deep learning model might incorrectly assume that only apples of a certain shape have bruises, thus failing to accurately detect bruises on apples of other shapes, resulting in poor generalization ability.

[0036] In existing technologies, semi-supervised learning (SSL) methods are typically used to train deep learning models on a large amount of unlabeled data, thereby alleviating the problem of insufficient training data. However, when the commonly used semi-supervised object detection training framework is directly applied to food defect detection tasks, it often fails to learn effectively. This is due to factors such as the teacher model's inability to adapt to the domain shift between labeled and unlabeled data, or the traditional high-confidence pseudo-label filtering strategy filtering out a large number of supervision signals that are of high value for fine-grained and highly similar food defect types, leading to training failures in deep learning models.

[0037] In view of this, this disclosure provides a neural network model training method that can reduce the impact of insufficient training data on model training and improve the accuracy and generalization ability of the trained target food defect recognition model. The neural network model training method provided in this disclosure is described in detail below.

[0038] Figure 1 A flowchart illustrating a neural network model training method according to an embodiment of this disclosure is shown. This neural network model training method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The neural network model training method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the neural network model training method can be executed by a server. Figure 1 As shown, the training method for this neural network model includes:

[0039] In step S101, data augmentation is performed on the preset real label sample image set to determine the real label augmented sample image set. Any real label augmented food sample image in the real label augmented sample image set is determined by local mixing based on at least two real label food sample images in the preset real label sample image set. Any real label augmented food sample image has at least one real label.

[0040] The specific details of the food mentioned here can be flexibly set according to actual usage needs. For example, it can be any kind of fruit, any kind of staple food, etc., and this disclosure does not impose specific limitations on it. A real-label food sample image can represent a food image with a real label, and its specific form can be flexibly set according to actual usage needs. This disclosure does not impose specific limitations on it. The specific content of the real label of any real-label food sample image can be flexibly set according to actual usage needs. For example, it can include the food type, food location, and food defect type of each food item in the real-label food sample image, etc., and this disclosure does not impose specific limitations on it.

[0041] A preset set of real-label sample images can represent a training dataset that includes multiple real-label food sample images. Specific methods for obtaining the preset set of real-label sample images can be found in related technical implementations, and this disclosure does not impose specific limitations on it.

[0042] By locally blending at least two real-label food sample images from a pre-defined set of real-label sample images, the diversity of food-irrelevant features such as color and shape in the images can be increased. This reduces spurious associations between food defects and irrelevant features in the training data dimension, resulting in real-label enhanced food sample images. This process can also be called data augmentation based on local blending (BboxMixUp).

[0043] By augmenting food sample images with real labels, a larger set of real-label augmented sample images with higher feature diversity can be constructed, thereby reducing the impact of insufficient training data on subsequent deep learning model training and improving the accuracy and generalization ability of the trained target food defect recognition model.

[0044] The following sections will describe in detail the process of data augmentation of a preset set of real label sample images and the determination of the set of real label augmented sample images, based on possible implementations of this disclosure. These details will not be elaborated upon here.

[0045] In step S102, pseudo-labels are generated and consistency calibration is performed based on the first preset neural network and the preset unlabeled sample image set to determine the pseudo-label sample image set, wherein any pseudo-labeled food sample image in the pseudo-label sample image set has at least one target pseudo-label.

[0046] The first preset neural network, also known as the teacher model, can be flexibly set according to actual usage needs. For example, the first preset neural network can adopt the RTMDet-x model, etc. This disclosure does not make specific limitations on it.

[0047] The pre-defined unlabeled sample image set can represent a training dataset including multiple unlabeled food sample images. Unlabeled food sample images can represent food images containing any type of food without any labels. Their specific form can be flexibly set according to actual usage needs, and this disclosure does not impose specific limitations on them. It should be noted that any unlabeled food sample image typically contains only one type of food. For example, in the case of an unlabeled fruit sample image, the unlabeled fruit sample image will not simultaneously include two different types of fruit, in order to reduce the probability of recognition errors during pseudo-label generation.

[0048] Using a first preset neural network, pseudo-labels can be generated from a preset set of unlabeled sample images. Based on this, consistency calibration is performed on the pseudo-labels generated by the first preset neural network to determine the set of pseudo-labeled sample images. Each pseudo-labeled food sample image in the set of pseudo-labeled sample images has at least one target pseudo-label. The specific content of the target pseudo-label can be referenced from the content of the real label of any real-labeled food sample image; this disclosure does not impose specific limitations on this.

[0049] Consistency-Guided Pseudo-Label Calibration (CGPC) can represent the process of correcting errors and merging similar pseudo-labels generated by a first-preset neural network across multiple dimensions, including the contextual semantics, visual feature semantics, and spatial overlap of an image. The specific method of consistency calibration can be flexibly set according to actual usage requirements, and this disclosure does not impose specific limitations on it.

[0050] Typically, the pseudo-labels generated by the first preset neural network contain a lot of noise, such as pseudo-labels with incorrect food type labeling, pseudo-labels with incorrect food defect type labeling, and redundant pseudo-labels that repeatedly locate the same food. Based on this, consistency calibration can ensure the accuracy and reliability of the target pseudo-labels for each pseudo-labeled food sample image.

[0051] The following sections will describe in detail the process of generating pseudo-labels and performing consistency calibration based on the first preset neural network and the preset unlabeled sample image set, and determining the pseudo-label sample image set, in conjunction with the possible implementation methods of this disclosure. These details will not be elaborated here.

[0052] In step S103, the second preset neural network is semi-supervised iteratively trained based on the real label-enhanced sample image set and the pseudo-label sample image set to determine the target food defect recognition model. The target food defect recognition model is used to identify food defects in the food image to be identified and to determine the food defect recognition result of the food image to be identified.

[0053] By using a set of augmented sample images with real labels and a set of pseudo-labeled sample images, a semi-supervised iterative training of a second pre-defined neural network can be performed to determine the target food defect recognition model. The second pre-defined neural network, also known as the student model, can be flexibly configured according to actual usage requirements. For example, the second pre-defined neural network can adopt the RTMDet-x model, etc., and this disclosure does not impose specific limitations on it.

[0054] For specific methods of semi-supervised iterative training of the second preset neural network, refer to the semi-supervised learning framework in related technologies. For example, the gradient descent method can be used to update the model parameters of the second preset neural network, and the exponential moving average (EMA) method can be used to synchronously iteratively update the model parameters of the first preset neural network during the training process. This disclosure does not make specific limitations in this regard.

[0055] In one possible implementation, the method further includes: during the semi-supervised iterative training of the second preset neural network, simultaneously iteratively updating the model parameters of the first preset neural network using exponential moving average and buffer weight updates based on the model parameters of the second preset neural network.

[0056] Commonly used semi-supervised learning frameworks in the prior art often suffer from the problem that the teacher model cannot adapt to the domain shift between labeled and unlabeled data. Therefore, the neural network model training method of this disclosure also provides an improved semi-supervised learning framework. Specifically, during the semi-supervised iterative training of the second preset neural network, the model parameters of the first preset neural network can be synchronously and iteratively updated according to the model parameters of the second preset neural network using exponential moving average and buffer weight updates. This dynamic update of the buffer weights of the first preset neural network improves its adaptability to the statistical distribution of unlabeled data (i.e., the preset unlabeled sample image set) and reduces the probability of performance degradation due to data distribution shift.

[0057] The specific content of the buffer weights of the first preset neural network can be flexibly set according to actual usage requirements. For example, it may include the mean and square in the batch normalization layer. This disclosure does not make specific limitations on this.

[0058] Furthermore, instead of using a semi-supervised learning framework, a set of real-label augmented sample images can be directly used. In a fully supervised learning mode, the second preset neural network is iteratively trained. Relying solely on the data diversity of the real-label augmented sample image set reduces the training of the target food defect recognition model. Compared to other target detection models in the existing technology, this can also improve the accuracy and reliability of food defect recognition.

[0059] In addition to its application in food defect detection, the neural network training method provided in this disclosure can also be applied to other visual detection scenarios with high requirements for fine-grained detection, such as industrial part defect detection, crop pest and disease identification, and medical image lesion identification.

[0060] In this embodiment, data augmentation is performed on a preset set of real-label sample images to construct a larger set of real-label augmented sample images with higher feature diversity. Furthermore, any real-label augmented food sample image in the set is determined by local mixing of at least two real-label food sample images from the preset set. Each real-label augmented food sample image has at least one real label, thereby reducing the impact of insufficient training data on subsequent deep learning model training and improving the accuracy and generalization ability of the trained target food defect recognition model. Based on the first preset neural network and the preset unlabeled sample image set, pseudo-label generation and consistency calibration can be performed to ensure the accuracy and reliability of the target pseudo-labels for each pseudo-labeled food sample image. This determines the pseudo-label sample image set, providing high-quality supervision signals for the semi-supervised training of the second preset neural network, further improving the accuracy of the trained target food defect recognition model. Based on the real label-enhanced sample image set and the pseudo-label sample image set, the second preset neural network can be semi-supervised iteratively trained to determine the target food defect recognition model, which can be used to identify food defects in the food image to be identified, and determine the food defect recognition result of the food image to be identified, thereby improving the accuracy and reliability of food defect recognition. It also has high adaptability to application scenarios with complex backgrounds, variable lighting, or complex defect morphologies.

[0061] In one possible implementation, data augmentation is performed on a preset set of real label sample images to determine a set of real label augmented sample images. This includes: for any real label food sample image in the preset set of real label sample images, determining a candidate real label food sample image corresponding to the real label food sample image based on the real label of the real label food sample image, wherein the candidate real label food sample image corresponding to any real label food sample image has at least one label of the same type as the real label food sample image; and for any real label food sample image in the preset set of real label sample images, locally blending the real label food sample image and the corresponding candidate real label food sample image to determine the real label augmented food sample image corresponding to the real label food sample image.

[0062] For any real-label food sample image in the preset real-label sample image set, real-label food sample images with at least one label of the same type can be selected from the preset real-label sample image set based on the real label of the real-label food sample image, so as to serve as candidate real-label food sample images corresponding to the real-label food sample image.

[0063] Figure 2 A schematic diagram illustrating a localized hybrid data enhancement process according to an embodiment of the present disclosure is shown. Figure 2 As shown in (a), the preset set of real label sample images includes multiple real label food sample images, and any real label food sample image includes a type of fruit. Based on the real labels included in each real label food sample image, all real label food sample images are classified, and one real label food sample image and its corresponding candidate real label food sample image are determined from them.

[0064] For any real-label food sample image in the preset real-label sample image set, the two real-label food sample images can be locally blended based on the real-label food sample image and any similar label of the corresponding candidate real-label food sample image. This determines the real-label enhanced food sample image corresponding to the real-label food sample image. By ensuring the consistency of the real label type, the real-label enhanced food sample image obtained by local blending enhancement has high realism and reliability, reducing the possibility of food or food defects that cannot exist in real-world scenarios.

[0065] In one possible implementation, for any real-label food sample image in a preset set of real-label sample images, the real-label food sample image and the corresponding candidate real-label food sample image are locally blended to determine the real-label enhanced food sample image corresponding to the real-label food sample image. This includes: for any real-label food sample image in the preset set of real-label sample images, based on any label of the same type that the real-label food sample image and the corresponding candidate real-label food sample image have, determining the target region in the real-label food sample image and the candidate region in the corresponding candidate real-label food sample image; performing weighted fusion on the target region and the candidate region to determine the blending result; replacing the target region with the blending result to determine the real-label enhanced food sample image corresponding to the real-label food sample image.

[0066] For any real-label food sample image in a preset set of real-label sample images, based on any label of the same type shared by the real-label food sample image and its corresponding candidate real-label food sample image, a target region can be determined in the real-label food sample image as the first instance of local blending. Similarly, based on any label of the same type shared by the real-label food sample image and its corresponding candidate real-label food sample image, a candidate region can be determined in the candidate real-label food sample image as the second instance of local blending. Using similar labels for instance selection in local blending (i.e., determining the target region and candidate region) ensures that local blending occurs between instances of the same type (e.g., two foods of the same type with the same type of food defects), reducing the possibility of generating foods or food defects that are impossible to exist in a real-world scenario. This process can be called Mixing Candidate Selection.

[0067] For any real-label food sample image, a weighted fusion can be performed on the target region in the real-label food sample image and the candidate region in the corresponding candidate real-label food sample image to obtain a blended result. Replacing the target region in the real-label food sample image with the blended result yields the real-label enhanced food sample image corresponding to the real-label food sample image. This process can be called a local blending operation.

[0068] The specific method of weighted fusion can be flexibly set according to actual usage needs, and this disclosure does not impose specific limitations on it.

[0069] In one example, for any real-label food sample image, the target region in the real-label food sample image can be cropped to obtain a first cropping result; the candidate region in the candidate real-label food sample image corresponding to the real-label food sample image can be cropped to obtain a second cropping result, and the size of the second cropping result can be adjusted to be the same as the size of the first cropping result to obtain a scaled second cropping result; according to a preset distribution, a random mixing coefficient is determined; according to the random mixing coefficient, the first cropping result and the scaled second cropping result are weighted and fused to determine the mixing result. The process of weighted fusion according to the random mixing coefficient can be expressed as formula (1):

[0070]

[0071] Where MixedRegion represents the pixel value of the blended result; λ represents the random blending coefficient; TargetRegion represents the pixel value of the first cropped result; and CandiateRegion represents the pixel value of the second cropped result after scaling at each size.

[0072] The specific form of the preset distribution can be flexibly set according to actual usage requirements. For example, a Beta(α, β) distribution can be used. This disclosure does not impose any specific limitations on this.

[0073] Based on the above Figure 2 For example, Figure 2 As shown in (b), the white bounding box in the real-label food sample image represents the target region, and the white bounding box in the candidate real-label food sample image represents the candidate region. The target region in the real-label food sample image is cropped to obtain a first cropping result; the candidate region in the candidate real-label food sample image is cropped to obtain a second cropping result; the size of the second cropping result is adjusted to be the same as the size of the first cropping result to obtain a scaled-down second cropping result, which is then weighted and fused to obtain a mixed result; the target region in the real-label food sample image is replaced using the mixed result to determine the real-label enhanced food sample image corresponding to the real-label food sample image.

[0074] Through the above process, the real-labeled enhanced food sample images can have diversity of food-irrelevant features such as background, shape, and color. Thus, when training the second preset neural network using the real-labeled enhanced sample image set, the second preset neural network is forced to focus on the feature information in each real-labeled enhanced food sample image that is truly related to food defects and affects the accuracy of food defect recognition, thereby improving the generalization ability and accuracy of the trained target food defect recognition model.

[0075] In one possible implementation, for any real-label food sample image in a preset set of real-label sample images, the real-label food sample image and the corresponding candidate real-label food sample image are locally blended to determine the real-label enhanced food sample image corresponding to the real-label food sample image. This includes: for any real-label food sample image in the preset set of real-label sample images, performing feature extraction on the real-label food sample image and the corresponding candidate real-label food sample image respectively to obtain a first feature map of the real-label food sample image and a second feature map of the corresponding candidate real-label food sample image; determining a target feature region in the first feature map and a candidate feature region in the second feature map based on any label of the same type shared by the real-label food sample image and the corresponding candidate real-label food sample image; performing weighted fusion on the target feature region and the candidate feature region to obtain a feature blending result; and determining the real-label enhanced food sample image corresponding to the real-label food sample image based on the feature blending result.

[0076] The specific method for weighted fusion of the target feature region and the candidate feature region to obtain the feature mixing result can be flexibly set according to actual usage requirements, and this disclosure does not impose specific limitations on it.

[0077] In addition, the feature mixing result of any real labeled food sample image can be directly used as the input of the second preset neural network, which can also achieve the effect of improving the diversity of training data used by the second preset neural network.

[0078] In one possible implementation, pseudo-label generation and consistency calibration are performed based on a first preset neural network and a preset set of unlabeled sample images to determine the set of pseudo-labeled sample images. This includes: for any unlabeled food sample image in the preset set of unlabeled sample images, generating at least one initial pseudo-label for the unlabeled food sample image based on the first preset neural network to obtain a pseudo-labeled food sample image corresponding to the unlabeled food sample image; for any pseudo-labeled food sample image, filtering all initial pseudo-labels of the pseudo-labeled food sample image according to a preset confidence threshold to determine multiple first pseudo-labels of the pseudo-labeled food sample image; performing consistency calibration on the multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image; and determining the set of pseudo-labeled sample images based on each pseudo-labeled food sample image and its corresponding at least one target pseudo-label.

[0079] For any unlabeled food sample image in a preset set of unlabeled sample images, a first preset neural network can be used to generate at least one initial pseudo-label for the unlabeled food sample image, thereby obtaining a pseudo-labeled food sample image corresponding to the unlabeled food sample image. The specific method for generating the pseudo-label using the first preset neural network can be found in related technologies and is related to the actual form of the first preset neural network; this disclosure does not specifically limit it.

[0080] Typically, pseudo-labels generated by the first pre-defined neural network contain a large amount of noise, such as pseudo-labels with incorrect food type labels, pseudo-labels with incorrect food defect type labels, and redundant pseudo-labels that repeatedly locate the same food. Therefore, to ensure the accuracy of pseudo-labels, common semi-supervised learning frameworks in existing technologies usually employ a high-confidence pseudo-label filtering strategy to directly filter pseudo-labels generated by the teacher model. However, this pseudo-label filtering strategy, which relies solely on confidence, easily filters out pseudo-labels that are actually accurate but have relatively low confidence.

[0081] In view of this, in this embodiment of the present disclosure, for any pseudo-labeled food sample image, all initial pseudo-labels of the pseudo-labeled food sample image can be initially screened according to a preset confidence threshold with a relatively low value to determine multiple first pseudo-labels of the pseudo-labeled food sample image; then, through consistency calibration, the multiple first pseudo-labels of the pseudo-labeled food sample image are calibrated and filtered, thereby improving the retention rate of accurate but relatively low confidence pseudo-labels, so that the target pseudo-labels of the pseudo-labeled food sample image can include more food defects with high inter-category similarity and fine granularity.

[0082] The specific method for determining the confidence level of each initial pseudo-label in any pseudo-labeled food sample image can refer to the implementation methods in related technologies, and this disclosure does not impose specific limitations on it. The specific value of the preset confidence threshold can be flexibly set according to actual usage needs, and should be lower than the confidence screening threshold conditions used in common semi-supervised learning frameworks. For example, it can be set to 0.35, etc., and this disclosure does not impose specific limitations on it.

[0083] Based on each pseudo-labeled food sample image and its corresponding at least one target pseudo-label, a set of pseudo-labeled sample images with high accuracy and reliability can be determined as a supervision signal in the semi-supervised training process. This set, together with the set of real label-enhanced sample images, is used to train the second preset neural network. This further reduces the impact of the small amount of training data on the model performance of the trained target food defect recognition model, and improves the accuracy of the target food defect recognition model without increasing the labeling cost of the training data.

[0084] In one possible implementation, consistency calibration is performed on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image, including: determining the target food type of the pseudo-labeled food sample image based on multiple pseudo-labels to be calibrated, wherein the pseudo-label to be calibrated is a first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing visual semantic consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image; and performing context consistency calibration on multiple pseudo-labels to be calibrated based on the target food type of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0085] Typically, any unlabeled food sample image usually contains only one type of food. For example, if the unlabeled food sample image is an unlabeled fruit sample image, it may include both "flawless apples" and "rotten apples", but not oranges. However, due to the recognition error of the first preset neural network, among the multiple first pseudo-labels of the corresponding pseudo-labeled food sample image, there may be a first pseudo-label indicating that the food type is oranges.

[0086] Figure 3 A schematic diagram illustrating a conformance calibration process according to an embodiment of the present disclosure is shown. Figure 3 As shown, the pseudo-labeled food sample image includes three strawberries, and the first pseudo-label of the food sample image is represented by a colored annotation box. The color of the annotation box indicates the food type and the type of food blemish, and the position of the annotation box indicates the location of the food. Specifically, a red annotation box represents the first pseudo-label of "food type is strawberry, food blemish type is rotten"; a green annotation box represents the first pseudo-label of "food type is strawberry, food blemish type is no blemish"; and a purple annotation box represents the first pseudo-label of "food type is cherry, food blemish type is no blemish".

[0087] like Figure 3 As shown in (a), the pseudo-labeled food sample image only includes three strawberries and no cherries, but the first pseudo-label 2 is the first pseudo-label of the purple annotation box, indicating "the food type is cherry and the food defect type is no defect", which shows that there is a labeling error of the food type.

[0088] Therefore, consistency calibration can include context-semantic consistency calibration to correct erroneous pseudo-labels for food types. Specifically, for multiple pseudo-labels to be calibrated in any given pseudo-labeled food sample image, the food type with the highest frequency of occurrence indicated by the multiple pseudo-labels can be determined as the target food type for that pseudo-labeled food sample image. Here, the pseudo-label to be calibrated is either the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing visual semantic consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image.

[0089] For any pseudo-labeled food sample image with multiple pseudo-labels to be calibrated, context consistency calibration is performed on the multiple pseudo-labels to be calibrated based on the target food type of the pseudo-labeled food sample image. The food type marked in all pseudo-labels to be calibrated is uniformly corrected to the target food type, thereby obtaining at least one target pseudo-label of the pseudo-labeled food sample image.

[0090] Based on the above Figure 3 For example, Figure 3 As shown in (b), after context consistency calibration, the first pseudo-label 2 is corrected to a green label box indicating "food type is strawberry, food defect type is no defect", which is the target pseudo-label 2.

[0091] In one possible implementation, performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: determining multiple regional visual features of the pseudo-labeled food sample image based on multiple pseudo-labels to be calibrated, wherein the pseudo-labels to be calibrated are the first pseudo-labels of the pseudo-labeled food sample image, or pseudo-labels obtained after performing contextual consistency calibration and / or spatial consistency calibration on the multiple first pseudo-labels of the pseudo-labeled food sample image; dividing the multiple pseudo-labels to be calibrated of the pseudo-labeled food sample image into at least one group of pseudo-labels to be calibrated based on the similarity between any two regional visual features; and performing visual semantic consistency calibration on all pseudo-labels within any group of pseudo-labels to be calibrated to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0092] Normally, different foods with the same type of food defects in any unlabeled food sample image will have high similarity in terms of visual features; however, due to the recognition error of the first preset neural network, the corresponding pseudo-labeled food sample image may contain different foods with the same type of food defects, but the labeling error of the first pseudo-label has different types of food defects.

[0093] Based on the above Figure 3 For example, Figure 3 As shown in (a), the food indicated by the first pseudo-label 1 and the food indicated by the first pseudo-label 4 are actually strawberries with the food defect type of rotten. However, the first pseudo-label 4 is a green label box that reads "food type is strawberry, food defect type is no defect", which indicates an error in labeling the food defect type.

[0094] Therefore, consistency calibration can include visual-semantic consistency calibration, which corrects incorrect pseudo-labels for food defect types at the visual feature level. Specifically, for multiple pseudo-labels to be calibrated in any pseudo-labeled food sample image, multiple regions of the pseudo-labeled food sample image can be determined; wherein, the pseudo-label to be calibrated is the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image.

[0095] The specific methods for determining the visual characteristics of a region can be flexibly set according to actual usage needs, and this disclosure does not impose specific limitations.

[0096] In one possible implementation, for any pseudo-label to be calibrated in any pseudo-labeled food sample image, the labeled region of the pseudo-label to be calibrated can be determined in the pseudo-labeled food sample image; using a preset visual feature extraction network, the labeled region of the pseudo-label to be calibrated can be processed to determine the regional visual features of the pseudo-label to be calibrated. The specific form of the preset visual feature extraction network can be flexibly set according to actual usage requirements, and this disclosure does not impose specific limitations on it.

[0097] In one example, to avoid cognitive bias caused by overfitting in the first and second preset neural networks within the semi-supervised learning framework, the preset visual feature extraction network can be set as a pre-trained external visual backbone network independent of the semi-supervised learning framework, such as the RegNet model or DINOv2.

[0098] In one example, the preset visual feature extraction network can be set as a feature extractor within a first preset neural network. For instance, visual feature extraction can be achieved by fixing the weights of the shallow layers of the first preset neural network. This method can improve the independence of the semi-supervised learning framework itself and reduce dependence on external models. However, the recognition error of the first preset neural network itself may affect the effectiveness of visual semantic consistency calibration.

[0099] For multiple regions of visual features in any pseudo-labeled food sample image, the similarity between each region's visual features can be determined. Based on the similarity between any two regions' visual features, the multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image can be divided into at least one group of pseudo-labels to be calibrated. The specific method for determining the similarity between any two regions' visual features can refer to implementation methods in related technologies, and this disclosure does not specifically limit it. The specific method for dividing the pseudo-label group based on the similarity between any two regions' visual features can be flexibly set according to actual usage requirements, and this disclosure does not specifically limit it.

[0100] In one possible implementation, for any two regions of visual features in any pseudo-labeled food sample image, if the similarity between these two regions is greater than or equal to a preset similarity threshold, the pseudo-labels corresponding to these two regions are assigned to the same pseudo-label group. The specific value of the preset similarity threshold can be flexibly set according to actual usage requirements; for example, it can be set to 0.9, etc., and this disclosure does not impose a specific limitation on it.

[0101] For any pseudo-label group to be calibrated, perform visual semantic consistency calibration on all pseudo-labels in the group, and uniformly correct the food defect types marked in all pseudo-labels to the most frequent food defect types, thereby determining at least one target pseudo-label for the food sample image of the pseudo-label.

[0102] In one possible implementation, consistency calibration is performed on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image, including: determining the overlap, intersection, and union ratio (IoU) of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image based on the multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image, wherein the pseudo-label to be calibrated is a first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or visual semantic consistency calibration on the multiple first pseudo-labels of the pseudo-labeled food sample image; and performing spatial consistency calibration based on the IoU of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0103] Due to the recognition error of the first preset neural network, any pseudo-labeled food sample image may contain redundant first pseudo-labels that repeatedly label the same food.

[0104] Based on the above Figure 3 For example, Figure 3As shown in (a), the first pseudo-label 3 and the first pseudo-label 4 are annotations for the same food in the food sample image of the pseudo-label. The corresponding annotation boxes of the two have obvious spatial overlap. Therefore, the first pseudo-label 3 and the first pseudo-label 4 have redundant first pseudo-labels that are repeatedly annotated.

[0105] Therefore, consistency calibration can include spatial consistency, which removes duplicate pseudo-labels from the spatial dimension. Specifically, for multiple pseudo-labels to be calibrated in a pseudo-labeled food sample image, the intersection-over-union (IoU) ratio of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image can be determined. Here, the pseudo-label to be calibrated is either the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or visual-semantic consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image.

[0106] The specific method for determining the overlap, crossover, and union ratio of any two pseudo-tags to be calibrated can be flexibly set according to actual usage requirements, and this disclosure does not impose specific limitations on it.

[0107] In one example, for any pseudo-label to be calibrated in any pseudo-labeled food sample image, the labeled region of the pseudo-label to be calibrated can be determined in the pseudo-labeled food sample image; based on the labeled regions of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image, the overlap ratio (CLU) of the two pseudo-labels to be calibrated can be determined using the Non-Maximum Suppression (NMS) algorithm.

[0108] For any two pseudo-labels to be calibrated in any pseudo-labeled food sample image, if the overlap ratio (Intersection over Union) of these two pseudo-labels exceeds a preset threshold, spatial consistency calibration is performed. Either the pseudo-label with the lower confidence level is removed, or the pseudo-label with the lower confidence level is directly deleted, thereby determining at least one target pseudo-label for the food sample image. The specific value of the preset threshold can be flexibly set according to actual usage requirements; for example, the preset threshold can be set to 0.7, etc., and this disclosure does not impose a specific limitation on it.

[0109] Based on the above Figure 3 For example, Figure 3 As shown in (b), after spatial consistency calibration, the annotation box of the first pseudo-label 4 was directly deleted, and the annotation box of the first pseudo-label 3 with higher accuracy was determined as the target pseudo-label 3.

[0110] Preferably, the multiple first pseudo-labels generated based on the first preset neural network and preliminarily screened can be calibrated and screened in three dimensions in the order of contextual semantic consistency calibration, visual semantic consistency calibration and spatial consistency calibration.

[0111] Referring to Table 1, using the neural network training method provided in this embodiment, a target food defect recognition model is constructed with the RTMDet-x model as the backbone network (i.e., the first preset neural network and the second preset neural network), and its performance is compared with common detection methods in the prior art. Performance evaluation metrics include: mean Average Precision at IoU=0.50:0.05:0.95 (mAP) from 0.5 to 0.95. 50:95 The mean average precision at IoU=0.50 (mAP) is calculated using the intersection-union ratio (IoU) threshold of 0.5. 50 The mean average precision at IoU=0.75 (mAP) is also measured. 75 The target food defect recognition model constructed using the neural network training method provided in this disclosure outperforms the commonly used detection methods in the prior art as described in Table 1 on all three performance evaluation indicators. This indicates that the target food defect recognition model constructed using the neural network training method provided in this disclosure has high performance in real-world food defect detection scenarios with many food types and food defect types, complex backgrounds, and limited training data.

[0112] Table 1

[0113]

[0114] Referring to Table 2, for the same backbone network, the performance of the trained food defect recognition model was evaluated under four scenarios: without using the neural network training method provided in this disclosure; using only the local mixed data augmentation in the neural network training method provided in this disclosure; using both the local mixed data augmentation and the improved semi-supervised learning framework in the neural network training method provided in this disclosure; using the local mixed data augmentation in the neural network training method provided in this disclosure; and using the improved semi-supervised learning framework and consistency calibration (i.e., the complete neural network training method). The performance evaluation metrics included the mean Average Precision at IoU = 0.50:0.05:0.95 (mAP) with intersection-over-union (IoU) thresholds ranging from 0.5 to 0.95. 50:95The mean average precision at IoU=0.50 (mAP) is calculated using the intersection-union ratio (IoU) threshold of 0.5. 50 The mean average precision at IoU=0.75 (mAP) is also measured. 75 Each additional method included in the neural network training method provided in this disclosure improves the performance of the trained food defect recognition model across the three performance evaluation metrics. This demonstrates that the neural network training method provided in this disclosure can improve the performance of the trained food defect recognition model in multiple ways.

[0115] Table 2

[0116]

[0117] In this embodiment, data augmentation is performed on a preset set of real-label sample images to determine a larger set of real-label augmented sample images with higher feature diversity. Furthermore, any real-label augmented food sample image in the set is determined by local mixing of at least two real-label food sample images from the preset set. Each real-label augmented food sample image has at least one real label, thereby reducing the impact of insufficient training data on subsequent deep learning model training and improving the accuracy and generalization ability of the trained target food defect recognition model. Based on the first preset neural network and the preset unlabeled sample image set, pseudo-label generation and consistency calibration can be performed to ensure the accuracy and reliability of the target pseudo-labels for each pseudo-labeled food sample image. This determines the pseudo-label sample image set, providing high-quality supervision signals for the semi-supervised training of the second preset neural network, further improving the accuracy of the trained target food defect recognition model. Based on the real label-enhanced sample image set and the pseudo-label sample image set, the second preset neural network can be semi-supervised iteratively trained to determine the target food defect recognition model, which can be used to identify food defects in the food image to be identified, and determine the food defect recognition result of the food image to be identified, thereby improving the accuracy and reliability of food defect recognition. It also has high adaptability to application scenarios with complex backgrounds, variable lighting, or complex defect morphologies.

[0118] According to another aspect of the present disclosure, a food defect recognition method is provided, which can utilize a target food defect recognition model with high recognition performance and generalization ability to perform food defect recognition on the food image to be recognized, thereby improving the accuracy and reliability of the food defect recognition results and being able to adapt to application scenarios in the real world with complex backgrounds, variable lighting, or complex defect morphologies.

[0119] Figure 4 A flowchart illustrating a food defect identification method according to an embodiment of this disclosure is shown. This food defect identification method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The food defect identification method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the food defect identification method can be executed by a server. Figure 4 As shown, the food defect identification method includes:

[0120] In step S401, the image of the food to be identified is determined.

[0121] The food image to be identified can represent any food image that needs to be identified for food defects. Its specific form can be flexibly set according to actual usage requirements, and this disclosure does not impose any specific limitations on it.

[0122] The specific method for determining the image of the food to be identified can be referred to the implementation methods in related technologies. For example, any image acquisition device can be used to acquire images of any food, and this disclosure does not make any specific limitations in this regard.

[0123] In step S402, the target food defect recognition model is used to identify food defects in the food image to be identified, and the food defect recognition result of the food image to be identified is determined. The target food defect recognition model is trained according to the above method.

[0124] The food image to be identified is input into the food defect recognition model obtained based on the aforementioned publicly disclosed neural network training method. The model then identifies food defects in the food image, determines the food defect recognition result, improves the accuracy and reliability of the food defect recognition result, and can adapt to application scenarios in the real world with complex backgrounds, variable lighting, or complex defect morphologies.

[0125] In this embodiment of the disclosure, after determining the food image to be identified, the target food defect recognition model trained according to the above method can be used to identify food defects in the food image to be identified, and the food defect recognition result of the food image to be identified can be determined. By utilizing the high recognition performance and generalization ability of the target food defect recognition model, the accuracy and reliability of the food defect recognition result can be improved, and it can adapt to application scenarios in the real world with complex backgrounds, variable lighting or complex defect morphologies.

[0126] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0127] In addition, this disclosure also provides a neural network model training device, which can be used to implement any neural network model training method provided in this disclosure. The corresponding technical solutions and descriptions can be found in the relevant descriptions in the method section, and will not be repeated here.

[0128] Figure 5 A block diagram of a neural network model training apparatus according to an embodiment of the present disclosure is shown. Figure 5 As shown, the device 500 includes:

[0129] The local blending data augmentation module 501 is used to perform data augmentation on a preset set of real label sample images and determine a set of real label augmented sample images. Any real label augmented food sample image in the set of real label augmented sample images is determined by local blending based on at least two real label food sample images in the preset set of real label sample images. Any real label augmented food sample image has at least one real label.

[0130] The pseudo-label generation and calibration module 502 is used to generate pseudo-labels and perform consistency calibration based on the first preset neural network and the preset unlabeled sample image set, and to determine the pseudo-label sample image set, wherein any pseudo-labeled food sample image in the pseudo-label sample image set has at least one target pseudo-label.

[0131] The semi-supervised learning module 503 is used to perform semi-supervised iterative training on the second preset neural network based on the real label augmented sample image set and the pseudo label sample image set to determine the target food defect recognition model. The target food defect recognition model is used to identify food defects in the food image to be identified and to determine the food defect recognition result of the food image to be identified.

[0132] In one possible implementation, the local blending data augmentation module 501 is configured to: for any real-label food sample image in a preset set of real-label sample images, determine a candidate real-label food sample image corresponding to the real-label food sample image based on the real label of the real-label food sample image, wherein the candidate real-label food sample image corresponding to any real-label food sample image has at least one label of the same type as the real-label food sample image; and for any real-label food sample image in the preset set of real-label sample images, perform local blending of the real-label food sample image and the corresponding candidate real-label food sample image to determine the real-label augmented food sample image corresponding to the real-label food sample image.

[0133] In one possible implementation, the local hybrid data enhancement module 501 is used to: for any real-label food sample image in a preset set of real-label sample images, determine the target region in the real-label food sample image and the candidate region in the corresponding candidate real-label food sample image based on any label of the same type that the real-label food sample image and the corresponding candidate real-label food sample image have; perform weighted fusion on the target region and the candidate region to determine the hybrid result; replace the target region with the hybrid result to determine the real-label enhanced food sample image corresponding to the real-label food sample image.

[0134] In one possible implementation, the pseudo-label generation and calibration module 502 is configured to: for any unlabeled food sample image in a preset unlabeled sample image set, generate at least one initial pseudo-label for the unlabeled food sample image according to a first preset neural network, thereby obtaining a pseudo-labeled food sample image corresponding to the unlabeled food sample image; for any pseudo-labeled food sample image, filter all initial pseudo-labels of the pseudo-labeled food sample image according to a preset confidence threshold, thereby determining multiple first pseudo-labels of the pseudo-labeled food sample image; perform consistency calibration on the multiple first pseudo-labels of the pseudo-labeled food sample image, thereby determining at least one target pseudo-label of the pseudo-labeled food sample image; and determine a pseudo-labeled sample image set based on each pseudo-labeled food sample image and its corresponding at least one target pseudo-label.

[0135] In one possible implementation, the pseudo-label generation and calibration module 502 is configured to: determine the target food type of the pseudo-label food sample image based on multiple pseudo-labels to be calibrated, wherein the pseudo-label to be calibrated is a first pseudo-label of the pseudo-label food sample image, or a pseudo-label obtained after performing visual semantic consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-label food sample image; and perform context consistency calibration on multiple pseudo-labels to be calibrated of the pseudo-label food sample image based on the target food type of the pseudo-label food sample image to determine at least one target pseudo-label of the pseudo-label food sample image.

[0136] In one possible implementation, the pseudo-label generation and calibration module 502 is configured to: determine multiple regional visual features of the pseudo-label food sample image based on multiple pseudo-labels to be calibrated, wherein the pseudo-label to be calibrated is a first pseudo-label of the pseudo-label food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-label food sample image; divide the multiple pseudo-labels to be calibrated of the pseudo-label food sample image into at least one group of pseudo-labels to be calibrated based on the similarity between any two regional visual features; and perform visual semantic consistency calibration on all pseudo-labels to be calibrated within any group of pseudo-labels to be calibrated to determine at least one target pseudo-label of the pseudo-label food sample image.

[0137] In one possible implementation, the pseudo-label generation and calibration module 502 is configured to: determine the overlap, intersection, and union ratio (IoU) of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image based on a plurality of pseudo-labels to be calibrated in the pseudo-labeled food sample image, wherein the pseudo-label to be calibrated is the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or visual semantic consistency calibration on a plurality of first pseudo-labels in the pseudo-labeled food sample image; and perform spatial consistency calibration based on the IoU of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

[0138] In one possible implementation, the device 500 further includes a semi-supervised learning adjustment module, used to: synchronously iteratively update the model parameters of the first preset neural network by using exponential moving average and buffer weight updates based on the model parameters of the second preset neural network during the semi-supervised iterative training process.

[0139] In addition, this disclosure also provides a food defect identification device, which can be used to implement any of the food defect identification methods provided in this disclosure. The corresponding technical solutions and descriptions can be found in the relevant records in the method section, and will not be repeated here.

[0140] Figure 6 A block diagram of a food defect detection device according to an embodiment of the present disclosure is shown. Figure 6 As shown, the device 600 includes:

[0141] Image determination module 601 is used to determine the image of the food to be identified.

[0142] The food defect recognition module 602 is used to recognize food defects in the food image to be recognized using a target food defect recognition model, and to determine the food defect recognition result of the food image to be recognized. The target food defect recognition model is trained according to the above method.

[0143] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0144] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0145] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, electronic device 1900 may be provided as a server or terminal device. (Refer to...) Figure 7 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0146] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0148] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a neural network model, characterized in that, include: Data augmentation is performed on a preset set of real label sample images to determine a set of real label augmented sample images. Any real label augmented food sample image in the set of real label augmented sample images is determined by local mixing of at least two real label food sample images in the preset set of real label sample images. Any real label augmented food sample image has at least one real label. Based on the first preset neural network and the preset unlabeled sample image set, pseudo-label generation and consistency calibration are performed to determine the pseudo-label sample image set, wherein any pseudo-labeled food sample image in the pseudo-label sample image set has at least one target pseudo-label. Based on the real label-enhanced sample image set and the pseudo-label sample image set, a second preset neural network is semi-supervised iteratively trained to determine a target food defect recognition model. The target food defect recognition model is used to identify food defects in the food image to be identified and to determine the food defect recognition result of the food image to be identified.

2. The method according to claim 1, characterized in that, The step of performing data augmentation on a preset set of real label sample images, and determining the set of real label augmented sample images, includes: For any real-label food sample image in the preset real-label sample image set, a candidate real-label food sample image corresponding to the real-label food sample image is determined based on the real label of the real-label food sample image. The candidate real-label food sample image corresponding to any real-label food sample image has at least one label of the same type as the real-label food sample image. For any real-label food sample image in the preset real-label sample image set, the real-label food sample image and the corresponding candidate real-label food sample image are locally blended to determine the real-label enhanced food sample image corresponding to the real-label food sample image.

3. The method according to claim 2, characterized in that, The step of locally blending any real-label food sample image from the preset set of real-label sample images with the corresponding candidate real-label food sample images to determine the real-label enhanced food sample image corresponding to the real-label food sample image includes: For any real-label food sample image in the preset real-label sample image set, the target region in the real-label food sample image and the candidate region in the corresponding candidate real-label food sample image are determined based on any label of the same type that the real-label food sample image and the corresponding candidate real-label food sample image have. The target region and the candidate region are weighted and fused to determine the fusion result; The target region is replaced with the blending result to determine the real-label enhanced food sample image corresponding to the real-label food sample image.

4. The method according to any one of claims 1 to 3, characterized in that, The step of generating pseudo-labels and performing consistency calibration based on a first preset neural network and a preset unlabeled sample image set, and determining the pseudo-label sample image set, includes: For any unlabeled food sample image in the preset unlabeled sample image set, at least one initial pseudo-label is generated for the unlabeled food sample image according to the first preset neural network, so as to obtain the pseudo-labeled food sample image corresponding to the unlabeled food sample image. For any pseudo-labeled food sample image, based on a preset confidence threshold, all initial pseudo-labels of the pseudo-labeled food sample image are filtered to determine multiple first pseudo-labels of the pseudo-labeled food sample image. Consistency calibration is performed on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image; The set of pseudo-label sample images is determined based on each pseudo-label food sample image and its corresponding at least one target pseudo-label.

5. The method according to claim 4, characterized in that, The step of performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: Based on multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image, the target food type of the pseudo-labeled food sample image is determined, wherein the pseudo-label to be calibrated is the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing visual semantic consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image. Based on the target food type of the pseudo-labeled food sample image, context consistency calibration is performed on multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

6. The method according to claim 4, characterized in that, The step of performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: Based on multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image, multiple regional visual features of the pseudo-labeled food sample image are determined, wherein the pseudo-label to be calibrated is the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or spatial consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image. Based on the similarity between the visual features of any two regions, the multiple pseudo-labels to be calibrated in the pseudo-label food sample image are divided into at least one group of pseudo-labels to be calibrated. For any pseudo-label group to be calibrated, perform visual semantic consistency calibration on all pseudo-labels in the pseudo-label group to determine at least one target pseudo-label for the food sample image of the pseudo-label.

7. The method according to claim 4, characterized in that, The step of performing consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image includes: Based on multiple pseudo-labels to be calibrated in the pseudo-labeled food sample image, determine the overlap, crossover, and union ratio of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image, wherein the pseudo-label to be calibrated is the first pseudo-label of the pseudo-labeled food sample image, or a pseudo-label obtained after performing contextual consistency calibration and / or visual semantic consistency calibration on multiple first pseudo-labels of the pseudo-labeled food sample image. Spatial consistency calibration is performed based on the overlap cross-union ratio of any two pseudo-labels to be calibrated in the pseudo-labeled food sample image to determine at least one target pseudo-label of the pseudo-labeled food sample image.

8. The method according to any one of claims 1 to 3, characterized in that, The method further includes: During the semi-supervised iterative training of the second preset neural network, the model parameters of the first preset neural network are synchronously updated using exponential moving average and buffer weight updates based on the model parameters of the second preset neural network.

9. A method for identifying food defects, characterized in that, include: Identify the food image to be recognized; Using a target food defect recognition model, food defects are identified in the food image to be identified, and the food defect recognition result of the food image to be identified is determined, wherein the target food defect recognition model is trained by the method according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images