A food foreign object detection method and system based on deep learning

Abstract

The present invention discloses a method and system for detecting foreign objects in food based on deep learning, including an image acquisition module and a foreign object analysis module connected thereto; the image acquisition module: for acquiring RGB images and penetration images of food; the foreign object analysis module: inputting the food images corresponding to the RGB images, the food images corresponding to the penetration images, color features and shape features into a foreign object detection model, relating to the technical field of food detection, and solving the technical problem that when using X-ray images of food and analyzing the foreign objects therein, the sensitivity for detecting low-density foreign objects in food X-ray images is too low, resulting in difficulty in meeting the current high-quality requirements for food; taking the color situation in food as one of the judgment bases and taking the shape features of food as one of the judgment bases, and judging through the trained foreign object detection model can avoid a large missed detection rate when detecting foreign objects in food by a single image.

Description

Technical Field

[0001] The present invention belongs to the field of food detection, involves deep learning technology, and specifically is a food foreign object detection method and system based on deep learning. Background Art

[0002] With the continuous improvement of living standards, various countries have correspondingly formulated strict safety detection regulations and laws. Therefore, the detection of foreign objects in food is a key link to ensure product quality and consumer safety. The sources of foreign objects in food generally enter through poor management in the production workshop, or the mixing of production supplies, or the mixing in raw materials / primary packaging materials, and other foreign objects. These foreign objects will have a serious negative impact on the quality and taste of the product, reduce consumers' trust in the brand, also reduce their overall eating experience, and pose certain risks to consumers' health.

[0003] A Chinese patent discloses a method for detecting foreign objects in food (Publication No.: CN100472206C). The method includes: collecting X-ray images of the food to be detected; performing contrast enhancement processing; calculating the number of peaks in the row gray distribution curve; obtaining the structure width; obtaining a foreign-object-free X-ray image simulating the food to be detected; and performing an image subtraction operation step to obtain a foreign object target image in the food. When monitoring foreign objects in food, this patent uses X-ray images and analyzes the foreign object situation therein. However, since the food X-ray images have too low sensitivity for detecting low-density foreign objects, this technology is difficult to meet the current high-quality requirements for food detection. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; for this purpose, the present invention proposes a food foreign object detection method and system based on deep learning, which is used to solve the technical problem that when using X-ray images and analyzing the foreign object situation therein, since the food X-ray images have too low sensitivity for detecting low-density foreign objects, it is difficult to meet the current high-quality requirements for food. The present invention solves the above problems by obtaining RGB images and penetration images containing food, extracting color features from the RGB images, using the color situation of the food as one of the judgment bases, extracting the shape features of the food from the penetration images combined with the RGB images, and using the shape features as one of the judgment bases, and then judging through a trained foreign object detection model.

[0005] To achieve the above object, the first aspect of the present invention provides a food foreign object detection method based on deep learning, including the following steps:

[0006] Obtain a captured image containing food; segment the food in the captured image through edge detection to obtain a food image; wherein, the captured image includes: an RGB image and a penetration image;

[0007] Extract color features from the food image corresponding to the RGB image, and extract shape features from the food images corresponding to the RGB image and the penetration image;

[0008] Input the food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features, and the shape features into the foreign object detection model. The foreign object detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; among them, the foreign object detection model is obtained based on a deep learning model.

[0009] It should be noted that: the penetration image can be an X-ray image;

[0010] By obtaining the RGB image and the X-ray image and jointly performing recognition and analysis, the accuracy in detecting low-density foreign objects can be avoided.

[0011] Preferably, the segmentation of the food in the captured image by edge detection includes:

[0012] Compress the captured image, and extract the food in the compressed food image through an edge detection algorithm to obtain a food area. Then, align the compressed food image, and extract the area covered by the food area in the food image, so as to obtain an RGB area and a penetration area;

[0013] Align the RGB area and the penetration area, and extract the overlapping area after alignment in the RGB area and the penetration area, so as to obtain the food image corresponding to the RGB image and the penetration image.

[0014] Preferably, the alignment of the RGB area and the penetration area includes:

[0015] Randomly select n stroke curves from the edges of the RGB area and the penetration area; where n is a positive integer;

[0016] Extract the curve features of the stroke curves on the RGB area and the penetration area respectively, and superimpose the RGB area and the penetration area according to the curve features; where the superimposing condition is to superimpose at the positions where the curve features of the stroke curves on the RGB area are the same as those of the stroke curves on the penetration area.

[0017] It should be noted that when the edge detection algorithm recognizes the edge of an object, it will also recognize the edge of impurities in the food, and the RGB area and the penetration area are respectively the largest enclosed areas in the RGB image and the penetration image;

[0018] Preprocessing the image in the above way can ensure that the calculation amount of the two images does not increase significantly while ensuring the accuracy of target extraction.

[0019] Preferably, the curve feature is a curvature sequence composed of the curvatures of a plurality of points on the stroke curve;

[0020] The obtaining method of the curvature sequence includes:

[0021] Establish a plane rectangular coordinate system, coincide the starting end of the stroke curve with the origin, and calculate the curvature of the corresponding point on the stroke curve according to a plurality of preset abscissas to obtain a curvature sequence; wherein, the starting end of the stroke curve is: one end starting in the clockwise or counterclockwise direction in the stroke curve.

[0022] Preferably, extracting the color feature from the food image corresponding to the RGB image includes:

[0023] Label the RGB color values of each pixel in the food image corresponding to the RGB image, and calculate the brightness mean and standard deviation of each pixel channel; integrate the mean and standard deviation of each pixel channel into a color feature;

[0024] When using such numerical values as color features to detect jelly and transparent beverages, once foreign objects with different colors appear, it is easy to cause large fluctuations in the brightness mean of each pixel channel, thereby improving the detection accuracy.

[0025] Preferably, extracting the shape feature from the food images corresponding to the RGB image and the penetration image includes:

[0026] Map the stroke curves of the RGB region and the penetration region to the complex plane, encode the stroke curves of the RGB region and the penetration region into frequency domain coefficients through fast Fourier transform, and mark the first M low-frequency coefficients as shape features.

[0027] Preferably, the obtaining method of the foreign object detection model includes:

[0028] Construct a detection model based on a deep neural network and introduce an attention mechanism, train the detection model with a training data set, and test the trained detection model with a test data set. When the test accuracy reaches the preset accuracy, obtain a foreign object detection model; wherein, both the training data set and the test data set are composed of input data and output data, the input data includes food images, food images corresponding to penetration images, color features and shape features, and the output data is a detection label.

[0029] It should be noted that the detection labels include abnormal labels and normal labels, which can be replaced by the number 0 and the number 1 respectively.

[0030] It further includes: obtaining the color features and shape features of normal objects in the food, and inputting the RGB image, the penetration image, the color features, and the shape features into the foreign object detection model; wherein, the normal objects are features that do not affect the edibility of the food; for example, air bubbles in jelly, tea leaves in tea beverages, etc.

[0031] The present invention also provides a food foreign object detection system based on deep learning, including an image acquisition module and a foreign object analysis module connected thereto;

[0032] The image acquisition module: is used to acquire the RGB image and the penetration image of the food;

[0033] The foreign object analysis module: is used to segment the food in the RGB image and the penetration image to obtain the corresponding food images; extract the color features and shape features from the food images corresponding to the RGB image and the penetration image; and,

[0034] Input the food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features, and the shape features into the foreign object detection model, the foreign object detection model outputs a detection label, and match the detection result of the corresponding food according to the detection label; wherein, the foreign object detection model is obtained based on a deep learning model.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] (1) In the present invention, the RGB image and the penetration image containing the food are respectively obtained, the color features are extracted from the RGB image, the color situation in the food is used as one of the judgment bases, and the shape features of the food are extracted from the penetration image in combination with the RGB image, and the shape features are used as one of the judgment bases. Then, the trained foreign object detection model is used to judge it, which can avoid a large missed detection rate when detecting foreign objects in the food with a single image.

[0037] (2) In the present invention, when performing segmentation processing on the RGB image and the penetration image, first, the contour of the food is identified by an edge detection algorithm, and the RGB region corresponding to the RGB image and the penetration region corresponding to the penetration image are selected again in an overlapping manner to obtain the position of the corresponding food in the image. This method can not only ensure the accuracy of the selection, but also does not require the use of a high-precision detection algorithm when using the edge detection algorithm. Compared with the prior art, by improving the relevant parameters of the operator (such as adjusting the high and low thresholds of the Canny operator or the kernel size of the Sobel operator, etc.), while ensuring its accuracy, the processing computing power for the RGB image and the penetration image can be reduced. Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0039] Figure 1 Schematic flow chart of the food foreign object detection method of the present invention;

[0040] Figure 2 Schematic flow chart for segmenting the food in the captured image of the present invention;

[0041] Figure 3 Schematic flow chart for aligning the RGB region and the penetration region in the present invention. Detailed implementation manners

[0042] The following will clearly and completely describe the technical solutions of the present invention in combination with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0043] Please refer to Figures 1 - 3 , the first aspect embodiment of the present invention provides a food foreign object detection method based on deep learning, including the following steps:

[0044] Obtain a captured image containing food; segment the food in the captured image through edge detection to obtain a food image; wherein, the captured image includes: an RGB image and an X-ray image;

[0045] Extract color features from the food image corresponding to the RGB image, and extract shape features from the food images corresponding to the RGB image and the penetration image;

[0046] Input the food image corresponding to the RGB image, the food image corresponding to the X-ray image, the color features, and the shape features into the foreign object detection model. The foreign object detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label.

[0047] In this embodiment, segmenting the food in the captured image through edge detection includes:

[0048] Compress the RGB image, and extract the food in the compressed food image through an edge detection algorithm to obtain the food region corresponding to the RGB image;

[0049] Compress the X-ray image, and extract the food in the compressed food image through an edge detection algorithm to obtain the food area corresponding to the X-ray image;

[0050] Align the food area corresponding to the compressed RGB image with the food area corresponding to the X-ray image, extract the area covered by the food area corresponding to the RGB image in the food image to obtain the RGB area; extract the area covered by the food area corresponding to the X-ray image in the food image to obtain the penetration area;

[0051] It should be noted that in order to improve the accuracy of covering the food during superposition, when extracting the RGB area and the penetration area here, different object detection algorithms can be used. Through different object detection algorithms, it is also possible to avoid the situation of more covered areas in the image caused by the defects of a single technology, resulting in an increase in computing power.

[0052] Align the RGB area and the penetration area, and extract the overlapping area after alignment in the RGB area and the penetration area, so as to obtain the food image corresponding to the RGB image and the X-ray image.

[0053] By compressing the image, the amount of data to be processed is reduced, the consumption of computing resources is reduced, and the main outline of the food is retained, so that the computing power requirement can be further reduced. The specific compression size can be set according to the computing power.

[0054] In this embodiment, the food to be detected is a semi-transparent hemispherical jelly. Aligning the RGB area and the penetration area includes:

[0055] Divide the edges of the RGB area and the penetration area into several stroke curves at equal intervals, and randomly select 9 stroke curves with the same relative positions;

[0056] Exemplarily, the method of selecting 9 stroke curves with the same relative positions is as follows: use the highest point or the lowest point on the edge as the reference point and extend a specified length on both sides of the edge on the edge to obtain a stroke curve, and take 8 stroke curves at equal intervals along the edge;

[0057] It should be noted that two stroke curves can also be determined with the highest point and the lowest point as the reference points at the same time, and the remaining stroke curves are taken on the edge at equal intervals.

[0058] Extract the curve features of the stroke curves on the RGB area and the penetration area respectively, and superimpose the RGB area and the penetration area according to the curve features; among them, the superimposing condition is to superimpose the positions where the curve features of the stroke curves on the RGB area are the same as the curve features of the stroke curves on the penetration area.

[0059] It should be noted that the RGB region and the penetration region are the largest closed regions in the RGB image and the penetration image respectively, to avoid identifying only part of the food due to other texture anomalies such as internal stratification in the food.

[0060] The curve feature in this embodiment is a curvature sequence composed of the curvatures of several points on the stroked curve;

[0061] The acquisition method of the above curvature sequence includes:

[0062] Establish a plane rectangular coordinate system, coincide the starting end of the stroked curve with the origin, and calculate the curvature of the corresponding points on the stroked curve according to a preset plurality of abscissas to obtain a curvature sequence; wherein, the starting end of the stroked curve is: one end starting in the clockwise or counterclockwise direction in the stroked curve;

[0063] In this embodiment, one pixel is used as a coordinate unit, and the calculation method of the curvature of each point is K = (dy / dx²) / (1+(dy / dx)²)^(3 / 2), and the curvature calculation method is a prior art and will not be introduced in detail here;

[0064] Since errors often occur during edge detection, and using a high-precision edge detection algorithm leads to a large amount of calculation in its calculation process. By performing edge detection on the content of the captured image separately and using the overlapping part as the final food image, this method can ensure the accuracy of food extraction and reduce its calculation amount at the same time.

[0065] In this embodiment, extracting color features from the food image corresponding to the RGB image includes:

[0066] Label the RGB color values of each pixel in the food image corresponding to the RGB image, and calculate the brightness mean (μR, μG, μB,) and standard deviation (σR, σG, σB) of each pixel channel; integrate the mean and standard deviation of each pixel channel into color features;

[0067] Among them, ; where represents the brightness of the red pixel channel, N is the total number of pixels, and Ri is the R value of the i-th pixel; similarly, it can be known that , Gi is the G value of the i-th pixel, , Bi is the B value of the i-th pixel; specifically, , and σG, σB are similarly known and will not be elaborated here.

[0068] When detecting foreign objects in jelly and translucent beverages through the above content, it is possible to quantify the color distribution differences and be sensitive to the color shift of low-density foreign objects.

[0069] In another preferred embodiment, the food image corresponding to the RGB image is converted to the Lab space to extract the statistics of brightness L, red-green contrast a, and blue-yellow contrast b. The statistics of brightness L, red-green contrast a, and blue-yellow contrast b, together with the brightness mean and standard deviation of each pixel channel, are used as color features.

[0070] In this embodiment, shape features are extracted from the food images corresponding to the RGB image and the penetration image, including:

[0071] The stroke curves of the RGB region and the penetration region are mapped to the complex plane in the form of a discrete point set The specific form is , and the stroke curves of the RGB region and the penetration region are encoded as frequency domain coefficients through the fast Fourier transform (FFT); where H is the total number of discrete points on the stroke curve; the specific frequency domain coefficients: , n = 0, 1,..., H - 1, where F(n) is the nth frequency domain coefficient, is the complex exponential basis function, is the complex coordinate of the kth contour point; the first M low-frequency coefficients among all frequency domain coefficients are marked as shape features.

[0072] The method for obtaining the foreign object detection model in this embodiment includes:

[0073] A detection model is constructed based on a deep neural network and an attention mechanism is introduced. The detection model is trained with a training dataset and the trained detection model is tested with a test dataset. The test accuracy calculation formula: PR = TP / (TP + FP); where PR represents the test accuracy of the trained detection model, TP represents the amount of sample data correctly detected, and FP represents the amount of sample data wrongly detected; when the test accuracy reaches the preset accuracy, a foreign object detection model is obtained; when the test accuracy does not reach the predicted accuracy, the detection model is optimized and adjusted, and then retrained and tested; where the training dataset and the test dataset are both composed of input data and output data. The input data includes food images, food images corresponding to penetration images, color features, and shape features, and the output data is the detection label.

[0074] As another preferred embodiment, in order to avoid the influence of normal substances such as fruit grains and pulp contained in jelly on the detection results, this embodiment further includes: obtaining the color features and shape features of normal objects in food, and inputting the RGB image, penetration image, color features and shape features into the foreign object detection model; wherein, the normal object is a feature that does not affect the edibility of food; for example, air bubbles and pulp in jelly, tea leaves in tea beverages, fruit grain precipitates in fruit juices, etc.

[0075] Taking the color features and shape features of normal objects as input data as well, when training through multiple hidden layers in a deep neural network, the misjudgment probability of the foreign object detection model for normal objects can be reduced, and thus its detection accuracy can be further improved. It should be noted that when training the foreign object detection model at this time, the color features and shape features of normal objects also need to be added to the input data.

[0076] The present invention also provides a food foreign object detection system based on deep learning, including an image acquisition module and a foreign object analysis module connected thereto;

[0077] The image acquisition module: is used to acquire the RGB image and penetration image of food;

[0078] The foreign object analysis module: is used to segment the food in the RGB image and penetration image to obtain the corresponding food image; extract the color features and shape features from the food images corresponding to the RGB image and penetration image; and,

[0079] Input the food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features and shape features into the foreign object detection model, and the foreign object detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; wherein, the foreign object detection model is obtained based on a deep learning model.

[0080] Some of the data in the above formula are calculated by removing the dimension and taking their numerical values. The formula is obtained by software simulation of a large amount of collected data to get a formula closest to the real situation; the preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0081] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A food foreign object detection method based on deep learning, characterized in that, It includes the following steps: Obtain a captured image containing food; Segment the food in the captured image through edge detection to obtain a food image; wherein, the captured image includes: an RGB image and a penetration image; Extract color features from the food image corresponding to the RGB image, and extract shape features from the food images corresponding to the RGB image and the penetration image; Input the food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features, and the shape features into a foreign object detection model. The foreign object detection model outputs a detection label, and match the detection result of the corresponding food according to the detection label; wherein, the foreign object detection model is obtained based on a deep learning model; Segment the food in the captured image through edge detection, including: Compress the RGB image, and extract the food in the compressed food image through an edge detection algorithm to obtain the food region corresponding to the RGB image; Compress the penetration image, and extract the food in the compressed food image through an edge detection algorithm to obtain the food region corresponding to the penetration image; Align the food region corresponding to the compressed RGB image with the food region corresponding to the penetration image, extract the region covered by the food region corresponding to the RGB image in the food image to obtain the RGB region; extract the region covered by the food region corresponding to the penetration image in the food image to obtain the penetration region; Randomly select n stroke curves from the edges of both the RGB region and the penetration region; where n is a positive integer; Extract curve features from the stroke curves on the RGB region and the penetration region respectively, and superimpose the RGB region and the penetration region according to the curve features; wherein, the superimposing condition is to superimpose at the positions where the curve features of the stroke curves on the RGB region are the same as those of the stroke curves on the penetration region; Extract the overlapping region after alignment in the RGB region and the penetration region, so as to obtain the food image corresponding to the RGB image and the penetration image; The curve feature is a curvature sequence composed of the curvatures of several points on the stroke curve.

2. The method for detecting foreign objects in food based on deep learning according to claim 1, wherein, The obtaining method of the curvature sequence includes: Establish a plane rectangular coordinate system, coincide the starting end of the stroke curve with the origin, and calculate the curvature of the corresponding points on the stroke curve according to a plurality of preset abscissas to obtain a curvature sequence; wherein, the starting end of the stroke curve is: one end starting in the clockwise or counterclockwise direction in the stroke curve.

3. The food foreign object detection method based on deep learning according to claim 1, characterized in that The extracting the color features from the food image corresponding to the RGB image includes: Label the RGB color values of each pixel in the food image corresponding to the RGB image, and calculate the brightness mean and standard deviation of each pixel channel; integrate the brightness mean and standard deviation of each pixel channel into color features.

4. A method for detecting foreign objects in food based on deep learning according to claim 1, characterized in that, The extracting the shape features from the food images corresponding to the RGB image and the penetration image includes: Map the stroke curves of the RGB region and the penetration region to the complex plane, encode the stroke curves of the RGB region and the penetration region into frequency domain coefficients through fast Fourier transform, and mark the first M low-frequency coefficients as shape features.

5. A food foreign object detection method based on deep learning according to claim 1, characterized in that The obtaining method of the foreign object detection model includes: A detection model is constructed based on a deep neural network and by introducing an attention mechanism. The detection model is trained using a training dataset and then tested using a test dataset. When the test accuracy reaches the preset accuracy, a foreign object detection model is obtained. Among them, both the training dataset and the test dataset are composed of input data and output data. The input data includes food images, food images corresponding to penetration images, color features, and shape features, and the output data is detection labels.

6. The method for detecting foreign objects in food based on deep learning according to claim 1, wherein It further includes: Obtaining the color features and shape features of normal objects in food, and inputting the RGB image, penetration image, color features, and shape features into the foreign object detection model. Among them, normal objects are features that do not affect the edibility of food.

7. A food foreign object detection system based on deep learning, which is applied to a food foreign object detection method based on deep learning according to any one of claims 1-6, characterized in that, It includes an image acquisition module and a foreign object analysis module connected thereto; The image acquisition module: is used to acquire the RGB image and penetration image of food; The foreign object analysis module: is used to segment the food in the RGB image and penetration image to obtain the corresponding food images; extract the color features and shape features from the food images corresponding to the RGB image and penetration image; And, Inputting the food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features, and the shape features into the foreign object detection model. The foreign object detection model outputs detection labels, and the detection results of the corresponding food are matched according to the detection labels. Among them, the foreign object detection model is obtained based on a deep learning model.

Images