Deep learning-based food foreign matter detection method and system

Abstract

The invention discloses a food foreign matter detection method and system based on deep learning. The system comprises an image acquisition module and a foreign matter analysis module connected with the image acquisition module. The image acquisition module is used for acquiring an RGB image and a penetration image of food; and the foreign matter analysis module is used for inputting a food image corresponding to the RGB image, a food image corresponding to the penetration image, color features and shape features into a foreign matter detection model. As the sensitivity of the food X-ray image for detecting low-density foreign matters is too low, the current high-quality requirement on the food is difficult to identify and conform. According to the method, the color condition in the food is used as one of judgment bases, the shape feature of the food is used as one of judgment bases, and the situation that when the foreign matter in the food is detected through a single image, the large omission ratio occurs can be avoided by judging the food through the trained foreign matter detection model.

Description

Technical Field

[0001] The present invention belongs to the field of food detection and relates to deep learning technology, and specifically to a food foreign body detection method and system based on deep learning. Background Art

[0002] With the continuous improvement of living standards, various countries have also formulated strict safety testing regulations and laws. Therefore, foreign matter detection in food is a key link in ensuring product quality and consumer safety. Foreign matter in food generally comes from poor management of the production workshop, production supplies, raw materials / primary packaging materials and other foreign matter. These foreign matter will have a serious negative impact on the quality and taste of the product, reduce consumers' trust in the brand, reduce their overall eating experience, and pose certain risks to consumers' health.

[0003] A Chinese patent discloses a method for detecting foreign matter in food (publication number: CN100472206C), which includes: collecting X-ray images of the inspected food; performing contrast enhancement processing; calculating the number of peaks in the grayscale distribution curve; obtaining the structure width; obtaining a simulated X-ray image of the inspected food without foreign matter; performing an image subtraction operation step to obtain a target image of foreign matter in the food. This patent uses X-ray images to monitor foreign matter in food and analyzes the foreign matter therein, but because the sensitivity of food X-ray images for low-density foreign matter detection is too low, this technology is difficult to meet the current high-quality requirements for food detection. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a food foreign matter detection method and system based on deep learning, which is used to solve the technical problem that when using X-ray images and analyzing the foreign matter situation therein, the sensitivity of food X-ray images for low-density foreign matter detection is too low, which makes it difficult to identify the foreign matter that meets the current high-quality requirements for food. The present invention obtains RGB images and penetration images containing food, extracts color features from the RGB images, uses the color situation in the food as one of the judgment bases, and extracts the shape features of the food from the penetration image combined with the RGB image, and uses the shape features as one of the judgment bases, and then judges it through a trained foreign matter detection model to solve the above-mentioned problem.

[0005] To achieve the above object, the first aspect of the present invention provides a food foreign body detection method based on deep learning, comprising the following steps: Acquire a captured image containing food; segment the food in the captured image by edge detection to obtain a food image; wherein the captured image includes: an RGB image and a penetration image; Extract color features from the food image corresponding to the RGB image, and extract shape features from the food image corresponding to the RGB image and the penetration image; The food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features and the shape features are input into the foreign body detection model. The foreign body detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; wherein the foreign body detection model is obtained based on a deep learning model.

[0006] It should be noted that: the penetration image may be an X-ray image; By acquiring RGB images and X-ray images for joint recognition and analysis, it is possible to avoid the accuracy of detecting low-density foreign objects.

[0007] Preferably, segmenting the food in the captured image by edge detection includes: Compress the captured image, extract the food in the compressed food image through an edge detection algorithm to obtain the food area, align the compressed food image, extract the area covered by the food area in the food image, and thus obtain the RGB area and the penetration area; The RGB area and the penetration area are aligned, and the aligned overlapping area between the RGB area and the penetration area is extracted, so as to obtain the food image corresponding to the RGB image and the penetration image.

[0008] Preferably, aligning the RGB area and the penetration area includes: Randomly select n stroke curves from the edges of the RGB area and the penetration area; where n is a positive integer; Curve features are extracted for the stroke curves on the RGB area and the penetration area respectively, and the RGB area is overlapped with the penetration area according to the curve features; wherein the overlap condition is that the curve features of the stroke curve on the RGB area are overlapped at the same position as the curve features of the stroke curve on the penetration area.

[0009] It should be noted that when the edge detection algorithm identifies the edge of its object, it will also identify the edge of impurities in the food, and the RGB area and penetration area are the largest closed areas in the RGB image and the penetration image respectively; Preprocessing the images in the above manner can ensure that the amount of calculation for the two images is not significantly increased while ensuring the accuracy of target extraction.

[0010] Preferably, the curve feature is a curvature sequence composed of the curvatures of a plurality of points on the stroke curve; The method for obtaining the curvature sequence includes: A plane rectangular coordinate system is established, the starting end of the stroked curve is coincident with the origin, and the curvature of the corresponding point on the stroked curve is calculated according to a plurality of preset horizontal coordinates to obtain a curvature sequence; wherein the starting end of the stroked curve is: an end in the stroked curve that starts in a clockwise or counterclockwise direction.

[0011] Preferably, extracting color features from the food image corresponding to the RGB image includes: Label the RGB color value of each pixel in the food image corresponding to the RGB image, calculate the brightness mean and standard deviation of each pixel channel; integrate the mean and standard deviation of each pixel channel into color features; When using this type of value as a color feature to detect jelly and transparent beverages, once foreign objects of different colors appear, it is easy to cause a large fluctuation in the mean brightness of each pixel channel, thereby improving the accuracy of its detection.

[0012] Preferably, extracting shape features from the food image corresponding to the RGB image and the penetration image includes: The stroke curves of the RGB area and the penetration area are mapped to the complex plane, and the stroke curves of the RGB area and the penetration area are encoded into frequency domain coefficients through fast Fourier transform, and the first M low-frequency coefficients are marked as shape features.

[0013] Preferably, the method for acquiring the foreign body detection model includes: A detection model is constructed based on a deep neural network and an attention mechanism is introduced. The detection model is trained through a training data set, and the trained detection model is tested through a test data set. When the test accuracy reaches a preset accuracy, a foreign body detection model is obtained; wherein, the training data set and the test data set are both composed of input data and output data, the input data includes food images, food images corresponding to the penetration images, color features and shape features, and the output data is a detection label.

[0014] It should be noted that the detection labels include abnormal labels and normal labels, which can be replaced by the number 0 and the number 1 respectively.

[0015] It also includes: obtaining color features and shape features of normal objects in food, and inputting RGB images, penetration images, color features and shape features into a foreign body detection model; wherein normal objects are features that do not affect the consumption of food; for example, bubbles in jelly, tea leaves in tea beverages, etc.

[0016] The present invention also provides a food foreign body detection system based on deep learning, comprising an image acquisition module and a foreign body analysis module connected thereto; The image acquisition module is used to acquire RGB images and penetration images of food; The foreign body analysis module is used to segment the food in the RGB image and the penetration image to obtain the corresponding food image; extract color features and shape features from the food image corresponding to the RGB image and the penetration image; and The food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features and the shape features are input into the foreign body detection model. The foreign body detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; wherein the foreign body detection model is obtained based on a deep learning model.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) In the present invention, an RGB image and a penetration image containing food are obtained respectively, color features are extracted from the RGB image, and the color situation in the food is used as one of the judgment bases. The shape features of the food are extracted from the penetration image combined with the RGB image, and the shape features are used as one of the judgment bases. The judgment is then made using a trained foreign matter detection model, which can avoid a large missed detection rate when detecting foreign matter in food with a single image.

[0018] (2) In the present invention, when segmenting the RGB image and the penetration image, the outline of the food is first identified by an edge detection algorithm, and the RGB area corresponding to the RGB image and the penetration area corresponding to the penetration image are overlapped to select the position of the corresponding food in the image again. This method can not only ensure the accuracy of the selection, but also can use the edge detection algorithm without the need to use a high-precision detection algorithm. Compared with the prior art, by improving operator-related parameters (such as adjusting the high and low thresholds of the Canny operator or the kernel size of the Sobel operator, etc.), it can reduce the processing power of the RGB image and the penetration image respectively while ensuring their accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 Schematic diagram of the process of the food foreign body detection method of the present invention; Figure 2 A schematic diagram of the process of segmenting food in a captured image according to the present invention; Figure 3 It is a schematic diagram of the process of aligning the RGB area and the penetration area in the present invention. DETAILED DESCRIPTION

[0021] The technical scheme of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1-Figure 3 The first aspect of the present invention provides a method for detecting foreign matter in food based on deep learning, comprising the following steps: Acquire a photographed image containing food; segment the food in the photographed image by edge detection to obtain a food image; wherein the photographed image includes: an RGB image and an X-ray image; Extract color features from the food image corresponding to the RGB image, and extract shape features from the food image corresponding to the RGB image and the penetration image; The food image corresponding to the RGB image, the food image corresponding to the X-ray image, the color features and the shape features are input into the foreign body detection model. The foreign body detection model outputs the detection label, and the detection result of the corresponding food is matched according to the detection label.

[0023] In this embodiment, segmenting the food in the captured image by edge detection includes: Compress the RGB image, and extract the food in the compressed food image by using an edge detection algorithm to obtain the food area corresponding to the RGB image; Compressing the X-ray image, and extracting the food in the compressed food image by using an edge detection algorithm to obtain the food area corresponding to the X-ray image; Align the food area corresponding to the compressed RGB image with the food area corresponding to the X-ray image, extract the area covered by the food area corresponding to the RGB image in the food image to obtain the RGB area; extract the area covered by the food area corresponding to the X-ray image in the food image to obtain the penetration area; It should be noted that in order to improve the accuracy of food coverage during superposition, different target detection algorithms can be used when extracting the RGB area and the penetration area. Different target detection algorithms can also avoid the situation where too many areas in the image are covered due to the defects of a single technology, resulting in increased computing power.

[0024] The RGB area and the penetration area are aligned, and the aligned overlapping area between the RGB area and the penetration area is extracted, so as to obtain the food image corresponding to the RGB image and the X-ray image.

[0025] By compressing the image, the amount of data to be processed is reduced, the consumption of computing resources is reduced, and the outline of the main body of the food is retained, thereby further reducing the computing power requirements. The specific compression size can be set according to the computing power. In this embodiment, the food to be detected is a semi-transparent hemispherical jelly, and the RGB area and the penetration area are aligned, including: The edges of the RGB area and the penetration area are divided into a number of stroke curves at equal intervals, and nine stroke curves with the same relative position are randomly selected; Exemplarily, the method of selecting 9 stroke curves with the same relative position is as follows: taking the highest point or the lowest point on the edge as the reference point and extending the edge to both sides by a specified length to obtain a stroke curve, and taking 8 stroke curves at equal intervals along the edge; It should be noted that two stroke curves may be determined with the highest point and the lowest point as reference points at the same time, and then the remaining stroke curves may be taken on the edge in an equidistant manner.

[0026] Curve features are extracted for the stroke curves on the RGB area and the penetration area respectively, and the RGB area is overlapped with the penetration area according to the curve features; wherein the overlap condition is that the curve features of the stroke curve on the RGB area are overlapped at the same position as the curve features of the stroke curve on the penetration area.

[0027] It should be noted that the RGB area and the penetration area are the largest closed areas in the RGB image and the penetration image, respectively, to avoid the internal stratification phenomenon and other texture anomalies in the food, which may lead to the recognition of only part of the food.

[0028] The curve feature in this embodiment is a curvature sequence composed of the curvatures of several points on the stroke curve; The above-mentioned curvature sequence is obtained by: Establish a plane rectangular coordinate system, make the starting point of the stroke curve coincide with the origin, and calculate the curvature of the corresponding point on the stroke curve according to a plurality of preset horizontal coordinates to obtain a curvature sequence; wherein the starting point of the stroke curve is: an end of the stroke curve that starts in a clockwise or counterclockwise direction; In this embodiment, one pixel is used as a coordinate unit, and the curvature of each point is calculated as K= (dy / dx²) / (1+(dy / dx)²)^(3 / 2), and the curvature calculation method is the existing technology and will not be introduced in detail here; Since errors often occur during edge detection, and the use of high-precision edge detection algorithms leads to a large amount of calculation in the calculation process, this method can ensure the accuracy of food extraction while reducing the amount of calculation by performing edge detection on the contents of the captured image separately and using the overlapping parts as the final food image.

[0029] In this embodiment, color features are extracted from the food image corresponding to the RGB image, including: Label the RGB color value of each pixel in the food image corresponding to the RGB image, calculate the brightness mean (μR, μG, μB,) and standard deviation (σR, σG, σB) of each pixel channel; integrate the mean and standard deviation of each pixel channel into color features; in, ;in Represents the brightness of the red pixel channel, N is the total number of pixels, and Ri is the R value of the i-th pixel; similarly, , Gi is the G value of the i-th pixel, , Bi is the B value of the i-th pixel; specifically, , and σG and σB can be understood in the same way and will not be elaborated on here.

[0030] Through the above content, when detecting foreign matter in jellies and translucent beverages, it is possible to quantify the color distribution difference and be sensitive to the color shift of low-density foreign matter.

[0031] In another preferred embodiment, the food image corresponding to the RGB image is converted into Lab space to extract statistics of brightness L, red-green contrast a, and blue-yellow contrast b, and the statistics of brightness L, red-green contrast a, and blue-yellow contrast b are used together with the brightness mean and standard deviation of each pixel channel as color features.

[0032] In this embodiment, shape features are extracted from the food image corresponding to the RGB image and the penetration image, including: The stroke curves of the RGB area and the penetration area are divided into discrete point sets The form of mapping to the complex plane is expressed as , and encode the stroke curves of the RGB area and the penetration area into frequency domain coefficients through fast Fourier transform (FFT); where H is the total number of discrete points on the stroke curve; the specific frequency domain coefficients are: , n=0,1,…,H-1, where F(n) is the nth frequency domain coefficient, is the complex exponential basis function, is the complex coordinate of the kth contour point; the first M low-frequency coefficients in all frequency domain coefficients are marked as shape features.

[0033] The method for obtaining the foreign body detection model in this embodiment includes: A detection model is constructed based on a deep neural network and an attention mechanism is introduced. The detection model is trained through a training data set, and the trained detection model is tested through a test data set. The test accuracy calculation formula is: PR=TP / (TP+FP); wherein PR represents the test accuracy of the trained detection model, TP represents the amount of sample data detected correctly, and FP represents the amount of sample data detected incorrectly; when the test accuracy reaches the preset accuracy, a foreign body detection model is obtained; when the test accuracy does not reach the predicted accuracy, the detection model is optimized and adjusted, and retrained and tested; wherein the training data set and the test data set are both composed of input data and output data, the input data includes food images, food images corresponding to penetration images, color features and shape features, and the output data is a detection label.

[0034] As another preferred embodiment, in order to prevent normal substances such as fruit particles and pulp contained in jelly from affecting the detection results, the present embodiment also includes: obtaining color features and shape features of normal objects in food, and inputting RGB images, penetration images, color features and shape features into a foreign body detection model; wherein normal objects are features that do not affect the consumption of food; for example, bubbles and pulp in jelly, tea leaves in tea beverages, fruit particle sediments in juice, etc.; By also taking the color features and shape features of normal objects as input data and training them through multiple hidden layers in a deep neural network, the probability of misjudging normal objects by the foreign object detection model can be reduced, thereby further improving its detection accuracy. It should be noted that the input data for training the foreign object detection model at this time also needs to include the color features and shape features of normal objects.

[0035] The present invention also provides a food foreign body detection system based on deep learning, comprising an image acquisition module and a foreign body analysis module connected thereto; The image acquisition module is used to acquire RGB images and penetration images of food; The foreign body analysis module is used to segment the food in the RGB image and the penetration image to obtain the corresponding food image; extract color features and shape features from the food image corresponding to the RGB image and the penetration image; and The food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features and the shape features are input into the foreign body detection model. The foreign body detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; wherein the foreign body detection model is obtained based on a deep learning model.

[0036] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0037] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A food foreign body detection method based on deep learning, characterized in that: The following steps are involved: Acquire a photographic image containing food; The food in the captured image is segmented by edge detection to obtain a food image; wherein the captured image includes: an RGB image and a penetration image; Extract color features from the food image corresponding to the RGB image, and extract shape features from the food image corresponding to the RGB image and the penetration image; The food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features and the shape features are input into the foreign body detection model. The foreign body detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; wherein the foreign body detection model is obtained based on a deep learning model.

2. A food foreign body detection method based on deep learning according to claim 1, characterized in that: The step of segmenting the food in the captured image by edge detection includes: The food in the compressed food image is extracted by edge detection algorithm to obtain the food area, and then the compressed food image is aligned to extract the area covered by the food area in the food image, so as to obtain the RGB area and the penetration area; The RGB area and the penetration area are aligned, and the aligned overlapping area between the RGB area and the penetration area is extracted, so as to obtain the food image corresponding to the RGB image and the penetration image.

3. The method for detecting foreign matter in food based on deep learning according to claim 2, characterized in that: The aligning of the RGB area and the penetration area includes: Randomly select n stroke curves from the edges of the RGB area and the penetration area; where n is a positive integer; Curve features are extracted for the stroke curves on the RGB area and the penetration area respectively, and the RGB area is overlapped with the penetration area according to the curve features; wherein the overlap condition is that the curve features of the stroke curve on the RGB area are overlapped at the same position as the curve features of the stroke curve on the penetration area.

4. The method for detecting foreign matter in food based on deep learning according to claim 3, characterized in that: The curve feature is a curvature sequence composed of the curvatures of a plurality of points on the stroke curve.

5. The method for detecting foreign matter in food based on deep learning according to claim 4, characterized in that: The method for obtaining the curvature sequence includes: A plane rectangular coordinate system is established, the starting end of the stroked curve is coincident with the origin, and the curvature of the corresponding point on the stroked curve is calculated according to a plurality of preset horizontal coordinates to obtain a curvature sequence; wherein the starting end of the stroked curve is: an end in the stroked curve that starts in a clockwise or counterclockwise direction.

6. The method for detecting foreign matter in food based on deep learning according to claim 4, characterized in that: The step of extracting color features from the food image corresponding to the RGB image includes: The RGB color value of each pixel in the food image corresponding to the RGB image is labeled, and the brightness mean and standard deviation of each pixel channel are calculated; the mean and standard deviation of each pixel channel are integrated into color features.

7. The method for detecting foreign matter in food based on deep learning according to claim 1, characterized in that: The shape features are extracted from the food image corresponding to the RGB image and the penetration image, including: The stroke curves of the RGB area and the penetration area are mapped to the complex plane, and the stroke curves of the RGB area and the penetration area are encoded into frequency domain coefficients through fast Fourier transform, and the first M low-frequency coefficients are marked as shape features.

8. The method for detecting foreign matter in food based on deep learning according to claim 1, characterized in that: The method for obtaining the foreign body detection model includes: A detection model is constructed based on a deep neural network and an attention mechanism is introduced. The detection model is trained through a training data set, and the trained detection model is tested through a test data set. When the test accuracy reaches a preset accuracy, a foreign body detection model is obtained; wherein, the training data set and the test data set are both composed of input data and output data, the input data includes food images, food images corresponding to penetration images, color features and shape features, and the output data is a detection label.

9. The method for detecting foreign matter in food based on deep learning according to claim 1, characterized in that: Also includes: The color features and shape features of normal objects in food are obtained, and the RGB image, penetration image, color features and shape features are input into the foreign body detection model; among which, normal objects are features that do not affect the consumption of food.

10. A food foreign body detection system based on deep learning, applied to a food foreign body detection method based on deep learning according to any one of claims 1 to 9, characterized in that: It includes an image acquisition module and a foreign body analysis module connected thereto; The image acquisition module is used to acquire RGB images and penetration images of food; The foreign body analysis module is used to segment the food in the RGB image and the penetration image to obtain the corresponding food image; extract color features and shape features from the food image corresponding to the RGB image and the penetration image; as well as, The food image corresponding to the RGB image, the food image corresponding to the penetration image, the color features and the shape features are input into the foreign body detection model. The foreign body detection model outputs a detection label, and the detection result of the corresponding food is matched according to the detection label; wherein the foreign body detection model is obtained based on a deep learning model.

Images