A food color feature quality inspection system and method based on multi-color space fusion

Abstract

The invention relates to the technical field of food color feature quality inspection, and in particular to a food color feature quality inspection system and method based on multi-color space fusion, comprising a color space selection module, wherein the output end of the color space selection module is electrically connected to an area reconstruction and removal module for removing food packaging reflective light, the output end of the area reconstruction and removal module for removing food packaging reflective light is electrically connected to a multi-color space feature fusion module, and the output end of the multi-color space feature fusion module is electrically connected to a color feature quality inspection module. By combining HSV color space, Lab color space and traditional RGB image recognition technology, components of different color spaces are extracted to perform random fusion of multi-color spaces, so as to obtain richer color feature information, improve the color feature information, and at the same time use a reconstruction algorithm to reduce the influence of illumination changes on color feature extraction, so as to improve the extraction accuracy of color features of bread food.

Description

Technical Field

[0001] The present invention relates to the technical field of food color feature quality inspection, and in particular to a food color feature quality inspection system and method based on multi-color space fusion. Background Art

[0002] In the baking industry, color information is closely related to food quality. However, due to the complexity of environmental anomalies in the food production process, the differences in lighting conditions in different stores, and the continuous updating of new and old foods, it is difficult for existing technologies to achieve ideal results in terms of model recognition speed, new product recognition efficiency, food quality inspection score, quality inspection efficiency, and model accuracy. At present, the image recognition method of the food recognition system is based on the RGB color space, which encodes the color information in the image independently through three channels: red, green, and blue. Although the RGB color space is widely used, it has some significant problems in the process of color feature extraction: sensitivity to light changes leads to distortion of color information, the inability to separate the saturation and brightness of the color, and inaccurate color matching leads to reduced color matching and recognition accuracy. Summary of the invention

[0003] The purpose of the present invention is to provide a food color feature quality inspection system and method based on multi-color space fusion to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions: A food color feature quality inspection system based on multi-color space fusion, comprising a color space selection module, the output end of the color space selection module is electrically connected to an area reconstruction and removal of food packaging reflective light module, the output end of the area reconstruction and removal of food packaging reflective light module is electrically connected to a multi-color space feature fusion module, and the output end of the multi-color space feature fusion module is electrically connected to a color feature quality inspection module.

[0005] A food color feature quality inspection method based on multi-color space fusion, comprising a color space selection module (1), an area reconstruction and food packaging reflection removal module (2), a multi-color space feature fusion module (3) and a color feature quality inspection module (4), wherein the color space selection module (1) comprises an RGB color space, an HSV color space and a Lab color space, and the steps of the RGB color space are: S1.RGB color model represents color, which can simplify the system architecture and design. In order to reduce the massive data brought by real color, CIE proposed to use the addition of the three primary color wavelengths of R, G, and B to achieve the representation, transmission and restoration of natural colors. Other colors can be generated by adding different RGB components. The formula is:

[0006] Where C is the mixed color, r, g, b are the T units used for matching, 1T unit red light is equal to 0.30lm, 1T unit green light is equal to 0.59lm, 1T unit blue light is equal to 0.11lm, The steps of the HSV color space are: S2.HSV color space contains three color channels: H, S, and V. Hue represents a pure color on the color wheel; Saturation represents the percentage of pure color in the color; and Value represents the brightness of the color. The V channel value conforms to the human eye's perception of color. There is a nonlinear relationship between RGB and HSV. Assuming that the three channels of the RGB color space have a value range of [0,255], the calculation steps for converting to the HSV model are as follows: Calculate brightness (V):

[0007] Calculate saturation (S):

[0008] Calculate chromaticity (H):

[0009] There is another simple calculation formula for transformation, as shown below: ; The steps of the Lab color space are: S3. It consists of three elements: illuminance L and a and b related to color. L represents luminosity, which is equivalent to brightness, with a value range of [0,100]. a represents the range from red to green, and b represents the range from blue to yellow. The value ranges of a and b are [+120,-120]. All colors are composed of the interactive changes of these three values. When converting RGB to Lab color space, it is divided into two steps: first convert to XYZ space, and then convert to Lab space. The specific conversion formula is as follows:

[0010]

[0011] in, , X, Y, Z are the three stimulus values ​​of a pixel in the image, , , It is the tristimulus value of the standard illuminant corresponding to the illumination source used to collect the image.

[0012] As a preferred solution of the present invention, the steps of area reconstruction and removing the food packaging reflective module (2) are: S4. Locate the center point of the package reflection, use a 5*5 template to traverse each pixel in the food image, find the center point of the template as the maximum pixel mark of the template, and record it as the candidate center point; the second step is to use Sobel approximation to screen the candidate center points, and add a 3*3 area with e as the center of the candidate center point, as follows:

[0013] Among them, a, b, c, d, e, f, g, h, and i represent the grayscale value of a pixel respectively. There are 4 straight lines passing through the center point e, and each straight line divides the area into 2 parts. Each part has 3 pixels, called sub-areas. The maximum value of d1, d2, d3, and d4 is calculated by the formula and compared with the threshold.

[0014] The image representation in area reconstruction is defined as a morphological opening and closing operation to eliminate the distortion and information loss caused by the morphological opening and closing filter: is the identification diagram, where Perform area opening reconstruction. is the output image; is the identification diagram, where Perform area closed reconstruction, For the output image.

[0015] As a preferred solution of the present invention, the multi-color space feature fusion module (3) comprises the following steps: S5. The original color histogram is changed to the color moment to characterize the color features of the image. Because the color information of food images is mainly concentrated in the low-order moments, the color moments are divided into first-order moments, second-order moments and third-order moments. The first-order moment is used to express the mean information of the image, including the overall overview of the image; the second-order moment is used to express the variance information of the image, including some detailed changes; the third-order moment is used to express the slope information of the image. The following are the three low-order moments of the image color. In formula (3-1), Represents the xth color component of the yth pixel in the food image:

[0016] First, split the original image into 9 sub-images, extract the color features of the color space mixture for each sub-image, add some other channel values ​​in the color space based on the original monochrome technology that only uses grayscale values, and transform one of the sub-images of the image to HSV, Lab, and RGB color spaces respectively. In different color spaces, collect the values ​​of different channels of H, S, V, R, G, B, L, a, and b of the color feature subspaces extracted by HSV, RGB, and Lab of the food image to calculate the corresponding color moments. According to the formula, the corresponding components are calculated to obtain three low-order color moments, and then the corresponding first-order moments are fused to obtain the color space mixed brightness component. By analogy, the corresponding second-order and third-order moments are obtained. In the formula, , , are the weight factors corresponding to the color space, is the corresponding component of the color space, and then the corresponding components of different color spaces are calculated , three color moments of the hue component of the color space are obtained by fusing the order moments corresponding to the two components, and finally 9 color feature vectors of a sub-image fused in the color space are obtained. The color feature vectors of other sub-images are extracted using a similar method, and finally the color feature vectors of the 9 sub-images fused in the color space are obtained, so as to characterize the color characteristics of the food image.

[0017] As a preferred solution of the present invention, the color feature quality inspection module (4) comprises the following steps: S6. Calculate the cosine similarity of the feature vectors in the two feature libraries. The cosine value of the angle between the vectors represents the similarity of the two vectors. The larger the cosine value, the more similar the two vectors are, that is, the more similar the quality inspection food and the standard food images are, and they belong to the same category of food. The formula is:

[0018] In the formula, A and B represent two eigenvectors, and Represent the components of vectors A and B respectively, n represents the dimension of the feature vector, and in this method, n is set to 128. The cosine similarity x between the vector in the quality inspection feature library and the vector in the identification feature library of the food is calculated by the above method, and then combined with the weight factor of the color feature in the food , perform quality inspection score calculation, and improve the system mathematical formula design as follows:

[0019] In the formula, Represents the color weight factor of the food, x represents the cosine similarity result, y is a constant variable, which can be flexibly adjusted in the system according to business needs. The value range is [0, 1]. In this method, the value of y is 0.5.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. In the present invention, by combining HSV color space, Lab color space and traditional RGB image recognition technology, components of different color spaces are extracted to perform random fusion of multiple color spaces to obtain richer color feature information and improve the color feature information. At the same time, a reconstruction algorithm is used to reduce the impact of illumination changes on color feature extraction, thereby improving the extraction accuracy of color features of bread foods. The color quality inspection score is calculated through more comprehensive color feature information combined with similarity calculation results, and the original low score is improved to a higher score, thereby meeting the demand for low quality inspection scores for store business.

[0021] 2. In the present invention, by comprehensively considering the characteristics of hue, saturation and brightness, the system can still stably identify under different lighting conditions, improve the color matching effect, and apply this technology to the bread food recognition system and promote it to other baked foods and different types of food quality inspection and classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a system block diagram of the present invention; Figure 2 is a schematic diagram of a first point control of the present invention; Figure 3 This is a schematic diagram of the first point control of the present invention.

[0023] In the figure: 1. Color space selection module; 2. Area reconstruction to remove food packaging reflection module; 3. Multi-color space feature fusion module; 4. Color feature quality inspection module. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0025] For examples, see Figure 1-Figure 3 , the present invention provides a technical solution: A food color feature quality inspection system based on multi-color space fusion, comprising a color space selection module 1, the output end of the color space selection module 1 is electrically connected to an area reconstruction and removal of food packaging reflective light module 2, the output end of the area reconstruction and removal of food packaging reflective light module 2 is electrically connected to a multi-color space feature fusion module 3, and the output end of the multi-color space feature fusion module 3 is electrically connected to a color feature quality inspection module 4.

[0026] A food color feature quality inspection method based on multi-color space fusion, comprising a color space selection module (1), an area reconstruction and food packaging reflection removal module (2), a multi-color space feature fusion module (3) and a color feature quality inspection module (4), wherein the color space selection module (1) comprises an RGB color space, an HSV color space and a Lab color space, and the steps of the RGB color space are: S1.RGB color model represents color, which can simplify the system architecture and design. In order to reduce the massive data brought by real color, CIE proposed to use the addition of the three primary color wavelengths of R, G, and B to achieve the representation, transmission and restoration of natural colors. Other colors can be generated by adding different RGB components. The formula is:

[0027] Where C is the mixed color, r, g, b are the T units used for matching, 1T unit red light is equal to 0.30lm, 1T unit green light is equal to 0.59lm, 1T unit blue light is equal to 0.11lm, The steps of HSV color space are: S2.HSV color space contains three color channels: H, S, and V. Hue represents a pure color on the color wheel; Saturation represents the percentage of pure color in the color; and Value represents the brightness of the color. The V channel value conforms to the human eye's perception of color. There is a nonlinear relationship between RGB and HSV. Assuming that the three channels of the RGB color space have a value range of [0,255], the calculation steps for converting to the HSV model are as follows: Calculate brightness (V):

[0028] Calculate saturation (S):

[0029] Calculate chromaticity (H):

[0030] There is another simple calculation formula for transformation, as shown below: ; The steps of Lab color space are: S3. It consists of three elements: illuminance L and a and b related to color. L represents luminosity, which is equivalent to brightness, with a value range of [0,100]. a represents the range from red to green, and b represents the range from blue to yellow. The value ranges of a and b are [+120,-120]. All colors are composed of the interactive changes of these three values. When converting RGB to Lab color space, it is divided into two steps: first convert to XYZ space, and then convert to Lab space. The specific conversion formula is as follows:

[0031]

[0032] in, , X, Y, Z are the three stimulus values ​​of a pixel in the image, , , It is the tristimulus value of the standard illuminant corresponding to the illumination source used to collect the image.

[0033] Among them, by combining the HSV color space, Lab color space and traditional RGB image recognition technology, components of different color spaces are extracted for random fusion of multiple color spaces to obtain richer color feature information and improve the color feature information. At the same time, the reconstruction algorithm is used to reduce the impact of light changes on color feature extraction, and improve the extraction accuracy of color features of bread foods. Through more comprehensive color feature information, the color quality inspection score is calculated in combination with the similarity calculation results, and the original low score is improved to a higher score, meeting the demand for low quality inspection scores for store business.

[0034] The steps of area reconstruction to remove the food packaging reflective module (2) are as follows: S4. Locate the center point of the package reflection, use a 5*5 template to traverse each pixel in the food image, find the center point of the template as the maximum pixel mark of the template, and record it as the candidate center point; the second step is to use Sobel approximation to screen the candidate center points, and add a 3*3 area with e as the center of the candidate center point, as follows:

[0035] Among them, a, b, c, d, e, f, g, h, and i represent the grayscale value of a pixel respectively. There are 4 straight lines passing through the center point e, and each straight line divides the area into 2 parts. Each part has 3 pixels, called sub-areas. The maximum value of d1, d2, d3, and d4 is calculated by the formula and compared with the threshold.

[0036] The image representation in area reconstruction is defined as a morphological opening and closing operation to eliminate the distortion and information loss caused by the morphological opening and closing filter: is the identification diagram, where Perform area opening reconstruction. is the output image; is the identification diagram, where Perform area closed reconstruction, For the output image.

[0037] Among them, by comprehensively considering characteristics such as hue, saturation and brightness, the system can still perform stably recognition under different lighting conditions, improving the color matching effect. This technology is applied to the bread food recognition system and extended to other baked foods and different types of food quality inspection and classification tasks.

[0038] The multi-color space feature fusion module (3) includes the following steps: S5. The original color histogram is changed to the color moment to characterize the color features of the image. Because the color information of food images is mainly concentrated in the low-order moments, the color moments are divided into first-order moments, second-order moments and third-order moments. The first-order moment is used to express the mean information of the image, including the overall overview of the image; the second-order moment is used to express the variance information of the image, including some detailed changes; the third-order moment is used to express the slope information of the image. The following are the three low-order moments of the image color. In formula (3-1), Represents the xth color component of the yth pixel in the food image:

[0039] First, split the original image into 9 sub-images, extract the color features of the color space mixture for each sub-image, add some other channel values ​​in the color space based on the original monochrome technology that only uses grayscale values, and transform one of the sub-images of the image to HSV, Lab, and RGB color spaces respectively. In different color spaces, collect the values ​​of different channels of H, S, V, R, G, B, L, a, and b of the color feature subspaces extracted by HSV, RGB, and Lab of the food image to calculate the corresponding color moments. According to the formula, the corresponding components are calculated to obtain three low-order color moments, and then the corresponding first-order moments are fused to obtain the color space mixed brightness component. By analogy, the corresponding second-order and third-order moments are obtained. In the formula, , , are the weight factors corresponding to the color space, is the corresponding component of the color space, and then the corresponding components of different color spaces are calculated , three color moments of the hue component of the color space are obtained by fusing the order moments corresponding to the two components, and finally 9 color feature vectors of a sub-image fused in the color space are obtained. The color feature vectors of other sub-images are extracted using a similar method, and finally the color feature vectors of the 9 sub-images fused in the color space are obtained, so as to characterize the color characteristics of the food image.

[0040] The color feature quality inspection module (4) includes the following steps: S6. Calculate the cosine similarity of the feature vectors in the two feature libraries. The cosine value of the angle between the vectors represents the similarity of the two vectors. The larger the cosine value, the more similar the two vectors are, that is, the more similar the quality inspection food and the standard food images are, and they belong to the same category of food. The formula is:

[0041] In the formula, A and B represent two eigenvectors, and Represent the components of vectors A and B respectively, n represents the dimension of the feature vector, and in this method, n is set to 128. The cosine similarity x between the vector in the quality inspection feature library and the vector in the identification feature library of the food is calculated by the above method, and then combined with the weight factor of the color feature in the food , perform quality inspection score calculation, and improve the system mathematical formula design as follows:

[0042] In the formula, Represents the color weight factor of the food, x represents the cosine similarity result, y is a constant variable, which can be flexibly adjusted in the system according to business needs. The value range is [0, 1]. In this method, the value of y is 0.5.

[0043] Workflow of the present invention: When the food color feature quality inspection system and method based on multi-color space fusion designed by the present invention is working, a food target subject detection model and a food target feature extraction model are adopted. The subject detection model is intended to detect all food targets in the image captured by the camera, and the feature extraction model is intended to perform category feature extraction and color problem feature extraction on the food targets output by the subject detection model and perform similarity calculation with the label features in the image library to obtain the correct classification and recognition and quality inspection score of the food targets, decouple the food detection and recognition tasks, and complete the extraction of color and texture features in parallel during the process of food feature extraction, thereby accelerating the efficiency of food recognition and quality inspection; at the same time, this method based on prior color and texture feature retrieval has strong robustness and reproducibility, and can well filter environmental and light noises through feature convolution, thereby ensuring the fairness and accuracy of the system in food color and texture quality inspection.

[0044] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A food color feature quality inspection system based on multi-color space fusion, comprising a color space selection module (1), characterized in that: The output end of the color space selection module (1) is electrically connected to the area reconstruction and removal of food packaging reflective light module (2), the output end of the area reconstruction and removal of food packaging reflective light module (2) is electrically connected to the multiple color space feature fusion module (3), and the output end of the multiple color space feature fusion module (3) is electrically connected to the color feature quality inspection module (4).

2. A food color feature quality inspection method based on multi-color space fusion, applicable to the food color feature quality inspection system based on multi-color space fusion as described in claim 1, comprising a color space selection module (1), an area reconstruction and removal of food packaging reflection module (2), a multi-color space feature fusion module (3) and a color feature quality inspection module (4), characterized in that: The color space selection module (1) includes RGB color space, HSV color space and Lab color space, and the steps of the RGB color space are: S1.RGB color model represents color, which can simplify the system architecture and design. In order to reduce the massive data brought by real color, CIE proposed to use the addition of the three primary color wavelengths of R, G, and B to achieve the representation, transmission and restoration of natural colors. Other colors can be generated by adding different RGB components. The formula is: Where C is the mixed color, r, g, b are the T units used for matching, 1T unit red light is equal to 0.30lm, 1T unit green light is equal to 0.59lm, 1T unit blue light is equal to 0.11lm, The steps of the HSV color space are: S2.HSV color space contains three color channels: H, S, and V. Hue represents a pure color on the color wheel; Saturation represents the percentage of pure color in the color; and Value represents the brightness of the color. The V channel value conforms to the human eye's perception of color. There is a nonlinear relationship between RGB and HSV. Assuming that the three channels of the RGB color space have a value range of [0,255], the calculation steps for converting to the HSV model are as follows: Calculate brightness (V): Calculate saturation (S): Calculate chromaticity (H): There is another simple calculation formula for transformation, as shown below: ； The steps of the Lab color space are: S3. It consists of three elements: illuminance L and a and b related to color. L represents luminosity, which is equivalent to brightness, with a value range of [0,100]. a represents the range from red to green, and b represents the range from blue to yellow. The value ranges of a and b are [+120,-120]. All colors are composed of the interactive changes of these three values. When converting RGB to Lab color space, it is divided into two steps: first convert to XYZ space, and then convert to Lab space. The specific conversion formula is as follows: in, , X, Y, Z are the three stimulus values ​​of a pixel in the image, , , It is the tristimulus value of the standard illuminant corresponding to the illumination source used to collect the image.

3. The method for food color feature quality inspection based on multi-color space fusion according to claim 2, characterized in that: The steps of area reconstruction and removing the food packaging reflective module (2) are as follows: S4. Locate the center point of the package reflection, use a 5*5 template to traverse each pixel in the food image, find the center point of the template as the maximum pixel mark of the template, and record it as the candidate center point; the second step is to use Sobel approximation to screen the candidate center points, and add a 3*3 area with e as the center of the candidate center point, as follows: Among them, a, b, c, d, e, f, g, h, and i represent the grayscale value of a pixel respectively. There are 4 straight lines passing through the center point e, and each straight line divides the area into 2 parts. Each part has 3 pixels, called sub-areas. The maximum value of d1, d2, d3, and d4 is calculated by the formula and compared with the threshold. The image representation in area reconstruction is defined as a morphological opening and closing operation to eliminate the distortion and information loss caused by the morphological opening and closing filter: is the identification diagram, where Perform area opening reconstruction. is the output image; is the identification diagram, where Perform area closed reconstruction, For the output image.

4. The method for food color feature quality inspection based on multi-color space fusion according to claim 2, characterized in that: The multi-color space feature fusion module (3) comprises the following steps: S5. The original color histogram is changed to the color moment to characterize the color features of the image. Because the color information of food images is mainly concentrated in the low-order moments, the color moments are divided into first-order moments, second-order moments and third-order moments. The first-order moment is used to express the mean information of the image, including the overall overview of the image; the second-order moment is used to express the variance information of the image, including some detailed changes; the third-order moment is used to express the slope information of the image. The following are the three low-order moments of the image color. In formula (3-1), Represents the xth color component of the yth pixel in the food image: First, split the original image into 9 sub-images, extract the color features of the color space mixture for each sub-image, add some other channel values ​​in the color space based on the original monochrome technology that only uses grayscale values, and transform one of the sub-images of the image to HSV, Lab, and RGB color spaces respectively. In different color spaces, collect the values ​​of different channels of H, S, V, R, G, B, L, a, and b of the color feature subspaces extracted by HSV, RGB, and Lab of the food image to calculate the corresponding color moments. According to the formula, the corresponding components are calculated to obtain three low-order color moments, and then the corresponding first-order moments are fused to obtain the color space mixed brightness component. By analogy, the corresponding second-order and third-order moments are obtained. In the formula, , , are the weight factors corresponding to the color space, is the corresponding component of the color space, and then the corresponding components of different color spaces are calculated , three color moments of the hue component of the color space are obtained by fusing the order moments corresponding to the two components, and finally 9 color feature vectors of a sub-image fused in the color space are obtained. The color feature vectors of other sub-images are extracted using a similar method, and finally the color feature vectors of the 9 sub-images fused in the color space are obtained, so as to characterize the color characteristics of the food image.

5. The method for food color feature quality inspection based on multi-color space fusion according to claim 2, characterized in that: The color feature quality inspection module (4) comprises the following steps: S6. Calculate the cosine similarity of the feature vectors in the two feature libraries. The cosine value of the angle between the vectors represents the similarity of the two vectors. The larger the cosine value, the more similar the two vectors are, that is, the more similar the quality inspection food and the standard food images are, and they belong to the same category of food. The formula is: In the formula, A and B represent two eigenvectors, and Represent the components of vectors A and B respectively, n represents the dimension of the feature vector, and in this method, n is set to 128. The cosine similarity x between the vector in the quality inspection feature library and the vector in the identification feature library of the food is calculated by the above method, and then combined with the weight factor of the color feature in the food , perform quality inspection score calculation, and improve the system mathematical formula design as follows: In the formula, Represents the color weight factor of the food, x represents the cosine similarity result, y is a constant variable, which can be flexibly adjusted in the system according to business needs. The value range is [0, 1]. In this method, the value of y is 0.5.

Images