Multispectral full-surface real-time monitoring structure

Abstract

The invention provides a multispectral full-surface real-time monitoring structure, and aims to improve the efficiency of fruit state detection. The structure comprises a controller, an integrating sphere, a camera, a first reflector and a second reflector. The light source in the integrating sphere comprises a plurality of wavelengths, and the camera is arranged on the inner wall of the integrating sphere. And real-time monitoring on all surfaces of the fruit is realized by accurately calculating and setting the position of the reflecting mirror. The structure further comprises an optical filter array, a transmission device and a detector. Compared with the prior art, all the surfaces of the fruit can be monitored simultaneously, the detection efficiency is remarkably improved, the detection time is shortened, and the fruit state is comprehensively analyzed.

Description

Technical Field

[0001] The invention relates to a multi-spectral full-surface real-time monitoring structure, belonging to the technical field of hyperspectral. Background Art

[0002] To ensure the detection of the state of the fruit, the existing method uses a hyperspectral light source to illuminate the fruit and analyze the spectral characteristic images of the fruit under the illumination of light sources of different wavelengths to analyze the state of the fruit. However, the existing hyperspectral detection can only detect one surface at a time. If the surface of the fruit is to be fully detected, it is necessary to shoot different surfaces of the fruit multiple times. Shooting different surfaces of the fruit multiple times in monochrome will make the detection efficiency low and cannot achieve high-speed real-time detection. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a multi-spectral full-surface real-time monitoring structure to overcome the deficiencies of the prior art.

[0004] The technical solution of the present invention is: a multi-spectral full-surface real-time monitoring structure, the structure comprising a controller, an integrating sphere, a camera, a first reflector and a second reflector;

[0005] The light source in the integrating sphere includes more than one wavelength, the light source of the integrating sphere is located at the bottom of the inner cavity of the integrating sphere, and the light source of the integrating sphere is electrically connected to the controller;

[0006] The camera is arranged on the inner wall of the integrating sphere, the illumination direction of the camera passes through the center of the integrating sphere and is perpendicular to the vertical axis of the integrating sphere, and the camera is electrically connected to the controller;

[0007] The position of the first reflector is determined by the following method:

[0008] First, a two-dimensional Cartesian coordinate system is established by taking the plane perpendicular to the vertical axis and passing through the center of the integrating sphere as the coordinate plane, the center of the integrating sphere as the coordinate origin, the line connecting the coordinate origin to the camera as the y-axis, and the straight line perpendicular to the y-axis and passing through the center of the integrating sphere as the x-axis;

[0009] Then determine the minimum slope k of the mirror projection on the coordinate plane when the sample does not block the virtual image formed by the first reflector and the second reflector min ;

[0010] Then, the sample is divided into three partitions. The method of dividing into three partitions is to use three rays passing through the sample and on the coordinate plane to divide it equally. The angle between each ray is 120°. These three rays are called the first ray, the second ray and the third ray, wherein the first ray is collinear with the line connecting the coordinate origin and the camera but points in the opposite direction of the camera. The section of the arc formed by the edge of the circular model equally divided by the three rays, which is closest to the first reflector, is the first arc. The second arc and the third arc are counted counterclockwise from the first arc. The midpoint of the first arc is set to D, the midpoint of the second arc is set to D1, and the midpoint of the third arc is set to D2. The virtual image of the midpoint D of the first arc on the first reflector is D′, and the virtual image of D2 of the third arc on the second reflector is D′2. Calculate the optimal slope k of the projection of the first reflector and the second reflector on the coordinate plane when the optimal judgment criterion is met. opt , the optimal judgment criterion is: in the image captured by the camera, D1 is located at the horizontal midpoint of the sample, D′ is located at the horizontal midpoint of the virtual image formed by the first reflector, and D2 is located at the horizontal midpoint of the virtual image formed by the second reflector;

[0011] Finally, compare k opt With k min , if k opt >k min , then take k opt as the optimal slope of the projections of the first reflector and the second reflector on the coordinate plane;

[0012] According to k opt Determine the placement positions of the first reflector and the second reflector.

[0013] Specifically, the minimum slope k of the projection of the mirror on the coordinate plane when the sample does not block the virtual image formed by the first reflector and the second reflector is determined. min The method is:

[0014] 1) First determine the minimum slope k of the projection of the mirror on the coordinate plane when the virtual image is formed by the first reflector min , the determination method is:

[0015] S01, determining the distance R from the camera to the sample, approximating the projection of the sample on the coordinate plane as a circular model, and determining the radius r of the circular model;

[0016] S02. Establish the coordinate equation of the center point of the virtual image.

[0017]

[0018] Where O′ is the virtual image of the center O of the circular model in the first reflector, y O′ Indicates the vertical coordinate of the center point of the virtual image, x O′The abscissa of the center point of the virtual image, k represents the slope of the first reflector in the two-dimensional Cartesian coordinate system, and H represents the intercept of the first reflector in the two-dimensional Cartesian coordinate system;

[0019] S03, establish the first angle θ1, the second angle θ2 and the virtual image center angle θ O′ The equation,

[0020]

[0021] Wherein, the first tangent is a straight line passing through the camera and tangent to the circular model, and the first tangent is close to the first reflector, the first angle θ1 is the angle between the line connecting the camera and the coordinate origin and the first tangent, the second angle θ2 is the angle between the line connecting the camera and the virtual image O′ and the first tangent, θ O′ The first angle θ1 is the angle between the line connecting the camera and the coordinate origin and the line connecting the camera and the virtual image O′;

[0022] S04. Establish constraint equations.

[0023] θ1+θ2=θ O′ ;

[0024] S05. Combining the equations established in steps S03 and S04, and obtaining the minimum value k of the slope k of the first reflector in the two-dimensional Cartesian coordinate system by a numerical analysis method. min ;

[0025] 2) Then determine the minimum slope k of the projection of the mirror on the coordinate plane when the virtual image is formed by the second reflector min , the determination method is:

[0026] The minimum slope k of the projection of the mirror on the coordinate plane when the first reflector is projected on the coordinate plane with the y-axis of the two-dimensional Cartesian coordinate system as the midline to obtain the virtual image formed by the second reflector min .

[0027] Specifically, the optimal slope k of the first reflector is calculated when the optimal decision criterion is met: opt The method is:

[0028] Step 1: Establish the equation of D', where D' is the virtual image of D formed by the first reflector.

[0029]

[0030] Among them, x D represents the horizontal coordinate of D in the two-dimensional Cartesian coordinate system, y D represents the ordinate of D in the two-dimensional Cartesian coordinate system,

[0031] Step 2: Establish an equation for the optical axis, where the optical axis is a straight line connecting the camera position A(0, R) and the virtual image center O'.

[0032] (y O′ -R)x-(yR)x O′ =0;

[0033] Step 3: Substitute D' into the optical axis equation to obtain,

[0034] (y O′ -R)x D′ -(y D′ -R)x O′ =0;

[0035] Step 4: Combine the equations established in step S02, step 1 and step 3, and obtain the optimal slope k of the projection of the first reflector on the coordinate plane by numerical analysis method. opt .

[0036] Further, the optimal slope k of the second reflector is calculated when the optimal judgment criterion is met. opt The method is:

[0037] The optimal slope k of the projection of the mirror on the coordinate plane when the first reflector is projected on the coordinate plane with the y-axis of the two-dimensional Cartesian coordinate system as the midline to obtain the virtual image formed by the second reflector opt .

[0038] Furthermore, it also includes a filter array, which includes more than one filter that passes different wavelengths. The filter array is arranged in front of the camera lens, and the light passing through each filter can be completely irradiated onto the camera lens.

[0039] Furthermore, it also includes a transmission device, the plane where the transmission device is located coincides with the coordinate plane, the transmission device includes a first transmission part, a second transmission part and a third transmission part, the middle of the first transmission part passes through the coordinate origin, the input end of the first transmission part passes through the integrating sphere shell to connect to the outside, the input end of the second transmission part is connected to the output end of the first transmission part, the output end of the second transmission part is connected to the input end of the third transmission part, and the output end of the third transmission part bypasses the camera position and passes through the integrating sphere shell to connect to the outside.

[0040] Furthermore, it also includes a detector, which is arranged on the top of the integrating sphere, the detection direction of the detector is directly downward, and the detector is electrically connected to the controller.

[0041] The beneficial effects of the present invention are: compared with the prior art,

[0042] 1) The present invention enables multi-spectral technology to monitor all surfaces of the fruit simultaneously in real time by properly setting the positions of the first reflector and the second reflector, thus overcoming the limitation that the traditional technology can only detect a single surface of the fruit. This design significantly improves the detection efficiency because it allows a comprehensive analysis of the fruit status without increasing the number of additional shots, thereby reducing the total time required for detection;

[0043] 2) The present invention uses a filter array structure to enable the camera to simultaneously obtain data from multiple spectra, thereby comprehensively analyzing the state of the fruit without increasing the number of additional shots, thereby reducing the total time required for detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a structural schematic diagram of the present invention;

[0045] Figure 2 It is a circuit connection block diagram of the present invention;

[0046] Figure 3 The average spectrum of the slightly damaged, severely damaged and healthy areas of the apple of the present invention;

[0047] Figure 4 The structure diagram of the filter array and the filter effect diagram of the present invention;

[0048] Figure 5 This is a principle analysis diagram of the present invention when the virtual image is just not blocked;

[0049] Figure 6 It is a principle analysis diagram of the mirror of the present invention when the slope k is at the optimal position on the two-dimensional Cartesian coordinate system;

[0050] Figure 7 It is a schematic diagram for analyzing the change of the virtual image position when the slope k of the mirror of the present invention changes on a two-dimensional Cartesian coordinate system;

[0051] Figure 8 This is a real picture of the present invention when points D and D2 are located at the center of the virtual image and D1 is the center of the sample;

[0052] Fig. 9 This is a real picture of the present invention when the virtual image is blocked by the sample and the points D and D2 are not located at the center of the virtual image;

[0053] Fig.10 It is a flow chart of the image detection algorithm of the present invention;

[0054] Fig.11 This is the confusion matrix obtained after the experiment of the present invention. DETAILED DESCRIPTION

[0055] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0056] Implementation Example 1:

[0057] Prototype development

[0058] refer to Figure 1-2 , a multi-spectral full-surface real-time monitoring structure, the structure comprising a controller 4, an integrating sphere 7, a camera 5, a first reflector 2-1 and a second reflector 2-2;

[0059] The light source in the integrating sphere 7 includes more than one wavelength, the light source of the integrating sphere 7 is located at the bottom of the inner cavity of the integrating sphere 7, and the light source of the integrating sphere 7 is electrically connected to the controller 4;

[0060] The camera 5 is arranged on the inner wall of the integrating sphere 7, the illumination direction of the camera 5 passes through the center of the integrating sphere 7 and is perpendicular to the central axis of the integrating sphere 7 in the vertical direction, and the camera 5 is electrically connected to the controller 4;

[0061] It also includes a transmission device 1, the plane where the transmission device 1 is located coincides with the coordinate plane, the transmission device 1 includes a first transmission part 1-1, a second transmission part 1-2 and a third transmission part 1-3, the middle of the first transmission part 1-1 passes through the coordinate origin, the input end of the first transmission part 1-1 passes through the outer shell of the integrating sphere 7 to connect to the outside, the input end of the second transmission part 1-2 is connected to the output end of the first transmission part, the output end of the second transmission part 1-2 is connected to the input end of the third transmission part 1-3, and the output end of the third transmission part 1-3 bypasses the position of the camera 5 and passes through the outer shell of the integrating sphere 7 to connect to the outside.

[0062] The device further comprises a detector 3 , which is arranged on the top of the integrating sphere 7 . The detection direction of the detector 3 is directly downward, and the detector 3 is electrically connected to the controller 4 .

[0063] Figure 1 The schematic diagram of the prototype system is shown. The workflow is as follows: First, the detector 3 (infrared diffuse reflectance detector) detects the apple sample on the transmission device 1 and sends the acquisition signal to the controller 4 through the serial port. Secondly, after receiving the acquisition signal, the controller 4 activates the camera 5 (multispectral camera) to capture the multispectral image, and then processes and classifies these images online. Finally, the classification results are transmitted to the subordinate device (STM32 microcontroller) through the serial port, which controls the sorting device to guide the apples to the designated sorting box.

[0064] Characteristic wavelength selection

[0065] Figure 3 The blue box is the characteristic spectrum obtained by SPA.

[0066] Obtaining the characteristic spectrum of apple damage is the first step in building a full-surface multispectral imaging design. Initially, the spectral data of normal, slightly bruised, and severely bruised apple tissues were collected using our previously developed hyperspectral imaging system. Figure 3 As shown. Subsequently, based on these spectral distributions, characteristic spectra are extracted using the successive projection algorithm (SPA). SPA is a useful forward variable selection method that has been widely used in spectral imaging technology to develop simpler calibration models. To address the problem of collinearity, SPA starts with a single variable and iteratively adds new variables. This process continues until N variables with minimal redundancy are constructed through a series of projection operations on the columns of the spectral matrix. A detailed description of SPA can be found in these works. Using the SPA method, four characteristic wavelengths (505nm, 610nm, 688nm, 976nm) were selected from the original spectrum for further study.

[0067] Multispectral Multiview Imaging

[0068] After the characteristic wavelengths were determined, a multispectral multi-view imaging scheme was designed, combining a lens-filter array 6 and a mirror-based multi-angle imaging system to accelerate the imaging process and achieve real-time online multispectral monitoring. The scheme uses one imaging sensor to capture all multispectral data for the entire sample surface in a single shot. The lens array focuses and redistributes light from different angles and positions, enabling a single CCD to observe multiple points on an object simultaneously. By inserting a filter array 6 in front of the lens array, the wavelength of light passing through the lens is limited, allowing the spectral distribution image of the object under each filter to be observed. The mirror-based multi-angle imaging method captures the surface morphology of the entire sample from different angles in a single shot. The innovative combination of these two technologies enables the apple damage detection prototype system based on the multispectral multi-view imaging scheme to capture four characteristic wavelengths, each with three visual images of apples, such as Figure 3 This method significantly improves the speed of full-surface multispectral imaging on the production line, making the data collection process more efficient and convenient. The following sections will explain the lens-filter array and mirror-based multi-angle imaging technology in detail.

[0069] Multispectral Lens-Filter Array

[0070] refer to Figure 4 The structure of the filter array 6 is as follows: the filter array 6 includes more than one filter that passes different wavelengths. The filter array 6 is arranged in front of the lens of the camera 5, and the light passing through each filter can be completely irradiated onto the lens of the camera 5.

[0071] Figure 4CCD with multispectral lens-filter array 6: (a) Physical structure. (b) 3D structural design of lens-filter array 6. The first and second stages are the structures of fixed lenses and filters. (c) Imaging example of the full-surface multispectral apple damage detection prototype.

[0072] The development of lens-filter arrays for multispectral imaging uses 3D printing technology, and its structure is as follows Figure 3 As shown. The main structure is printed with PLA material and contains four convex lenses (3 mm in diameter, focal length 3.2 mm) and filters, which are spaced 3.6 mm inside the main body. The main body is also equipped with a threaded ring that allows it to be rotated and embedded in an industrial camera. In the current work, a 1 / 1.7" industrial camera (Hijrobot, MV-CE200-10UM, China) with a resolution of 4024×3036 pixels (width: 7.6 mm, height: 5.7 mm) was used.

[0073] Despite the simplicity of the concept of a lens-filter array6, multispectral imaging solutions based on a multi-lens approach require relatively uniform irradiance in the angular distribution so that each filter and corresponding lens captures similar light power (Lapray et al., 2014). To address this issue, an integrating sphere7 optical system with a monochromatic LED was proposed to provide a uniform light source with adjustable intensity to illuminate the entire apple sample surface. However, integrating such an integrating sphere7 structure poses challenges. For example, how to match it with a dual-mirror imaging structure for full-surface acquisition.

[0074] Dual-mirror multi-surface imaging in an integrating sphere

[0075] As mentioned above, the system proposed in this paper uses dual side mirrors to achieve multiple views of the sample through a single CCD. This configuration allows the capture of morphological information from three different angles in a single shot, significantly increasing the observable surface area of ​​the sample. Although the proposed scheme obtains more spatial texture information, it is not without cost. Its system layout needs to be carefully designed. To solve the above problems, the present invention proposes the following method to arrange the mirror position:

[0076] The position of the first reflector 2-1 is determined by the following method:

[0077] First, a two-dimensional Cartesian coordinate system is established by taking a plane perpendicular to the vertical axis and passing through the center of the integrating sphere 7 as a coordinate plane, taking the center of the integrating sphere 7 as a coordinate origin, taking a line from the coordinate origin to the camera 5 as a y-axis, and taking a straight line perpendicular to the y-axis and passing through the center of the integrating sphere 7 as an x-axis;

[0078] Then determine the minimum slope k of the mirror projection on the coordinate plane when the sample does not block the virtual image formed by the first reflector 2-1 and the second reflector 2-2min ;

[0079] Determine the minimum slope k of the projection of the mirror on the coordinate plane when the sample does not block the virtual image formed by the first reflector 2-1 and the second reflector 2-2 min The method is:

[0080] 1) First determine the minimum slope k of the projection of the mirror on the coordinate plane when the virtual image is formed by the first reflector 2-1 min , the determination method is:

[0081] S01, determine the distance R from the camera to the sample, approximate the projection of the sample on the coordinate plane as a circular model, and determine the radius r of the circular model;

[0082] S02. Establish the coordinate equation of the center point of the virtual image.

[0083]

[0084] Wherein, O′ is the virtual image of the center O of the circular model formed in the first reflector 2-1, y O′ Indicates the vertical coordinate of the center point of the virtual image, x O′ The abscissa of the center point of the virtual image, k represents the slope of the first reflector 2-1 in the two-dimensional Cartesian coordinate system, and H represents the intercept of the first reflector 2-1 in the two-dimensional Cartesian coordinate system;

[0085] S03, establish the first angle θ1, the second angle θ2 and the virtual image center angle θ O′ The equation,

[0086]

[0087] The first tangent is a straight line passing through the camera 5 and tangent to the circular model, and the first tangent is close to the first reflector 2-1. The first angle θ1 is the angle between the line connecting the camera 5 and the origin of the coordinate system and the first tangent. The second angle θ2 is the angle between the line connecting the camera 5 and the virtual image O′ and the first tangent. O′ The first angle θ1 is the angle between the line connecting the camera 5 and the coordinate origin and the line connecting the camera 5 and the virtual image O′;

[0088] S04. Establish constraint equations.

[0089] θ1+θ2=θ O′ ;

[0090] S05. Combine the equations established in steps S03 and S04, and obtain the minimum value k of the slope k of the first reflector 2-1 in the two-dimensional Cartesian coordinate system by numerical analysis method. min ;

[0091] 2) Then determine the minimum slope k of the projection of the mirror on the coordinate plane when the virtual image formed by the second reflector 2-2 is min , the determination method is:

[0092] The minimum slope k of the projection of the mirror on the coordinate plane when the first reflector 2-1 is projected on the coordinate plane with the y-axis of the two-dimensional Cartesian coordinate system as the midline to obtain the virtual image formed by the second reflector 2-2 is min .

[0093] However, the above equations are only necessary conditions to avoid occlusion of the virtual image, and we further hope that the virtual image captured by the side mirror covers the entire surface of the sample and is uniformly captured by the camera and the mirror. Therefore, in this work, an empirical method for the selection of structural parameters is proposed, namely the use of a three-segment imaging scheme. Since the designed dual-mirror system uses three visual directions (one direct image from the camera and two reflected images from the mirrors), it is further suggested to make the sample more uniformly imaged in the corresponding directions, which is the principle of the proposed three-segment imaging condition.

[0094] Then, the sample is divided into three partitions. The method of dividing into three partitions is to use three rays passing through the sample and on the coordinate plane to divide it equally. The angle between each ray is 120°. These three rays are called the first ray, the second ray and the third ray, wherein the first ray is collinear with the line connecting the coordinate origin and the camera 5 but points in the opposite direction of the camera 5. The section of the arc formed by the edge of the circular model equally divided by the three rays, which is closest to the first reflector 2-1, is the first arc. Counting counterclockwise from the first arc, they are the second arc and the third arc. The midpoint of the first arc is set to D, the midpoint of the second arc is set to D1, and the midpoint of the third arc is set to D2. The virtual image of the midpoint D of the first arc on the first reflector 2-1 is D′, and the virtual image of D2 of the third arc on the second reflector 2-2 is D′2. Calculate the optimal slope k of the projection of the first reflector 2-1 and the second reflector 2-2 on the coordinate plane when the optimal judgment criterion is met. opt , the optimal judgment criterion is: in the image captured by the camera 5, D1 is located at the horizontal midpoint of the sample, D′ is located at the horizontal midpoint of the virtual image formed by the first reflector 2-1, and D2 is located at the horizontal midpoint of the virtual image formed by the second reflector 2-2;

[0095] Specifically, the optimal slope k of the first reflector 2-1 is calculated when the optimal judgment criterion is met: opt The method is:

[0096] Step 1: Establish the equation of D', where D' is the virtual image of D formed by the first reflector 2-1.

[0097]

[0098] Among them, xD represents the horizontal coordinate of D in the two-dimensional Cartesian coordinate system, y D represents the ordinate of D in the two-dimensional Cartesian coordinate system,

[0099] Step 2: Establish an equation for the optical axis, where the optical axis is a straight line connecting the camera position A(0, R) and the virtual image center O'.

[0100] (y O′ -R)x-(yR)x O′ =0;

[0101] Step 3: Substitute D' into the optical axis equation to obtain,

[0102] (y O′ -R)x D′ -(y D′ -R)x O′ =0;

[0103] Step 4: Combine the equations established in step S02, step 1 and step 3, and obtain the optimal slope k of the projection of the first reflector 2-1 on the coordinate plane by numerical analysis method. opt .

[0104] Calculate the optimal slope k of the second reflector 2-2 when the optimal judgment criterion is met opt The method is:

[0105] The optimal slope k of the projection of the mirror on the coordinate plane when the first reflector 2-1 is projected on the coordinate plane with the y-axis of the two-dimensional Cartesian coordinate system as the midline to obtain the virtual image formed by the second reflector 2-2 is opt .

[0106] Finally, compare k opt With k min , if k opt >k min , then take k opt As the optimal slope of the projection of the first reflector 2-1 and the second reflector 2-2 on the coordinate plane;

[0107] According to k opt Determine the placement positions of the first reflector 2-1 and the second reflector 2-2.

[0108] When the multispectral camera is organically combined with the dual-mirror full-surface imaging in the integrating sphere 7, the proposed full-surface multispectral solution can simultaneously capture the spectral images of all characteristic wavelengths at three different viewing angles through a single CCD in a single imaging process, e.g. Fig.10 As shown in the figure, designing matching image processing and classification algorithms is the key element to achieve online apple damage detection, which can be summarized into the following three steps:

[0109] I. Multispectral Data Cube

[0110] To construct a multispectral data cube, it is necessary to geometrically rearrange the pixels in the image on the multi-channel virtual image plane. Since the detector 3 captures the image immediately after identifying the sample, the position of the sample in each field of view remains relatively stable. Therefore, the multispectral data cube of the apple can be obtained by cropping and stacking. Fig.10 , 12 sub-images can be generated. These sub-images can be divided into three groups according to the imaging angle, each group contains four characteristic wavelength images of apples. By stacking these characteristic wavelength images, a multispectral data cube with three viewing angles can be created. In the subsequent analysis, the view images directly from the camera, sub-images ①, ②, ③, and ④, will be used as examples to demonstrate the following image processing and classification methods.

[0111] II. Apple Sample Segmentation

[0112] The grayscale environment in the cavity of the integrating sphere 7 is diverse and complex, and the apple needs to be segmented from the cavity environment before identifying the damaged area. The 505nm wavelength image is used for apple sample segmentation because the grayscale difference between the apple and the wall environment is most obvious at this wavelength. In the program, the sample is first subjected to 'Canny' edge detection. Then, morphological processing is performed to eliminate noise and fill areas to obtain a binary image of the apple.

[0113] III. Classification of injuries

[0114] The classification of apple damage follows a systematic process that starts with identifying and localizing damage based on grayscale differences. If no damage is detected, the apple is classified as healthy; otherwise, a multispectral strategy is used to assess the degree of damage. This strategy involves extracting multispectral feature data from the damaged area and using a classic support vector machine (SVM) network to distinguish between minor and major defects, such as Fig.10 shown.

[0115] For lesion detection and localization, the binary image of the apple obtained in step II is combined with the 976 nm spectral image to eliminate the background in the image of this wavelength. This wavelength was chosen because, based on visual inspection, it highlights the lesion information rather than the distribution of skin color. For the resulting image, the classical local adaptive thresholding method is used to localize the lesion area, generating a binary image of the lesions. This method processes the image locally and calculates the appropriate threshold based on the neighborhood of each pixel. If no lesions are identified, the apple is classified as healthy and the process ends. Instead, the following steps will be performed to assess the degree of damage.

[0116] By applying the binary damage image to the multispectral data cube obtained in step I, the key multispectral classification information of the damaged area can be extracted. The classic supervised learning algorithm Support Vector Machine (SVM) is used to analyze these classification data online and distinguish between slightly and severely damaged apples. SVM is widely used in classification and regression tasks to achieve fast classification by maximizing the margin between categories. The working details of SVM can be found in these studies (Wang et al., 2023; Yue et al., 2003).

[0117] For images captured from other perspectives (such as Fig.10 Finally, by performing a logical AND operation on the detection results of these three perspectives, the damage on the entire surface of an apple can be fully determined.

[0118] Results and discussion

[0119] The results of classifying damaged apples using the full-surface multispectral strategy are shown in Table 1(b). It is obvious that when a strategy similar to the traditional classification method is adopted to classify apples based on a single view, the detection accuracy of damaged apples is significantly lower (51.7% for the CCD front, 53.3% for the left mirror, and 51% for the right mirror). This poor performance is due to the limited surface area captured by the single view. When the damage is located in an area that is not visible from a single view, these damaged apples are incorrectly classified as healthy. This incomplete detection seriously affects the overall assessment of apple damage. In contrast, the three-view imaging scheme of this study captures multispectral information from the entire surface of the apple, achieving a comprehensive assessment. The overall accuracy rate reached 89.3%. Specifically, the accuracy rates for classifying healthy apples, slightly damaged apples, and severely damaged apples are 100%, 88%, and 80%, respectively, achieving significantly better results compared to the traditional single-view classification method. The confusion matrix of the classification results using the full-surface strategy is shown in Figure 1(b). Fig.11 It reveals that out of 100 slightly damaged apples, 12 were misclassified as severely damaged. Similarly, 20 out of 100 severely damaged apples were misclassified as slightly damaged.

[0120] Table 1. Classification accuracy of damaged apples

[0121]

[0122] The proposed system performs full-surface inspection of apples and achieves a real-time online screening speed of 3 samples per second. It expands the monitoring area from the front surface facing the imaging device to the traditionally unmonitored back surface. This extension minimizes the misclassification caused by the limited field of view inherent in single-view imaging. Processing more multispectral data on the apple surface increases the computational load. Nevertheless, the system compensates for this by adopting a fast data acquisition scheme, that is, using a single shot with a single CCD camera to capture multispectral information of the entire surface without mechanical movement, making this full-surface multispectral real-time online monitoring a reality.

[0123] The prototype system achieves real-time online screening speeds comparable to other state-of-the-art apple inspection methods. Specifically, the system is capable of real-time online screening at a rate of 3 samples per second. This speed is achieved thanks to the system's full-surface inspection capability, which extends from the front surface to the traditionally unmonitored rear surface, which minimizes misclassification due to the limited field of view inherent in single-view imaging. At the same time, although processing more multispectral data on the apple surface increases the computational load, the system compensates for this by adopting a fast data acquisition scheme. Specifically, the system uses a single shot of a single CCD camera to capture multispectral information of the entire surface without mechanical movement, making full-surface multispectral real-time online monitoring a reality. This fast data acquisition scheme and the ability to take a single shot make the system comparable in efficiency to other state-of-the-art apple inspection methods, such as 3 apples per second by Wenqian Huang et al. (Chen et al., 2015), 5 apples per second by Chi Zhang et al. (Zhang, C. et al., 2017), and 3 apples per second by Han Chang et al. (Han et al., 2023).

[0124] Mathematical derivation of the optimal value of slope k

[0125] 1. Mathematical derivation process for determining the minimum slope when there is no occlusion

[0126] First, a two-dimensional Cartesian coordinate system is established with a plane perpendicular to the central axis in the up-down direction and passing through the center of the integrating sphere 7 as the coordinate plane, the center of the integrating sphere 7 as the coordinate origin, the line connecting the coordinate origin to the camera 5 as the y-axis, and the straight line perpendicular to the y-axis and passing through the center of the integrating sphere 7 as the x-axis, where O represents the origin of the two-dimensional Cartesian coordinate system; once the radius r of the circular model, the sample size, the distance R from the camera to the sample, and the distance H between the mirror and the sample are determined, the virtual image captured by the camera is closely related to the slope of the placement of the first reflector 2-1 and the second reflector 2-2. Figure 5The relationship between the sample virtual image calculated based on the reflection imaging law and the change in the mirror slope is described in . It can be observed that as the mirror slope decreases, the virtual image moves downward and closer to the sample, eventually leading to occlusion. This phenomenon should be avoided during system design because occlusion prevents the camera from capturing a complete image.

[0127] Therefore, determining the critical minimum slope for mirror placement is crucial to the model. Figure 5 As shown in Figure 1, this critical condition occurs when the camera's line of sight touches both the sample's contour circle O and the virtual image circle O'. As the slope decreases, as shown in the figure, occlusion of the virtual image occurs. Conversely, a larger slope allows the sample's virtual image to be fully captured by the camera. Under this co-tangent condition, the geometric relationship in the figure reveals that the angle θ between the camera's line of sight to the virtual image center (AO') and the y-axis is O' , the angle θ1 between the line of sight and the y-axis, and the angle θ2 between AO' and the tangent line AM must satisfy the constraint relationship expressed in equation (1).

[0128] θ1+θ2=θ O′ (1)

[0129] The three angle values ​​in equation (1) can all be related to the structural parameters of the system, where θ1 can be determined by the inverse sine relationship in the right triangle △AEO.

[0130]

[0131] Equation (2) shows that θ1 is determined by the sample size r and the distance R from the camera to the sample.

[0132] To obtain θ2 and θ in equation (1) O′ , we need to first calculate the coordinates of the sample virtual center O', which can be determined based on the symmetry of the point about the line. The mirrored object in the coordinate system can be expressed as:

[0133] kx-yH=0 (3)

[0134] According to the coordinates of the sample center O(0,0) and equation (3), the coordinates of the virtual image O' can be expressed as:

[0135]

[0136] Once the coordinates of O' are obtained, θ2 can be determined from the inverse sine relationship in the right triangle ΔAMO'.

[0137]

[0138] θ O′It can be solved by the inverse tangent function in the right triangle △ACO'.

[0139]

[0140] Equations (1), (2), (5), and (6) determine the minimum slope (k) required for the mirror if the virtual image in the camera is not obstructed. These equations constrain the slope k by the system configuration parameters r, R, and H.

[0141] Here is an example to illustrate the use of these formulas: Consider a cavity design with a radius R of 200 mm, a mirror placed 60 mm from the sample H, and a sample size r of 20 mm. Using the numerical method, the critical value obtained from the above equations (1), (2), (5), and (6) is 0.53. Using a polymer model sphere to simulate these parameters, the results are shown in Figure 3 Several key observations can be drawn from the figure: 1) Figure 5 The red line in the figure is the critical line of the virtual image without occlusion. 2) When k is less than the critical value, the virtual image will be located within the red line, resulting in occlusion, such as Figure 7 shown.

[0142] 2. Mathematical derivation process of three-segment imaging

[0143] However, the above equations are only necessary conditions to avoid occlusion of the virtual image, and we further hope that the virtual image captured by the side mirror covers the entire surface of the sample and is uniformly captured by the camera and the mirror. Therefore, in this work, an empirical method for the selection of structural parameters is proposed, namely the use of a three-segment imaging scheme. Since the designed dual-mirror system uses three visual directions (one direct image from the camera and two reflected images from the mirrors), it is further suggested to make the sample more uniformly imaged in the corresponding directions, which is the principle of the proposed three-segment imaging condition.

[0144] To achieve this, a circular model divided into three equal parts was built, e.g. Figure 6 As shown in the figure, the three-segment imaging model is analyzed using the right mirror as an example. In this model, three different colors represent sub-areas monitored by the visual direction, and their centers are marked as D, D1, and D2, respectively. In order to minimize the impact of spatial displacement of the sample and system processing errors on the imaging area, these sub-areas should ideally be located in the center of the field of view. This requires that point D' be aligned with the optical axis (the optical axis is the virtual line from the center point of the camera lens to the center of the imaging object), and the sub-areas are symmetrically distributed within the visual range with D' as the center. Under this collinear condition, k can be obtained by the following three steps:

[0145] I. Coordinates of virtual image point D'

[0146] First, since D is the midpoint of the three equally divided sub-regions of the target, ∠COD is 30°. According to the trigonometric function of △COD, the coordinates of D can be expressed as:

[0147]

[0148] Then, D' is the point D that is symmetric about the mirror plane. Therefore, according to equations (3) and (7) and the principle of symmetry of a point about a line, the coordinates of D' can be expressed as:

[0149]

[0150] II. Expression of the optical axis

[0151] In an optical system, the optical axis is a virtual line from the center point of the camera lens to the center of the imaged object. Figure 4 In , the line connecting the camera space coordinate A and the center point O' of the sample virtual image is the optical axis of the sample virtual image. According to the coordinates of A(0, R) and point O', the straight line expression satisfied by any point (x, y) on the optical axis can be written using the two-point straight line equation:

[0152] (y O′ -R)x-(yR)x o′ =0 (9)

[0153] where xO', yO' are the coordinates of O' calculated by equation (4).

[0154] Analysis of III.k

[0155] According to the principle of the proposed three-segment imaging condition, in order to make the virtual image formed by the side more symmetrical, point D' must be located on the optical axis. In other words, the coordinate D' calculated by equation (8) should also satisfy equation (9). Therefore, substituting equation (8) into equation (9) yields the following equation:

[0156] (y O′ -R)x D′ -(y D′ -R)x O′ =0 (10)

[0157] Where xO', yO' are determined by equation (4), and xD', yD' are determined by equation (8).

[0158] A joint analysis of these equations (4), (8) and (10) shows that R, H and r are known structural parameters of the system. Therefore, by numerically solving equation (10), the required k can be obtained. The mirror configuration defined by this value allows to obtain the full spatial texture information of the apple surface from three equally divided fields of view.

[0159] Using the polymer sample example above, applying equations (4), (8) and (10) yields a k value of 0.72, which exceeds the critical threshold of 0.53 defined by equation (6). A k value greater than 0.53 ensures that the virtual image has no occlusion view, while a value of 0.72 ensures that the spatial texture information of the entire sample can be obtained from three equally divided directions. Application of the polymer sample model is shown in Fig. 9 As shown in the illustration, if it is both blocked and not at the center of the arc, the image will be like Fig. 9 shown.

[0160] In summary, we theoretically derived two necessary conditions for the proposed system. Equations (1), (2), (5), and (6) establish basic constraints to ensure that the virtual image is not obscured by the physical sample. At the same time, equations (4), (8), and (10) form additional constraints for the three-segment imaging scheme based on the double-sided mirror. When these two sets of conditions are satisfied simultaneously, the imaging of the sample is complete, uniform, and symmetric.

[0161] For the proposed prototype system, the structural parameters are as follows: the average radius (r) of the apple is about 37.5 mm, an integrating sphere (7) with a radius (R) of 300 mm is used as the illumination frame, and the mirror is placed at a distance of 125 mm (H) from the sample. The (k) value calculated according to equations (4), (8) and (10) is 0.76, which exceeds the critical threshold of 0.53 determined by equations (1), (2), (5) and (6). Therefore, this value is selected as the optimal slope for the mirror placement.

[0162] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A multi-spectral full-surface real-time monitoring structure, characterized in that: The structure comprises a controller (4), an integrating sphere (7), a camera (5), a first reflecting mirror (2-1) and a second reflecting mirror (2-2); The light source in the integrating sphere (7) includes more than one wavelength, the light source of the integrating sphere (7) is located at the bottom of the inner cavity of the integrating sphere (7), and the light source of the integrating sphere (7) is electrically connected to the controller (4); The camera (5) is arranged on the inner wall of the integrating sphere (7), the illumination direction of the camera (5) passes through the center of the integrating sphere (7) and is perpendicular to the central axis of the integrating sphere (7) in the vertical direction, and the camera (5) is electrically connected to the controller (4); The position of the first reflector (2-1) is determined by the following method: First, a two-dimensional Cartesian coordinate system is established by taking a plane perpendicular to the vertical axis and passing through the center of the integrating sphere (7) as a coordinate plane, taking the center of the integrating sphere (7) as a coordinate origin, taking a line from the coordinate origin to the camera (5) as a y-axis, and taking a straight line perpendicular to the y-axis and passing through the center of the integrating sphere (7) as an x-axis; Then determine the minimum slope k of the mirror projection on the coordinate plane when the sample does not block the virtual image formed by the first reflector (2-1) and the second reflector (2-2): min ; Then, the sample is divided into three partitions. The method of dividing into three partitions is to use three rays passing through the sample and on the coordinate plane to divide the sample equally. The angle between each ray is 120°. The three rays are called the first ray, the second ray and the third ray. The first ray is collinear with the line connecting the coordinate origin and the camera (5) but points in the opposite direction of the camera (5). The section of the arc formed by the edge of the circular model equally divided by the three rays, which is closest to the first reflector (2-1), is the first arc. The second arc and the third arc are counted counterclockwise from the first arc. The midpoint of the first arc is set to D, the midpoint of the second arc is set to D1, and the midpoint of the third arc is set to D2. The virtual image of the midpoint D of the first arc on the first reflector (2-1) is D′, and the virtual image of D2 of the third arc on the second reflector (2-2) is D′2. Calculate the optimal slope k of the projection of the first reflector (2-1) and the second reflector (2-2) on the coordinate plane when the optimal judgment criterion is met. opt , the optimal judgment criterion is: in the image captured by the camera (5), D1 is located at the horizontal midpoint of the sample, D′ is located at the horizontal midpoint of the virtual image formed by the first reflector (2-1), and D2 is located at the horizontal midpoint of the virtual image formed by the second reflector (2-2); Finally, compare k opt With k min , if k opt >k min , then take k opt As the optimal slope of the projection of the first reflector (2-1) and the second reflector (2-2) on the coordinate plane; According to k opt The placement positions of the first reflector (2-1) and the second reflector (2-2) are determined.

2. The multi-spectral full-surface real-time monitoring structure according to claim 1 is characterized in that: The method for determining the minimum slope k of the projection of the mirror on the coordinate plane when the sample does not block the virtual image formed by the first reflector (2-1) and the second reflector (2-2) min The method is: 1) First determine the minimum slope k of the projection of the mirror on the coordinate plane when the virtual image is formed by the first reflector (2-1) min , the determination method is: S01, determining the distance R from the camera to the sample, approximating the projection of the sample on the coordinate plane as a circular model, and determining the radius r of the circular model; S02. Establish the coordinate equation of the center point of the virtual image. Where O′ is the virtual image of the center O of the circular model in the first reflector (2-1), y O′ Indicates the vertical coordinate of the center point of the virtual image, x O′ The abscissa of the center point of the virtual image, k represents the slope of the first reflector (2-1) in the two-dimensional Cartesian coordinate system, and H represents the intercept of the first reflector (2-1) in the two-dimensional Cartesian coordinate system; S03, establish the first angle θ1, the second angle θ2 and the virtual image center angle θ O′ The equation, The first tangent is a straight line passing through the camera (5) and tangent to the circular model, and the first tangent is close to the first reflector (2-1). The first angle θ1 is the angle between the line connecting the camera (5) and the coordinate origin and the first tangent. The second angle θ2 is the angle between the line connecting the camera (5) and the virtual image O′ and the first tangent. O′ The first angle θ1 is the angle between the line connecting the camera (5) and the coordinate origin and the line connecting the camera (5) and the virtual image O′; S04. Establish constraint equations. θ1+θ2=θ O′ ; S05. Combining the equations established in steps S03 and S04, the minimum value k of the slope k of the first reflector (2-1) in the two-dimensional Cartesian coordinate system is obtained by numerical analysis method. min ; 2) Then determine the minimum slope k of the projection of the mirror on the coordinate plane when the virtual image is formed by the second reflector (2-2) min , the determination method is: The minimum slope k of the projection of the mirror on the coordinate plane when the first reflector (2-1) is projected on the coordinate plane with the y-axis of the two-dimensional Cartesian coordinate system as the midline to obtain the virtual image formed by the second reflector (2-2) min .

3. The multi-spectral full-surface real-time monitoring structure according to claim 2 is characterized in that: Calculate the optimal slope k of the first reflector (2-1) when the optimal criterion is met opt The method is: Step 1: Establish the equation of D', where D' is the virtual image of D formed by the first reflector (2-1). Among them, x D represents the horizontal coordinate of D in the two-dimensional Cartesian coordinate system, y D represents the ordinate of D in the two-dimensional Cartesian coordinate system, Step 2: Establish an equation for the optical axis, where the optical axis is a straight line connecting the camera position A(0, R) and the virtual image center O'. (y O′ -R)x-(y-R)x O′ =0; Step 3: Substitute D' into the optical axis equation to obtain, (y O′ -R)x D′ -(y D′ -R)x O′ =0; Step 4: Combine the equations established in step S02, step 1 and step 3, and obtain the optimal slope k of the projection of the first reflector (2-1) on the coordinate plane by numerical analysis method. opt .

4. The multi-spectral full-surface real-time monitoring structure according to claim 3 is characterized in that: Calculate the optimal slope k of the second reflector (2-2) when the optimal criterion is met opt The method is: The optimal slope k of the projection of the mirror on the coordinate plane when the mirror image of the first reflector (2-1) is made on the coordinate plane with the y axis of the two-dimensional Cartesian coordinate system as the midline to obtain the virtual image formed by the second reflector (2-2) opt .

5. The multi-spectral full-surface real-time monitoring structure according to claim 1, characterized in that: It also includes a filter array (6), wherein the filter array (6) includes more than one filter that passes different wavelengths. The filter array (6) is arranged in front of the lens of the camera (5), and the light passing through each filter can be completely irradiated onto the lens of the camera (5).

6. The multi-spectral full-surface real-time monitoring structure according to claim 1, characterized in that: It also comprises a transmission device (1), wherein the plane where the transmission device (1) is located coincides with the coordinate plane, the transmission device (1) comprises a first transmission part (1-1), a second transmission part (1-2) and a third transmission part (1-3), the middle of the first transmission part (1-1) passes through the coordinate origin, the input end of the first transmission part (1-1) passes through the outer shell of the integrating sphere (7) to connect to the outside, the input end of the second transmission part (1-2) is connected to the output end of the first transmission part, the output end of the second transmission part (1-2) is connected to the input end of the third transmission part (1-3), the output end of the third transmission part (1-3) bypasses the position of the camera (5) and passes through the outer shell of the integrating sphere (7) to connect to the outside, and the transmission device (1) is electrically connected to the controller (4).

7. The multi-spectral full-surface real-time monitoring structure according to claim 1, characterized in that: It also comprises a detector (3), wherein the detector (3) is arranged on the top of the integrating sphere (7), the detection direction of the detector (3) is directly downward, and the detector (3) is electrically connected to the controller (4).

Images