Intelligent fruit and vegetable rot monitoring device and method based on multi-modal sensing fusion

Abstract

The invention relates to an intelligent fruit and vegetable rot monitoring device and method based on multi-modal sensing fusion. The device comprises a visual sensing module, a fruit and vegetable rotten gas detection module, a central processing module, a display module, a wireless communication module and a user interaction terminal, the device can flexibly select fixed, movable or integrated monitoring, and meets the requirements of different monitoring scenes. According to the method, the concentration of fruit and vegetable rotten gas is monitored through a gas-sensitive sensing array, an artificial intelligence algorithm is combined with optical features to complete rotten degree quantification and category feature matching, data is converted into a three-dimensional space rotten degree thermodynamic diagram, and an early warning notification is pushed to a user interaction end. And full-chain intelligent management of fruit and vegetable rot from detection to feedback is realized. Aiming at a fruit and vegetable stacking storage scene, a gas-sensitive sensing device and a visual sensing device are creatively fused, the blind area limitation of a traditional single-point detection technology on stacking middle rot detection is broken through, and an innovative early warning scheme is provided for quickly and accurately monitoring fruit and vegetable rot.

Description

Technical Field

[0001] The present invention belongs to the technical field of food safety detection, and specifically relates to an intelligent monitoring device for fruit and vegetable spoilage based on multimodal sensing fusion and a method thereof. Different from the traditional single monitoring method, the present invention innovatively adopts gas-sensitive sensing and optical sensing fusion technology, which can conduct real-time and comprehensive monitoring and analysis of the changes in gas composition and appearance morphological characteristics of fruits and vegetables during the entire process of storage, transportation and sales. By adopting a gas-sensitive and optical feature fusion algorithm, the stacked fruit and vegetable spoilage detection blind spot is broken through, and the data is converted into a visualized three-dimensional spatial spoilage degree heat map, which can realize accurate identification of the fruit and vegetable spoilage state, and timely issue an early warning at the early stage of spoilage, thereby improving the timeliness and accuracy of fruit and vegetable spoilage monitoring, and providing a new solution for ensuring food safety. Background Art

[0002] Fruits and vegetables are piled up and stacked during storage and transportation, which often leads to local decay and the emission of organic amines and mycotoxin contamination and spread, posing food safety risks to consumers. Secondly, during the storage and transportation of fruits and vegetables, the incomplete cold chain facilities and the numerous distribution links exacerbate the decay of fruits and vegetables. Therefore, the development of a fast and accurate intelligent monitoring device and method for the decay of fruits and vegetables is of great significance to ensure the safety of public food.

[0003] Today, fruit and vegetable spoilage monitoring technologies mainly include spectral analysis and machine vision. Spectral analysis technology (Trends in Food Science & Technology, 2024, 150, 104612) identifies spoilage by detecting the spectral reflection / absorption characteristics of fruits and vegetables in specific bands, but it is difficult to be widely used in the circulation of agricultural products due to the large size, high cost and complex noise interference of the detection equipment. Machine vision (Current Research in Food Science2024, 8, 100723) analyzes the appearance characteristics of fruits and vegetables based on image processing algorithms to evaluate spoilage. However, it can only reflect the surface state of fruits and vegetables and lacks effective detection capabilities for internal spoilage in stacked storage scenarios, resulting in limited detection accuracy.

[0004] During the spoilage process of fruits and vegetables, organic and inorganic characteristic gases are often released (Foods 2023, 12, 3966). Gas-sensing technology, with its advantages of non-invasive detection, portable equipment, and simple operation, demonstrates unique value in the field of fruit and vegetable spoilage monitoring. By monitoring the change in the concentration of characteristic gases generated during the spoilage process, this technology can reflect the internal spoilage state of fruits and vegetables in real time, breaking through the limitations of traditional visual inspection that relies on surface characteristics and providing an innovative solution to the problem of detecting internal spoilage in stacked fruit and vegetable scenarios. Accordingly, the present invention proposes an intelligent monitoring device and method for fruit and vegetable spoilage based on multimodal perception fusion. By integrating gas-sensing and visual detection technologies, the detection timeliness and accuracy are effectively improved, providing an innovative solution for spoilage monitoring in the fruit and vegetable circulation link. Summary of the Invention

[0005] Based on the above-mentioned disadvantages and deficiencies in the prior art, the purpose of the present invention is to provide an intelligent monitoring device and method for fruit and vegetable spoilage based on multimodal perception fusion to solve the problems existing in the above-mentioned background technology.

[0006] To achieve the above purpose, the present invention provides the following technical solutions: An intelligent monitoring device for fruit and vegetable spoilage based on multimodal perception fusion, comprising: a visual sensing module, a fruit and vegetable spoilage gas detection module, a central processing module, a display module, a wireless communication module, and a user interaction terminal. Each module is as follows: The visual sensing module includes at least one camera, electrically connected to the input end of the central processing module, and is used to detect the categories of fruits and vegetables stored in the imaging area, including fruits and vegetables. Among them, fruits include but are not limited to apples, bananas, oranges, kiwifruits, and vegetables include but are not limited to potatoes, cabbages, cabbages, sweet potatoes.

[0007] The fruit and vegetable spoilage gas detection module consists of a gas sensor array, including but not limited to resistive gas sensors, electrochemically gas sensors, optically gas sensors, and is electrically connected to the input end of the central processing module, and is used to monitor the concentration of fruit and vegetable spoilage gases in the measurable space. The spoilage gases include but are not limited to alcohol, ether, ammonia, sulfide gases that are easily generated during fruit and vegetable spoilage.

[0008] The central processing module is used to receive, process, and send data. By running artificial intelligence algorithms, it outputs electrical control signals and data information, and is built-in with standard interfaces, including but not limited to digital interfaces, analog interfaces, bus interfaces, network interfaces, for connecting and communicating with other modules or systems.

[0009] The display module is electrically connected to the output end of the central processing module, and is used to display the fruit and vegetable images, categories, and spoilage degree information processed by the central processing module.

[0010] The wireless communication module is electrically connected to the output end of the central processing module, and is used to receive the warning information processed by the central processing module and transmit the information to the user interaction end through a wireless transmission method.

[0011] The user interaction end performs information interaction with the wireless communication module through a wireless connection method, and is used to receive the fruit and vegetable category information and spoilage degree information processed by the central processing module.

[0012] In the intelligent monitoring device for fruit and vegetable spoilage based on multi-modal perception fusion: The fruit and vegetable spoilage gas detection module is internally provided with a temperature adjustment module and an analog-to-digital conversion module. The temperature adjustment module is used to preheat various types of gas sensors, and the analog-to-digital conversion module is used to convert the monitored analog gas response signal into a digital signal; The outside of the fruit and vegetable spoilage gas detection module is provided with a semi-closed cavity, which is used to provide a defined semi-closed space to reduce measurement errors.

[0013] The intelligent monitoring device for fruit and vegetable spoilage based on multi-modal perception fusion further includes a storage module, a gas pump system, and a power supply module. Among them, the storage module includes but is not limited to an SD memory card, which is electrically connected to the central processing module and is used to store processing algorithms, including but not limited to image processing algorithms, gas-sensitive and optical feature fusion algorithms; the gas pump system is connected to the detection cavity to realize dynamic gas sampling in the monitoring space; the power supply module provides the energy required for the operation of the device.

[0014] The operation method of the intelligent monitoring device for fruit and vegetable spoilage based on multi-modal perception fusion includes the following steps: S1. Detect whether fruits and vegetables are stored in the target area through the visual sensing module. If so, turn off the visual sensing module and start the fruit and vegetable spoilage gas detection module; if not, repeat step S1; S2. The fruit and vegetable spoilage gas detection module transmits the monitored gas-sensitive response data to the central processing module through the analog-to-digital conversion module. The central processing module processes the signal and calculates the gas-sensitive response value, and judges whether the gas-sensitive response value exceeds the preset gas-sensitive spoilage threshold. If so, start the visual sensing module; if not, repeat step S2; S3. Based on the visual sensing module, perform real-time image acquisition on the spoilage area and transmit the information to the central processing module. Determine the fruit and vegetable category stored in the spoilage area and the appearance spoilage degree according to the preset image processing algorithm; S4. Display the image information in real time through the display module, adopt the gas-sensitive and optical feature fusion algorithm to perform fusion decision on the gas-sensitive and visual spoilage results, convert the data into a visual three-dimensional space spoilage degree heat map, and mark the fruit and vegetable position, fruit and vegetable category, and spoilage degree and output them to the display module; S5. Synchronize the information to the user interaction terminal through the wireless communication module, and at the same time give an early warning of the spoilage of fruits and vegetables.

[0015] Among them, the specific steps of step S1 include: Step S101. Start the camera and transmit the real-time image information of the monitoring area to the central processing module; Step S102. The central processing module judges whether the image contains fruits and vegetables according to the preset image processing algorithm. If the image does not contain fruits and vegetables, return to step S101; if the image contains fruits and vegetables, jump to step S103; Step S103. Turn off the vision sensing module, start the temperature regulation module, preheat various types of gas sensors in the gas sensor array. After the preheating is completed, the air pump starts to work intermittently, and the working interval is consistent with the measurement period of the gas sensor. The gas in the monitoring space is transmitted to the semi-closed cavity through the air pump; Step S104. The gas sensor array obtains real-time monitoring signals according to the measurement period, outputs digital signals through the analog-to-digital conversion module, and transmits the digital signals to the central processing module.

[0016] The application methods of the intelligent monitoring device for the spoilage of fruits and vegetables based on multi-modal perception fusion include but are not limited to the following three: (1) Fixed monitoring. The device can be arranged in batches in the space for storing fruits and vegetables (including but not limited to warehouses) to monitor the target space area. If spoilage of fruits and vegetables in a certain area is detected by the monitoring device in that area, the monitoring device in that area will transmit the monitoring results to the user interaction terminal and give an early warning of spoilage; (2) Mobile monitoring. The device can move to measure the spoilage degree of target fruits and vegetables in a subjective human control manner. If spoilage of fruits and vegetables is detected by the device, the device will mark and display the information through the display module and give an early warning of spoilage; (3) Integrated monitoring. The device can be electrically connected to various instruments or devices through the standard interface built into the central processing module, including but not limited to mechanical sorting mechanisms. The device monitors the spoilage degree of fruits and vegetables and outputs electrical control signals to control the operating state of the instruments to meet the requirements of different monitoring scenarios.

[0017] The intelligent monitoring device and method for the spoilage of fruits and vegetables based on multi-modal perception fusion are characterized in that: Set the odor standard for the initial spoilage of fruits and vegetables as the preset gas spoilage threshold of the gas sensor array, which is determined by the gas sensitivity response curve; The spoilage monitoring result for setting the warning moment is the spoilage degree and credibility (ranging from 0 to 1) after the decision fusion of the output result of the fruit and vegetable spoilage monitoring module and the output result of the visual sensing module. The fruit and vegetable spoilage degree includes: mild spoilage, moderate spoilage, and severe spoilage; It is set that when the response value in the gas-sensitive response curve obtained by the gas-sensitive sensing module reaches the preset gas-sensitive spoilage threshold, a spoilage warning is sent to the user interaction terminal, and the fruit and vegetable category and spoilage degree information are transmitted.

[0018] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects: The present invention uses visual sensing technology to form a condition-triggered switch, and uses gas-sensitive sensing technology as a continuous monitoring means, reducing the power consumption cost of continuously monitoring using image vision technology. On this basis, the present invention uses the response recovery curve generated by the monitored gas-sensitive response value to extract the gas-sensitive features hidden in the response recovery curve, and obtains the internal spoilage degree monitoring data by comparing with the preset gas-sensitive spoilage threshold. Finally, by combining the appearance spoilage degree monitoring data obtained by processing the image with the image processing algorithm, the spoilage recognition accuracy of the monitored fruits and vegetables is improved through fusion decision-making.

[0019] In addition to the emerging spectral imaging and machine vision single-parameter fruit and vegetable spoilage monitoring methods, the present invention provides a spoilage monitoring scheme that combines gas-sensitive and optical sensing synergistically, which has potential application scenarios in monitoring the spoilage of fruits and vegetables during storage and transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following further describes the specific embodiments of the present invention with reference to the drawings, where: Figure 1 It is a flowchart of the operation method steps of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion; Figure 2 It is a system block diagram of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion; Figure 3 It is a schematic diagram of the fixed application space structure of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion; Figure 4 It is a schematic diagram of the structure of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion; Figure 5 It is a flowchart of the operation of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion.

[0021] The labels in the figure are: 1. Target storage rack; 2. Intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion; 201. Storage module; 202. Central processing module; 203. Display module; 204. Visual sensing module; 205. Wireless communication module; 206. Semi-closed cavity; 207. Fruit and vegetable spoilage gas detection module; 208. Exhaust pipeline; 209. Intake pipeline; 210. Air pump; 211. Power supply module. Detailed implementation manners

[0022] The following further describes in detail and completely the specific implementation manners of the present invention in combination with the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described below are only used to explain the present invention and are not used to limit the present invention.

[0023] Embodiment 1 This embodiment provides an intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion, including: a visual sensing module, a fruit and vegetable spoilage gas detection module, a central processing module, a display module, a wireless communication module and a user interaction terminal. The specific details of each module are as follows: The visual sensing module includes at least one camera, which is electrically connected to the input end of the central processing module and is used to detect whether there are fruits and vegetables in the imaging area and determine the categories of fruits and vegetables. The categories of fruits and vegetables are divided into fruits and vegetables. Among them, fruits include but are not limited to apples, bananas, oranges, kiwifruits, and vegetables include but are not limited to potatoes, cabbages, cabbages, sweet potatoes.

[0024] The fruit and vegetable spoilage gas detection module is composed of a gas sensor array, including but not limited to a resistive gas sensor, an electro-chemical gas sensor, and an optical gas sensor, and is electrically connected to the input end of the central processing module and is used to monitor the concentration of the spoilage odor of fruits and vegetables in the measurable space. The spoilage gases include but are not limited to alcohols, ethers, ammonia, and sulfide gases that are easily generated when fruits and vegetables spoil. A temperature adjustment module and an analog-to-digital conversion module are provided inside the module. Among them, the temperature adjustment module is used to preheat the gas sensor array, and the analog-to-digital conversion module is used to convert the monitored analog gas-sensitive response signal into a digital signal. A semi-closed cavity is provided outside the module to provide a defined semi-closed space and reduce measurement errors.

[0025] The display module is electrically connected to the output end of the central processing module and is used to display the fruit and vegetable images, categories, and spoilage degree information processed by the central processing module.

[0026] The wireless communication module is electrically connected to the output end of the central processing module and is used to receive the warning information processed by the central processing module and transmit the information to the user interaction terminal through a wireless transmission method.

[0027] The user interaction terminal interacts with the wireless communication module through a wireless connection method, and is used to receive the fruit and vegetable category information and spoilage degree information processed by the central processing module.

[0028] In order to optimize the monitoring quality of the fruit and vegetable spoilage gas detection module, a semi-closed cavity is connected to an air pump. The air inlet pipeline of the air pump is connected to the gas in the monitoring space, and the air outlet pipeline of the air pump is connected to the semi-closed cavity.

[0029] Such as Figure 1 As shown, a running method of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion includes the following steps: S1. Detect whether fruits and vegetables are stored in the target area through the vision sensing module. If so, turn off the vision sensing module and start the fruit and vegetable spoilage gas detection module; if not, repeat step S1; S2. The fruit and vegetable spoilage gas detection module transmits the monitored gas-sensitive response data to the central processing module through the analog-to-digital conversion module. The central processing module processes the signal and calculates the gas-sensitive response value, and judges whether the gas-sensitive response value exceeds the preset gas-sensitive spoilage threshold. If so, start the vision sensing module; if not, repeat step S2; S3. Based on the vision sensing module, collect real-time images of the spoilage area and transmit the information to the central processing module. Determine the fruit and vegetable category stored in the spoilage area and the appearance spoilage degree according to the preset image processing algorithm; S4. Display the image information in real time through the display module. Adopt the gas-sensitive and optical feature fusion algorithm to fuse and make decisions on the spoilage results of gas sensitivity and vision, convert the data into a visual three-dimensional space spoilage degree heat map, and mark the fruit and vegetable position, fruit and vegetable category, and spoilage degree and output them to the display module; S5. Synchronize the information to the user interaction terminal through the wireless communication module, and at the same time give an early warning of fruit and vegetable spoilage.

[0030] Among them, step S1 specifically includes: Step S101. Start the camera and transmit the image information of the monitoring area to the central processing module in real time; Step S102. The central processing module judges whether the image contains fruits and vegetables according to the preset image processing algorithm. If the image does not contain fruits and vegetables, return to step S101; if the image contains fruits and vegetables, jump to step S103; Step S103. Turn off the vision sensing module and start the temperature regulation module to preheat various types of gas sensors in the gas sensor array. The preheating time depends on the length of the system idle time; after the preheating is completed, the air pump starts to work intermittently, and the working interval is consistent with the measurement period of the gas sensor. The gas in the monitoring space is transmitted to the semi-closed cavity through the air pump; Step S104: The gas sensor array obtains real-time monitoring signals according to the measurement period, outputs digital signals through the analog-to-digital conversion module, and transmits the digital signals to the central processing module.

[0031] Embodiment 2 This embodiment provides a multi-modal perception fusion-based intelligent monitoring system for fruit and vegetable spoilage, which is used for the multi-modal perception fusion-based intelligent monitoring device for fruit and vegetable spoilage described in Embodiment 1. As Figure 2 shown, the system mainly consists of a computing unit, a storage unit, a communication unit, and a result output unit. The computing unit further includes a gas detection unit and a visual sensing unit, which are respectively used to acquire and process gas-sensitive response information and image information; the storage unit is used to store processing algorithms, including but not limited to image processing algorithms, gas-sensitive and optical feature fusion algorithms; the communication unit is used to transmit the calculation results of the computing unit to the result output unit; the result output unit further includes a user interaction unit and a display unit, which are used to output and display the monitoring results and transmit them to the user interaction terminal.

[0032] The main operation idea of the system is as follows: First, the gas detection unit monitors the concentration of spoilage gas in the space and compares the concentration value with the preset gas-sensitive spoilage threshold; if the preset gas-sensitive spoilage threshold is reached, the visual sensing unit is activated to determine the category of fruits and vegetables stored in the monitoring area and the degree of appearance spoilage. Then, based on the output results of the gas sensitivity and vision of the computing unit, a fusion decision is made, and the data is transmitted to the result output unit through the communication unit for spoilage early warning. On the one hand, it is displayed in real time through the display unit, and on the other hand, the result is transmitted to the user interaction unit. The present invention makes full use of the fusion result of gas sensitivity and vision, improves the accuracy of predicting the spoilage degree of fruits and vegetables, and improves the technical problem that the existing single vision technology relies on surface appearance image information for spoilage detection. Due to the storage characteristics of stacking and piling of fruits and vegetables, internal spoilage is difficult to detect, resulting in a low recognition rate.

[0033] Embodiment 3 This embodiment provides a fixed operation mode of a multi-modal perception fusion-based intelligent monitoring device for fruit and vegetable spoilage, which is used for the operation method of a multi-modal perception fusion-based intelligent monitoring device for fruit and vegetable spoilage described in Embodiment 1. As Figure 3 shown, it is a schematic diagram of the fixed application space structure of the device, including a target storage rack 1 and a multi-modal perception fusion-based intelligent monitoring device 2 for fruit and vegetable spoilage.

[0034] Among them, the target storage rack 1 is a conventional multi-layer storage rack for storing fruits and vegetables to be monitored, generally placed in an open square in an orderly manner; on the side wall of each storage layer, a multi-modal perception fusion-based intelligent monitoring device 2 for fruit and vegetable spoilage is fixedly installed for spoilage early warning monitoring.

[0035] Specifically, the intelligent monitoring device 2 for spoilage of fruits and vegetables based on multi-modal perception fusion includes a storage module 201, a central processing module 202, a display module 203, a visual sensing module 204, a wireless communication module 205, a semi-closed cavity 206, a fruit and vegetable spoilage gas detection module 207, an air outlet pipeline 208, an air inlet pipeline 209, an air pump 210, and a power supply module 211.

[0036] Specifically, as Figure 4 shown, the artificial intelligence algorithm is stored in the storage module 201 and is electrically connected to the central processing module 202; the display module 203 is electrically connected to the central processing module 202. When fruit and vegetable spoilage is detected, the display module externally displays information, including fruit and vegetable images, categories, and spoilage degrees; the visual sensing module 204 is electrically connected to the central processing module 202 and includes at least one camera; the wireless communication module 205 is electrically connected to the central processing module 202; the semi-closed cavity 206 is located outside the fruit and vegetable spoilage gas detection module 207, provides a defined gas-sensing test space, reduces measurement errors, and is connected to the air outlet pipeline 208; the air outlet pipeline 208 and the air inlet pipeline 209 are respectively connected to the air outlet and air inlet of the air pump 210; the air pump 210 is electrically connected to the central processing module 202; the power supply module 211 is electrically connected to the central processing module 202 and the air pump 210 respectively to supply power to the central processing module 202 and the air pump 210.

[0037] Embodiment 4 This embodiment provides a mobile monitoring method for the intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion, which is used for the intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion described in Embodiment 1. As Figure 4 shown, it is the device structure diagram of the mobile monitoring method of this device, and the specific description is as in Embodiment 3 above. The usage way of the mobile monitoring method is: the spoilage degree of the target fruits and vegetables is monitored in an artificial control manner. The user holds the mobile device, brings the device close to the fruits and vegetables, and turns on the power to start the device; after the device operation is completed, the monitoring result is fed back in the display mode of the display module 203.

[0038] Specifically, as Figure 5 shown, the mobile operation process of the device is: Start the visual sensing module, collect real-time images and detect whether there are fruits and vegetables. If there are fruits and vegetables, turn off the visual sensing module; if there are no fruits and vegetables, continue to perform real-time image detection.

[0039] When there are fruits and vegetables, turn off the visual sensing module, start the fruit and vegetable spoilage gas detection module, collect and process the gas-sensitive response data, and judge whether the fruits and vegetables are spoiled according to the spoilage gas concentration. If they are not spoiled, the display module will display and mark "fresh state"; if they are spoiled, start the visual sensing module, collect real-time images and determine the fruit and vegetable category and the degree of appearance spoilage according to the preset image processing algorithm.

[0040] Adopt the gas-sensitive and optical feature fusion algorithm to make a fusion decision on the spoilage results of gas-sensing and vision, convert the data into a visual three-dimensional space spoilage degree heat map, and mark the fruit and vegetable position, fruit and vegetable category and spoilage degree and output them to the display module.

[0041] Embodiment 5 This embodiment provides an integrated monitoring method for a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion for the sorting link in the transportation of fruits and vegetables, which is used for the fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion described in Embodiment 1. Among them, the fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion is electrically connected to the mechanical sorting mechanism through a standard interface built in the central processing module. The storage module of the device stores a sorting control algorithm, and realizes the real-time positioning and automatic removal of spoiled fruits and vegetables by monitoring the spoilage degree of fruits and vegetables in real time.

[0042] Specifically, the integrated operation process of the device is as follows: drive the visual sensing module, collect real-time images and detect whether there are fruits and vegetables. If there are fruits and vegetables, the visual sensing module will be turned off; if there are no fruits and vegetables, continue to perform real-time image detection.

[0043] When there are fruits and vegetables, turn off the visual sensing module, start the fruit and vegetable spoilage gas detection module, collect and process the gas-sensitive response data, and judge whether the fruits and vegetables are spoiled according to the spoilage gas concentration. If they are not spoiled, the display module will display and mark "fresh state"; if they are spoiled, start the visual sensing module, collect real-time images and determine the fruit and vegetable category and the degree of appearance spoilage according to the preset image processing algorithm.

[0044] Adopt the gas-sensitive and optical feature fusion algorithm to make a fusion decision on the spoilage results of gas-sensing and vision, convert the data into a visual three-dimensional space spoilage degree heat map, and mark the fruit and vegetable position, fruit and vegetable category and spoilage degree and output them to the display module, and control the mechanical sorting mechanism to selectively remove spoiled fruits and vegetables by outputting an electrical control signal, so as to realize the real-time positioning and automatic removal of spoiled fruits and vegetables.

[0045] The above are only the preferred embodiments of the present invention, and all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Claims

1. An intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion, characterized in that, Including: A visual sensing module, a fruit and vegetable spoilage gas detection module, a central processing module, a display module, a wireless communication module, and a user interaction terminal. Among them: The visual sensing module includes at least one camera, electrically connected to the input end of the central processing module, and is used to detect the categories of fruits and vegetables stored in the camera area, including fruits and vegetables. Among them, fruits include but are not limited to apples, bananas, oranges, kiwifruits, and vegetables include but are not limited to potatoes, cabbages, cabbages, sweet potatoes; The fruit and vegetable spoilage gas detection module consists of a gas sensor array, including but not limited to a resistive gas sensor, an electrochemistry gas sensor, and an optical gas sensor, electrically connected to the input end of the central processing module, and is used to monitor the concentration of the spoilage odor of fruits and vegetables in the measurable space. The spoilage gases include but are not limited to alcohols, ethers, ammonia, and sulfide gases that are easily generated when fruits and vegetables spoil; The central processing module is used to receive, process, and send data. By running artificial intelligence algorithms, it outputs electrical control signals and data information, and is built-in with standard interfaces, including but not limited to digital interfaces, analog interfaces, bus interfaces, and network interfaces, for connecting and communicating with other modules or systems; The display module is electrically connected to the output end of the central processing module, and is used to display the fruit and vegetable images, categories, and spoilage degree information processed by the central processing module; The wireless communication module is electrically connected to the output end of the central processing module, and is used to receive the warning information processed by the central processing module, and transmit the information to the user interaction terminal through wireless transmission; The user interaction terminal conducts information interaction with the wireless communication module through a wireless connection method, including but not limited to mobile phones and computers, and is used to receive the fruit and vegetable category information and spoilage degree information processed by the central processing module.

2. The intelligent monitoring device for fruit and vegetable spoilage based on multi-modal perception fusion according to claim 1, characterized in that: The fruit and vegetable spoilage gas detection module is internally provided with a temperature adjustment module and an analog-to-digital conversion module. The temperature adjustment module is used to preheat various types of gas sensors, and the analog-to-digital conversion module is used to convert the monitored analog signal into a digital signal; The fruit and vegetable spoilage gas detection module is externally provided with a semi-closed cavity, which is used to provide a limited semi-closed space to reduce measurement errors.

3. The intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion according to claim 1, wherein: The application methods of the device include but are not limited to the following three: (1) Fixed monitoring. The device can be batch arranged in a space containing fruits and vegetables (including but not limited to warehouses) to monitor the target space area. If the fruits and vegetables in a certain area are detected to spoil by the monitoring device in that area, the monitoring device in that area will transmit the monitoring result to the user interaction terminal and give a spoilage warning; (2) Mobile monitoring. The device can move to monitor the spoilage degree of target fruits and vegetables in a manual control manner. If the fruits and vegetables are detected to spoil by the device, the device will mark and display the information through the display module and give a spoilage warning; (3)Integrated monitoring. The device can be electrically connected to a variety of instruments or devices through a standard interface built into the central processing module, including but not limited to mechanical sorting mechanisms. The device monitors the degree of spoilage of fruits and vegetables and outputs electrical control signals to control the operating state of the instruments, meeting the requirements of different monitoring scenarios.

4. The intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion according to claim 1, wherein: It further includes a storage module, an air pump system, and a power supply module. Among them, the storage module includes but is not limited to an SD memory card, which is electrically connected to the central processing module and is used to store processing algorithms, including but not limited to image processing algorithms, gas-sensing and optical feature fusion algorithms; the air pump system is connected to the detection cavity to achieve dynamic gas sampling in the monitoring space; the power supply module provides the energy required for the operation of the device.

5. The intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion according to claim 1, characterized in that: It includes the following steps: S1. Detect whether fruits and vegetables are stored in the target area through the vision sensing module. If so, turn off the vision sensing module and start the fruit and vegetable spoilage gas detection module; if not, repeat step S1; S2. The fruit and vegetable spoilage gas detection module transmits the monitored gas-sensing response data to the central processing module through the analog-to-digital conversion module. The central processing module processes the signal and calculates the gas-sensing response value, and judges whether the gas-sensing response value exceeds the preset gas-sensing spoilage threshold. If so, start the vision sensing module; if not, repeat step S2; S3. Based on the vision sensing module, perform real-time image acquisition of the spoilage area and transmit the information to the central processing module. Determine the fruit and vegetable categories stored in the spoilage area and the degree of appearance spoilage according to the preset image processing algorithm; S4. Display the image information in real time through the display module. Adopt the gas-sensing and optical feature fusion algorithm to fuse and make a decision on the spoilage results of gas-sensing and vision, convert the data into a visualized three-dimensional space spoilage degree heat map, and mark the fruit and vegetable position, fruit and vegetable category, and spoilage degree and output them to the display module; S5. Synchronize the information to the user interaction terminal through the wireless communication module and at the same time give an early warning of fruit and vegetable spoilage.

6. The operating method of an intelligent monitoring device for spoilage of fruits and vegetables based on multi-modal perception fusion according to claim 2, characterized in that: In step S1, the complete process is: Step S101. Start the camera and transmit the image information of the monitoring area to the central processing module in real time; Step S102. The central processing module judges whether the image contains fruits and vegetables according to the preset image processing algorithm. If the image does not contain fruits and vegetables, return to step S101; if the image contains fruits and vegetables, jump to step S103; Step S103. Turn off the vision sensing module and start the temperature adjustment module to adjust the temperature of the gas-sensing sensor array; after the temperature adjustment is completed, the air pump system extracts the gas in the monitoring space at a preset cycle and transports it to the detection cavity through the gas path; Step S104. The gas-sensing sensor array obtains real-time monitoring signals according to the measurement cycle, outputs digital signals through the analog-to-digital conversion module, and transmits the digital signals to the central processing module.

7. An operating method of a fruit and vegetable spoilage intelligent monitoring device based on multi-modal perception fusion, characterized in that: Set the odor standard for preliminary spoilage of fruits and vegetables as the preset gas-sensing spoilage threshold of the gas-sensing sensor array, which is determined by the gas-sensing response curve; The spoilage monitoring result for setting the warning time is the spoilage degree and credibility (ranging from 0 to 1) of fruits and vegetables after the decision-making fusion of the output results of the fruits and vegetables spoilage monitoring module and the visual sensing module. The spoilage degree of fruits and vegetables includes: mild spoilage, moderate spoilage, and severe spoilage; It is set that when the response value in the real-time response curve obtained by the fruits and vegetables spoilage gas detection module reaches the gas-sensitive spoilage threshold, it is considered that the fruits and vegetables have spoiled, that is, a spoilage warning is sent to the user interaction terminal, and the fruit and vegetable category and spoilage degree are transmitted.

Images