A harvesting system, method and medium based on electrical signals and multispectral

Abstract

This application discloses a harvesting system, method, and medium based on electrical signals and multispectral imaging in the field of computer technology. This application establishes a fully automated harvesting system with dual-dimensional detection of multispectral imaging and plant electrical signals through the collaborative operation of a control terminal, a drone, an edge computing node, a server, and a robot. The drone utilizes a multispectral camera to quickly scan the harvesting area, while the edge computing node performs image denoising and stitching, resulting in fewer and more concise images. For image processing, the server determines the first maturity level using multispectral data, then introduces plant electrical signals to detect the second maturity level, thereby continuously improving the accuracy of plant maturity determination. Finally, the multispectral data and plant electrical signals are fused to confirm whether harvesting is necessary. After harvesting confirmation, the server automatically issues execution instructions, enabling the robot to perform precise harvesting, realizing a complete process from area inspection and maturity detection to automated harvesting.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data acquisition system, method and medium based on electrical signals and multispectral imaging. Background Technology

[0002] Currently, automatic plant harvesting devices require manual driving or remote control to complete the harvesting work. Furthermore, the large amount of water mist in the mist-grown plant greenhouses makes it easy for the automatic plant harvesting devices to misjudge the maturity of the plants based on visual recognition methods, making it difficult to achieve accurate harvesting in complex environments. The degree of automation in harvesting also needs to be improved.

[0003] Therefore, how to improve the degree of automation and accuracy of harvesting is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a harvesting system, method and medium based on electrical signals and multispectral data to improve the degree of automated harvesting and harvesting accuracy.

[0005] In a first aspect, this application provides a data acquisition system based on electrical signals and multispectral imaging, comprising:

[0006] The control unit is used to send harvesting instructions to the drone;

[0007] The drone is used to scan the pre-set harvesting area in the harvesting instruction using a multispectral camera and transmit the multispectral image to the edge computing node;

[0008] Edge computing nodes are used to denoise and stitch multispectral images to obtain target spectral data and transmit the target spectral data to the server.

[0009] The server is used to calculate the first maturity based on the target spectral data; if the first maturity meets the preset first condition, the spectral similarity between the target spectral data and the mature spectral data is determined; if the spectral similarity meets the preset threshold, an electrical signal detection command is sent to the robot.

[0010] The robot is used to collect plant electrical signals within a preset harvesting area according to electrical signal detection instructions, and transmit the plant electrical signals to the server.

[0011] On the server side, a second maturity level is calculated based on plant electrical signals. If the second maturity level meets a preset second condition, a result on whether to harvest is generated based on the target spectral data and plant electrical signals. If harvesting is confirmed based on the result, an execution command is sent to the robot.

[0012] Robots are used to harvest crops in pre-defined harvesting areas according to executed instructions.

[0013] Optionally, the server is used to calculate a vegetation index based on the target spectral data and determine the first maturity level based on the vegetation index.

[0014] Optionally, the server is used to perform dimensionality reduction on the target spectral data and determine the principal component features of the target spectral data; and to determine the spectral similarity based on the similarity between the principal component features and the mature spectral features of the mature spectral data.

[0015] Optionally, the server is used to construct a time series model for plant electrical signals, perform noise reduction and feature extraction on the plant electrical signals based on the time series model to obtain the target electrical signal, and determine the second maturity based on the mean square value and wavelet entropy of the target electrical signal.

[0016] Optionally, the server-side is used to fuse the target spectral data and plant electrical signals using an attention mechanism, classify the fusion results, and obtain a result indicating whether or not to harvest.

[0017] Optionally, the server is used to generate a picking path based on the second maturity level, the distance the robot needs to move, and the robot's status; and send execution instructions, including the picking path, to the robot.

[0018] Accordingly, robots are used to travel along the harvesting path and harvest from the pre-set harvesting areas.

[0019] Optionally, a robot is used to record the number, coordinates, picking time, and maturity information of the harvested target after harvesting, and to place the harvested target into a sterile processing chamber.

[0020] Optionally, the server is used to collect historical harvesting data and use a prediction model to predict future harvesting parameters based on the historical harvesting data; the harvesting parameters include harvesting area, harvesting sequence, and harvesting frequency; and / or, to update the prediction model using historical harvesting data.

[0021] Secondly, this application provides a harvesting method based on electrical signals and multispectral data, applied to a harvesting system based on electrical signals and multispectral data, the harvesting system comprising: a control terminal, a drone, an edge computing node, a server, and a robot;

[0022] The control unit sends harvesting instructions to the drone;

[0023] The drone uses a multispectral camera to scan and obtain multispectral images of the pre-set harvesting area in the harvesting command, and then transmits the multispectral images to the edge computing node;

[0024] Edge computing nodes denoise and stitch multispectral images to obtain target spectral data, and then transmit the target spectral data to the server.

[0025] The server calculates the first maturity level based on the target spectral data. If the first maturity level meets the preset first condition, the server determines the spectral similarity between the target spectral data and the mature spectral data. If the spectral similarity meets the preset threshold, the server sends an electrical signal detection command to the robot.

[0026] The robot collects plant electrical signals within a pre-set harvesting area according to electrical signal detection instructions and transmits the plant electrical signals to the server.

[0027] The server calculates the second maturity level based on the plant's electrical signals; if the second maturity level meets the preset second condition, it generates a result on whether to harvest based on the target spectral data and the plant's electrical signals; if harvesting is confirmed based on the result, an execution command is sent to the robot.

[0028] The robot harvests in the preset harvesting area according to the instructions.

[0029] Thirdly, this application provides a harvesting method based on electrical signals and multispectral data, applied to a server, wherein the server is connected to a control terminal, a drone, an edge computing node, and a robot;

[0030] Accordingly, the harvesting method includes:

[0031] The first maturity level is calculated based on the target spectral data transmitted by the edge computing node; wherein, the target spectral data is obtained by the edge computing node through denoising and stitching of a multispectral image; the multispectral image is obtained by the UAV responding to the harvesting command sent by the control terminal, using the multispectral camera of the UAV to scan the preset harvesting area in the harvesting command;

[0032] If the first maturity meets the preset first condition, then the spectral similarity between the target spectral data and the mature spectral data is determined;

[0033] If the spectral similarity meets a preset threshold, an electrical signal detection command is sent to the robot so that the robot collects plant electrical signals in the preset harvesting area according to the electrical signal detection command and transmits the plant electrical signals to the server.

[0034] The second maturity level is calculated based on the plant's electrical signals;

[0035] If the second maturity meets the preset second condition, a result on whether to harvest is generated based on the target spectral data and the plant electrical signal;

[0036] If harvesting is confirmed based on the results, an execution command is sent to the robot so that the robot can harvest the preset harvesting area according to the execution command.

[0037] Fourthly, this application provides an electronic device, comprising:

[0038] Memory, used to store computer programs;

[0039] A processor for executing the computer program to implement the aforementioned disclosed method.

[0040] Fifthly, this application provides a non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned acquisition method based on electrical signals and multispectral imaging.

[0041] In a sixth aspect, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed acquisition method based on electrical signals and multispectral imaging.

[0042] As can be seen, this application utilizes a collaborative operation involving a control unit, drones, edge computing nodes, a server, and robots to form a fully automated harvesting system with dual-dimensional detection of multispectral imaging and plant electrical signals. Specifically, the drones, using multispectral cameras, can achieve rapid scanning of a large area of ​​the harvesting region. Then, the edge computing nodes perform image denoising and image stitching to reduce data transmission pressure, resulting in less and more concise image data, which is beneficial for the real-time performance and accuracy of subsequent image processing. For image processing, the server first determines the initial maturity level using multispectral data, then performs a secondary screening based on spectral similarity, and introduces plant electrical signals to detect the second maturity level. This continuous detection improves the accuracy of plant maturity judgment and reduces false positives and false negatives. Subsequently, the multispectral data and plant electrical signals are fused to confirm whether to harvest, further improving harvesting accuracy. After the maturity level is met and harvesting is confirmed, the server automatically issues execution instructions, enabling the robot to perform precise harvesting. This achieves a fully automated and intelligent process from area inspection and maturity detection to automated harvesting, improving harvesting efficiency and adapting to the high-efficiency and precise harvesting needs of large-scale planting scenarios.

[0043] Correspondingly, the acquisition method based on electrical signals and multispectral imaging, as well as the electronic device, medium, and program product provided in this application, also have the aforementioned technical effects. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1This is a schematic diagram of a harvesting system based on electrical signals and multispectral imaging disclosed in this application;

[0046] Figure 2 This is a flowchart of a harvesting method based on electrical signals and multispectral analysis disclosed in this application;

[0047] Figure 3 This is a schematic diagram of another harvesting system based on electrical signals and multispectral imaging disclosed in this application;

[0048] Figure 4 This is a schematic diagram of an electronic device disclosed in this application;

[0049] Figure 5 A server architecture diagram provided in this application;

[0050] Figure 6 A terminal structure diagram provided for this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0052] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0053] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Currently, automated plant harvesting devices require manual operation or remote control to complete the harvesting process. Furthermore, the visual recognition methods used by these devices to determine plant maturity are prone to misjudgment, making accurate harvesting in complex environments difficult, and the level of automation needs improvement. Therefore, this application provides a harvesting scheme based on electrical signals and multispectral analysis, enabling a fully automated and intelligent process from area inspection and maturity detection to automated harvesting, thereby improving both the degree of automation and harvesting accuracy.

[0055] See Figure 1 As shown in the illustration, this application discloses a data acquisition system based on electrical signals and multispectral imaging, comprising: a control terminal, a drone, an edge computing node, a server, and a robot. Multiple drones, edge computing nodes, and robots may be used.

[0056] The control unit is used to send harvesting instructions to the drone.

[0057] The drone is used to scan the pre-set harvesting area in the harvesting command using a multispectral camera and transmit the multispectral image to the edge computing node.

[0058] Edge computing nodes are used to denoise and stitch multispectral images to obtain target spectral data and transmit the target spectral data to the server.

[0059] The server is used to calculate the first maturity level based on the target spectral data. If the first maturity level meets the preset first condition, the spectral similarity between the target spectral data and the mature spectral data is determined. If the spectral similarity meets the preset threshold, an electrical signal detection command is sent to the robot.

[0060] The robot is used to collect plant electrical signals within a preset harvesting area according to electrical signal detection instructions, and then transmit the plant electrical signals to the server.

[0061] On the server side, a second maturity level is calculated based on plant electrical signals. If the second maturity level meets a preset second condition, a harvest decision is generated based on the target spectral data and plant electrical signals. If harvesting is confirmed based on the result, an execution command is sent to the robot. Specifically, a weighted fusion of the target spectral data and plant electrical signals is performed using an attention mechanism, and a harvest decision is generated based on the fused features. The weighted fusion of the target spectral data and plant electrical signals using the attention mechanism includes: feature encoding and feature transformation of the target spectral data and plant electrical signals to construct Q, K, and V matrices; and calculation using the attention mechanism based on the Q, K, and V matrices to obtain the fused features.

[0062] Robots are used to harvest crops in pre-defined harvesting areas according to executed instructions.

[0063] In this embodiment, the control terminal can be a handheld mobile terminal device or a fixed terminal device, and can communicate with other devices through an APP or web page in the control terminal. Based on the APP or web page in the control terminal, parameters such as preset harvesting area, harvesting order, harvesting time and frequency can be set, and various information such as first maturity, spectral similarity, second maturity, whether harvesting has occurred, robot travel path, and drone flight path can be queried. The server can generate a maturity and harvesting distribution map based on the first maturity, spectral similarity, second maturity and / or whether harvesting has occurred. The control terminal can query this distribution map so that users can intuitively observe the distribution of harvestable plants. Furthermore, during the process of the robot harvesting in the preset harvesting area according to the execution instructions, the distribution map can be updated in real time so that users can intuitively distinguish between harvested areas (areas that have been harvested), harvestable areas (areas that have been confirmed to be harvestable but have not yet been harvested), and unharvested areas (areas waiting to mature).

[0064] In one implementation, the server is used to calculate a vegetation index based on target spectral data, and to determine a first maturity level based on the vegetation index. The vegetation index calculation formula is as follows: NIR represents the reflectance in the near-infrared band (700-1300nm); RED represents the reflectance in the red band (600-700nm); the vegetation index ranges from -1 to 1, with positive values ​​indicating vegetation cover and larger values ​​indicating more lush vegetation. Based on this, after calculating the vegetation index from the target spectral data, the first maturity level can be determined. For example, this vegetation index can be directly used as the first maturity level. Alternatively, a mapping relationship between the vegetation index and maturity can be established in advance, and the specific value of the first maturity level can be determined based on this mapping relationship. For example, if a vegetation index of [-1, 0) corresponds to a maturity level of 0 and a vegetation index of [0, 1] corresponds to a maturity level of 1, then when the calculated vegetation index is 0.2, since 0.2 falls within [0, 1], the first maturity level is set to 1. Accordingly, if the preset first condition is a first threshold of 0.99, then the first maturity value is 1, which is greater than 0.99. Therefore, the first maturity of 1 is considered to meet the preset first condition, and the spectral similarity calculation step can be performed. In other words: if the first maturity is greater than the first threshold, the first maturity is considered to meet the preset first condition; otherwise, the first maturity is considered not to meet the preset first condition. If the first maturity does not meet the preset first condition, the spectral similarity calculation step is not performed.

[0065] Specifically, if the server confirms that the first maturity level does not meet the preset first condition, it will send a notification message to the control terminal, indicating that the maturity level confirmed by the vegetation index of the preset harvesting area has not yet reached the harvesting requirements. The control terminal can then further prompt the server to initiate a harvesting command again after a first set time interval. Similarly, if the server confirms that the spectral similarity does not meet the preset threshold, it can also send a notification message to the control terminal, indicating that the maturity level confirmed based on spectral similarity has not yet reached the harvesting requirements. The control terminal can then further prompt the server to initiate a harvesting command again after a second set time interval. Likewise, if the server confirms that the second maturity level does not meet the preset second condition, it can also send a notification message to the control terminal, indicating that the maturity level confirmed based on plant electrical signals has not yet reached the harvesting requirements. The control terminal can then further prompt the server to initiate a harvesting command again after a third set time interval. The first set time interval > the second set time interval > the third set time interval.

[0066] In one implementation, the server performs dimensionality reduction on the target spectral data and determines the principal component features of the target spectral data; then, it determines the spectral similarity based on the similarity between the principal component features and the mature spectral features of the mature spectral data.

[0067] In one implementation, the server constructs a time-series model for the plant electrical signal, performs noise reduction and feature extraction on the plant electrical signal based on the time-series model to obtain the target electrical signal, and determines a second maturity level based on the mean square value and wavelet entropy of the target electrical signal. Specifically, the second maturity level is determined by judging whether the mean square value and wavelet entropy of the target electrical signal are within a set range; if both the mean square value and wavelet entropy are within the set range, the second maturity level is A; otherwise, the second maturity level is B. A preset second condition is also set: the second maturity level is A.

[0068] In one implementation, the server is used to fuse target spectral data and plant electrical signals using an attention mechanism, classify the fusion results, and obtain a result indicating whether or not to harvest.

[0069] In one implementation, the server is configured to generate a picking path based on the second maturity level, the distance the robot needs to move, and the robot's state; send an execution instruction including the picking path to the robot; and accordingly, the robot is configured to travel along the picking path and harvest from the preset harvesting area.

[0070] In one implementation, a robot is used to record the number, coordinates, picking time, and maturity information of the harvested target after harvesting, and then place the harvested target into a sterile processing chamber.

[0071] In one implementation, the server is used to collect historical harvesting data, and use a prediction model to predict future harvesting parameters based on the historical harvesting data; the harvesting parameters include harvesting area, harvesting order, and harvesting frequency; and / or, use historical harvesting data to update the prediction model.

[0072] As can be seen, this embodiment utilizes a control terminal, drone, edge computing node, server, and robot to collaboratively form a fully automated harvesting system with dual-dimensional detection of multispectral imaging and plant electrical signals. Specifically, the drone uses a multispectral camera to achieve rapid scanning of a large area of ​​the harvesting region. The edge computing node then performs image denoising and image stitching to reduce data transmission pressure, resulting in less and more concise image data, which is beneficial for the real-time performance and accuracy of subsequent image processing. For image processing, the server first determines the initial maturity level using multispectral data, then performs a secondary screening based on spectral similarity, and introduces plant electrical signals to detect the second maturity level. This continuous detection improves the accuracy of plant maturity judgment and reduces false positives and false negatives. Subsequently, the multispectral data and plant electrical signals are fused to confirm whether to harvest, further improving harvesting accuracy. After the maturity level is met and harvesting is confirmed, the server automatically issues an execution command, enabling the robot to perform precise harvesting. This achieves a fully automated and intelligent process from area inspection and maturity detection to automated harvesting, improving harvesting efficiency and adapting to the high-efficiency and precise harvesting needs of large-scale planting scenarios.

[0073] The following describes a harvesting method based on electrical signals and multispectral imaging provided by an embodiment of this application. The harvesting method based on electrical signals and multispectral imaging described below can be referred to in conjunction with other embodiments described herein.

[0074] This application discloses a harvesting method based on electrical signals and multispectral imaging, applied to a harvesting system based on electrical signals and multispectral imaging. The harvesting system includes: a control terminal, a drone, an edge computing node, a server, and a robot.

[0075] See Figure 2 As shown in the embodiments of this application, the harvesting method disclosed includes:

[0076] S201, The control terminal sends a harvesting command to the UAV.

[0077] S202. The UAV uses a multispectral camera to scan and obtain multispectral images of the pre-set harvesting area in the harvesting command, and transmits the multispectral images to the edge computing node.

[0078] S203. The edge computing node performs denoising and stitching on the multispectral image to obtain the target spectral data, and transmits the target spectral data to the server.

[0079] S204. The server calculates the first maturity based on the target spectral data. If the first maturity meets the preset first condition, the server determines the spectral similarity between the target spectral data and the mature spectral data. If the spectral similarity meets the preset threshold, the server sends an electrical signal detection command to the robot.

[0080] S205. The robot collects plant electrical signals within the preset harvesting area according to the electrical signal detection instructions and transmits the plant electrical signals to the server.

[0081] S206. The server calculates the second maturity level based on the plant's electrical signal. If the second maturity level meets the preset second condition, it generates a result on whether to harvest based on the target spectral data and the plant's electrical signal. If harvesting is confirmed based on the result, an execution command is sent to the robot.

[0082] S207. The robot harvests in the preset harvesting area according to the execution instructions.

[0083] In another implementation, a harvesting method based on electrical signals and multispectral data is applied to a server, which connects to a control unit, a drone, an edge computing node, and a robot.

[0084] Accordingly, the harvesting methods include:

[0085] The first maturity level is calculated based on the target spectral data transmitted by the edge computing node; wherein, the target spectral data is obtained by the edge computing node through denoising and stitching of the multispectral image; the multispectral image is obtained by the UAV responding to the harvesting command sent by the control terminal, using the UAV's multispectral camera to scan the preset harvesting area in the harvesting command;

[0086] If the first maturity meets the preset first condition, then the spectral similarity between the target spectral data and the mature spectral data is determined;

[0087] If the spectral similarity meets the preset threshold, an electrical signal detection command is sent to the robot so that the robot can collect plant electrical signals in the preset harvesting area according to the electrical signal detection command and transmit the plant electrical signals to the server.

[0088] The second maturity level was calculated based on plant electrical signals.

[0089] If the second maturity meets the preset second condition, then a result on whether to harvest is generated based on the target spectral data and plant electrical signals;

[0090] If harvesting is confirmed based on the results, an execution command is sent to the robot so that the robot can harvest the pre-set harvesting area according to the execution command.

[0091] For more detailed information on the working process of each device in this embodiment, please refer to the relevant content disclosed in other embodiments, which will not be repeated here.

[0092] As can be seen, this embodiment provides a harvesting method for collaborative operation of various devices, which can realize the unmanned and intelligent process from regional inspection and maturity detection to automated harvesting, thereby improving the degree of automated harvesting and harvesting accuracy.

[0093] In one implementation, a method such as Figure 3 The harvesting system shown includes: a control unit, drones, edge computing nodes, a data center (i.e., a server), and robots.

[0094] The control unit is equipped with a human-machine interface (HMI) app that supports remote control. Users can use the HMI to understand the farm's situation, issue harvesting instructions, adjust other equipment parameters, and check farm growth warnings and growth status maps.

[0095] The drone is used to control the multispectral camera during flight, collecting multispectral images of all the plants (such as leafy vegetables) on the farm. The drone can perform scheduled, fixed-point patrols, flying along a fixed grid at a height of five meters and a spacing of five meters, flying once every two hours per day.

[0096] Edge computing nodes are deployed on edge servers and can perform median filtering and image stitching on multispectral images acquired by multispectral cameras. The processed images are then transmitted to the data center via 5G.

[0097] The data center plans the optimal route based on the location of mature plants and transmits the data to the robot via 5G. Mature plants are then identified based on a combination of multispectral images and plant electrical signals.

[0098] The robot is used to perform harvesting tasks. Upon receiving an execution command, the robot moves along a route planned by the data center between the misting plants, harvesting one plant before moving on to the next. The overall execution route can be S-shaped. The minimum distance between robots is set at two meters, and one robot can harvest from a maximum of three misting plants. During its movement, the robot uses a camera mounted on its upper part to detect loose debris. Detected loose objects can be transmitted via Bluetooth to other robots, allowing them to change course and avoid the obstacles.

[0099] The process of the robot collecting plant electrical signals includes the following steps: After the robot reaches the designated plant location, it first takes multispectral photographs of each plant in the misting device using a camera mounted on its upper part. The multispectral images of each plant are then transmitted to the data center, which uses this data to identify the specific mature plants (the data center calculates the corresponding vegetation index for each plant based on the corresponding multispectral image and confirms maturity based on the vegetation index; see the first maturity calculation process for details). Then, the data center confirms the mature plants based on their current maturity level and transmits the location of the mature target plants to the robot. The robot then collects plant electrical signals from the target plants. Specifically, the robot extends its robotic arm to grasp the plant stem, and inserts a platinum needle with electrical signals into the stem to collect electrical signal values ​​that indicate maturity. These values ​​are transmitted back to the data center via 5G for analysis and judgment. If a plant is confirmed to be mature based on its electrical signal value, the data center combines the electrical signal value and the multispectral image to determine whether the plant should be harvested. If harvesting is confirmed, the data center transmits the location of the plant to be harvested to the robot, which then drives the ultrasonic cutting blade at the bottom of its robotic gripper to perform ultrasonic vibration cutting on the plant's roots. After harvesting, the robot transmits information such as the plant's maturity and location back to the data center. Based on this information, the data center grades the plant's quality and transmits the grade information back to the robot. Upon receiving the grade information, the robot activates its mechanical grippers to pick up the plant and place it into the corresponding grade's collection basket at the rear of the robot. For example, if the robot has three collection baskets: premium, grade one, and grade two, and the grade information is grade one, the robot will activate its mechanical grippers to pick up the plant and place it in the grade one collection basket. After harvesting all the plants in the current mist-planting device, the robot moves on to the next mist-planting device and repeats the above process.

[0100] After the robot completes all its work, it informs the data center. The data center analyzes the robot's completion rate based on its location and the overall harvesting process, and then sends a charging instruction and the optimal return route back to the robot. The robot then returns to charge and waits for its next mission.

[0101] This embodiment shows Figure 3 In the process, the multispectral camera carried by the drone is mainly used to take preliminary multispectral images of the plants in the base during the drone's flight; the information transmission module uses a wired connection to transmit the images taken by the drone to the data center.

[0102] The data center includes an information receiving module that receives information transmitted from other information sending modules; an information processing module that processes externally transmitted information and selects key information; an information decision-making module that is the decision-making part of the data center, analyzing and making decisions based on the processed information; a human-computer interaction module for regulators to control the base system; a data storage module that stores decision-making plans to facilitate autonomous learning by the model; and an information sending module that sends the plan information from the decision-making module.

[0103] The robot's information receiving module can receive decision and route information from the data center; the driving module enables the robot to move autonomously along the route; the camera module is used to capture multispectral images of the plants using a multispectral camera when the robot reaches a designated location, and transmit the images back to the data receiving module in the data center; the robot arm module can drive the mechanical gripper module to move; the mechanical gripper module is used to grip leafy vegetables and carries an electrical signal probe module and an ultrasonic cutting module; the electrical signal probe module is used to collect electrical signals from the plants; the ultrasonic cutting module can cut the plants; and the collection module can place the harvested plants into the corresponding collection box.

[0104] based on Figure 3 The harvesting system shown in this embodiment further provides the following harvesting method, including:

[0105] Step 1: Start the harvesting system, initialize all equipment and prepare to start working.

[0106] Specifically, harvesting instructions are issued through the farm management app, and the system automatically configures harvesting parameters. These parameters include the harvesting area, harvesting order, and harvesting frequency. This information is generated by the data center using predictive models based on historical harvesting data, plant multispectral images, maturity distribution maps, and user-preset parameters such as the harvesting area, order, and frequency. In one example, the harvesting area is a 10m x 10m zone, the harvesting order is from left to right, and the harvesting frequency is twice daily.

[0107] Step 2: Multispectral image acquisition and processing.

[0108] The drone, equipped with a multispectral camera, scans the plant at preset intervals or according to environmental changes to acquire multispectral image data. The multispectral camera has a resolution of 640×480, a spectral range of 400-1000nm, and a scanning height of 5-30 meters; alternatively, the resolution can be set to 640×480, the spectral range to 400-1000nm, and the scanning height to 10 meters. The multispectral camera acquires blue, green, red, red-edge, and near-infrared spectral data of the plant. Spectral data acquisition uses an integrating sphere diffuse standard object as a reference, with image acquisition angles of 0°, 45°, and 80°, and three acquisitions at each angle; the average value is used as the final data.

[0109] Edge computing nodes perform noise filtering and stitching on multispectral image data. A median filtering algorithm is used to filter the multispectral image in each band to remove noise, and then a stitching method is used to stitch the panoramic multispectral images together.

[0110] The data center calculates vegetation indices on the processed multispectral images and uses a preset threshold of 0.7 for assessment. Vegetation indices above this threshold are considered preliminarily mature. Principal component analysis (PCA) is then used to extract the main information from the multispectral data, retaining 90% of the information content to obtain 14 principal components as feature vectors. These are then used for dimensionality reduction and data simplification in the multispectral images. Subsequently, the SAM spectral angle mapping method is used to calculate the spectral similarity between the feature vectors and the spectra of mature plants to further assess plant maturity. The spectral similarity between the feature vectors and the spectra of mature plants can be calculated using the formula... Calculate the included angle. The smaller the value, the more similar the spectra. The spectral characteristics of mature plants can be represented as Y = (y1, y2, y3... y...). n The spectral characteristics (i.e., eigenvectors) of the plant under test can be represented as X = (x1, x2, x3, ... x... n The similarity is assessed by calculating the angle between two features, that is, the angle between the two features. The smaller the angle, the more similar they are.

[0111] The PCA principal component analysis method can be represented by the formula Y=XW, where X is the matrix identifier of the original multispectral data, W is the matrix representation of the eigenvectors, and Y is the principal component score matrix after dimensionality reduction, which is used to extract the main variation features in the spectral data.

[0112] Step 3: Acquisition and analysis of plant electrical signals.

[0113] When the spectral similarity condition is met, the data center controls the robot to collect plant electrical signals. The robot first uses a multispectral camera to precisely locate the plant from which electrical signals need to be collected. Then, the robotic arm extends and extends a plant electrical signal probe, gently touching the base of the plant stem to collect the electrical signals. The robotic arm is a 6-axis articulated arm with a repeatability accuracy of ±0.1 mm and a gripping force of 50 N. The probe is made of platinum needles, with a diameter of 0.5 mm and a length of 10 mm, coated with an insulating layer. The collected electrical signals from the same plant are analyzed using an ARIMA time series model. Specifically, an ARIMA(2,1,2) model is used, with an intercept term of 0.5, a difference order of 1, and a maximum order of 2 for both the autoregressive and moving average terms. Alternatively, an ARIMA(1,1,1) model can be used, with an intercept term of 0.3, a difference order of 1, and a maximum order of 1 for both the autoregressive and moving average terms.

[0114] The ARIMA time series model can be represented as: B is the shift operator. and These are autoregressive and moving average polynomials, respectively; d is the difference order, used for time-series analysis and prediction of plant electrical signals; X t Let X represent the electrical signal collected from the plant at time t, where t represents the collection time. Electrical signals collected from the same plant at different times can be represented as: X1, X2, ..., X t .

[0115] After data collection, the plant electrical signals are transmitted to the data center for analysis: noise reduction and feature extraction are performed using wavelet transform. A db3 wavelet is used with a scale parameter a=2 and a translation parameter b=0 to perform continuous wavelet transform, decomposing the electrical signals. Then, the mean square value and wavelet entropy of the electrical signals are calculated to determine the plant's maturity state. The mean square value of the electrical signal is set to a range of 0.1-1.0, and the wavelet entropy is set to a range of 2.0-5.0. Maturity is defined as a mean square value greater than 0.6 and a wavelet entropy less than 3.5. Alternatively, the mean square value can be set to a range of 0.2-0.8, and the wavelet entropy to 2.5-4.0; maturity is defined as a mean square value greater than 0.5 and a wavelet entropy less than 3.0.

[0116] Step 4: Picking decision and sorting.

[0117] The data center employs an attention-based feature fusion algorithm that weights and fuses feature vectors and denoised electrical signals, and then makes a decision on whether to extract features based on the fused features. This algorithm uses an 8-head multi-head attention mechanism (256-dimensional hidden layers), with inputs of 14-dimensional feature vectors and electrical signal features (mean square value, wavelet entropy, etc.). A ResNet50 network is used as the base network, freezing the weights of the bottom convolutional layers and creating a new fully connected layer for classification. The decision on whether to extract features is based on the classification results.

[0118] Feature fusion based on the attention mechanism can be expressed as: Q, K, and V are the query, key, and value matrices, respectively, and d k Using the key vector dimension, this model can dynamically adjust the importance weights of spectral features and electrical signal features. Specifically, by encoding and transforming the plant spectral features (14-dimensional feature vector) and electrical signal features, the corresponding Q matrix, K matrix, and V matrix can be obtained.

[0119] In one implementation, the data center can also sort the harvested plants based on the peak value of their electrical signals. The peak value ranges from 10 to 100 μV, and normalization is used during sorting, taking the reciprocal of the peak value as the sorting criterion. The sorting results are used to determine the grade of each plant; for example, the first N plants in the sequence correspond to premium grade, the (N+1)th to (2N)th plants correspond to grade one, and the (2N+1)th to (4N)th plants correspond to grade two. It should be noted that the larger the peak value of the electrical signal (10–100 μV), the more abundant the nutrient accumulation, and the higher the plant's ranking.

[0120] Step 5: Path planning and execution.

[0121] The data center calculates priorities based on plant maturity, the robot's remaining distance, and the robot's status. The priority calculation formula is: Priority = 0.4 × Maturity + 0.3 × Distance + 0.3 × Robot Load. Based on this priority, the optimal path is determined on the planting base map. This path is sent to the robot, enabling it to perform harvesting tasks according to this path, and to avoid obstacles and locate itself in real time. The robot's maximum travel speed is 2.0 m / s, and obstacle avoidance uses a dynamic obstacle avoidance algorithm based on local programming.

[0122] Step 6: Data feedback and optimization.

[0123] The robot records harvested plant information and location data, including plant number, coordinates, harvest time, maturity, etc. The plants are then placed in a sterile treatment chamber, which uses ultraviolet light for 30 seconds for disinfection. All harvest-related data is transmitted back to the data center in real time. The data center uses this received historical harvest information to train a predictive model online, enabling the model to predict the parameters for the next harvest. For example, the predictive model might predict the next harvest time as 8:00-12:00 and 14:00-18:00.

[0124] As can be seen, this embodiment, by combining a multispectral camera and an electrical signal probe for data acquisition, can accurately identify and locate mature plants, improving the accuracy and quality of harvesting. Furthermore, by prioritizing the harvesting of plants with optimal nutrient accumulation through electrical signal peak values, the harvesting order is optimized, improving harvesting efficiency while ensuring the uniformity and quality stability of the harvested plants. Since the electrical signal detection is conducted through contact, it is unaffected by external environmental factors such as ambient light and humidity, thus exhibiting excellent resistance to environmental interference and ensuring the accuracy and reliability of the detection. The solution uses an app to issue harvesting commands, and the robot autonomously navigates to the target area, achieving unattended intelligent harvesting, significantly reducing the safety risks of manual operation and improving operational efficiency.

[0125] The following describes an electronic device provided by an embodiment of this application. The electronic device described below can be referred to in conjunction with other embodiments described herein. The electronic device in this embodiment may be a control terminal, a drone, an edge computing node, a server, or a robot, etc.

[0126] See Figure 4 As shown in the figure, an embodiment of this application discloses an electronic device, including:

[0127] Memory 401 is used to store computer programs;

[0128] Processor 402 is configured to execute the computer program to implement the method disclosed in any of the above embodiments.

[0129] Furthermore, embodiments of this application also provide an electronic device. The aforementioned electronic device can be, for example,... Figure 5 The server shown can also be as follows: Figure 6 The terminal shown. Figure 5 and Figure 6 These are all diagrams illustrating the structure of an electronic device according to an exemplary embodiment. The content in the diagrams should not be considered as any limitation on the scope of this application.

[0130] Figure 5 This is a schematic diagram of a server provided in an embodiment of this application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory stores a computer program, which is loaded and executed by the processor to implement the relevant steps in the harvesting process disclosed in any of the foregoing embodiments.

[0131] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0132] In addition, the memory, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system, computer programs and data, etc., and the storage method can be temporary storage or permanent storage.

[0133] The operating system manages and controls the various hardware devices and computer programs on the server to enable the processor to perform operations and processes on the data in the memory. It can be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs capable of performing the acquisition methods based on electrical signals and multispectral imaging disclosed in any of the foregoing embodiments, the computer programs may further include computer programs capable of performing other specific tasks. The data may include application update information and application developer information.

[0134] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer.

[0135] Typically, the terminal in this embodiment includes a processor and a memory.

[0136] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and coprocessors. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which handles computational operations related to machine learning.

[0137] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory is used to store at least the following computer program, which, after being loaded and executed by the processor, is capable of implementing the relevant steps in the acquisition method based on electrical signals and multispectral imaging disclosed in any of the foregoing embodiments, executed by the terminal side. In addition, the resources stored in the memory may also include operating systems and data, and the storage method may be temporary or permanent storage. The operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, application update information.

[0138] In some embodiments, the terminal may further include a display screen, an input / output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0139] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than illustrated.

[0140] The following describes a non-volatile storage medium provided in an embodiment of this application. The non-volatile storage medium described below can be referred to in conjunction with other embodiments described herein.

[0141] A non-volatile storage medium is provided for storing a computer program, wherein the computer program, when executed by a processor, implements the acquisition method based on electrical signals and multispectral imaging disclosed in the foregoing embodiments. The non-volatile storage medium is a computer-readable non-volatile storage medium, which, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon include an operating system, computer programs, and data, and the storage method can be temporary or permanent storage.

[0142] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0143] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed acquisition method based on electrical signals and multispectral imaging.

[0144] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments.

[0145] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0146] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of non-volatile storage medium known in the art.

[0147] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data acquisition system based on electrical signals and multispectral imaging, characterized in that, include: The control unit is used to send harvesting instructions to the drone; The drone is used to scan a pre-set harvesting area in the harvesting command using a multispectral camera, and transmit the multispectral image to an edge computing node. The edge computing node is used to denoise and stitch the multispectral image to obtain target spectral data, and transmit the target spectral data to the server. The server is used to calculate the first maturity level based on the target spectral data; If the first maturity meets the preset first condition, then the spectral similarity between the target spectral data and the mature spectral data is determined. If the spectral similarity meets the preset threshold, then an electrical signal detection command is sent to the robot. The robot is used to collect plant electrical signals within the preset harvesting area according to the electrical signal detection instruction, and transmit the plant electrical signals to the server. The server is used to calculate the second maturity level based on the plant's electrical signal; If the second maturity meets the preset second condition, a result on whether to harvest is generated based on the target spectral data and the plant electrical signal; if harvesting is confirmed based on the result, an execution command is sent to the robot. The robot is used to harvest in the preset harvesting area according to the execution instructions.

2. The acquisition system based on electrical signals and multispectral imaging according to claim 1, characterized in that, The server is used to calculate a vegetation index based on the target spectral data, and to determine the first maturity level based on the vegetation index.

3. The acquisition system based on electrical signals and multispectral imaging according to claim 1, characterized in that, The server is used to perform dimensionality reduction processing on the target spectral data, determine the principal component features of the target spectral data, and determine the spectral similarity based on the similarity between the principal component features and the mature spectral features of the mature spectral data.

4. The acquisition system based on electrical signals and multispectral imaging according to claim 1, characterized in that, The server is used to construct a time series model for the plant electrical signal, and to perform noise reduction and feature extraction on the plant electrical signal based on the time series model to obtain the target electrical signal; The second maturity level is determined based on the mean square value and wavelet entropy of the target electrical signal.

5. The acquisition system based on electrical signals and multispectral imaging according to claim 1, characterized in that, The server is used to fuse the target spectral data and the plant electrical signal using an attention mechanism, classify the fusion results, and obtain a result indicating whether or not to harvest.

6. The acquisition system based on electrical signals and multispectral imaging according to claim 1, characterized in that, The server is used to generate a picking path based on the second maturity level, the distance the robot needs to move, and the robot's status; and to send an execution instruction including the picking path to the robot. Accordingly, the robot is used to travel along the harvesting path and harvest from the preset harvesting area.

7. The acquisition system based on electrical signals and multispectral imaging according to claim 1, characterized in that, The robot is used to record the number, coordinates, picking time, and maturity information of the harvested target after harvesting, and then place the harvested target into a sterile processing chamber.

8. The acquisition system based on electrical signals and multispectral imaging according to any one of claims 1 to 7, characterized in that, The server is used to collect historical harvesting data and use a prediction model to predict future harvesting parameters based on the historical harvesting data; the harvesting parameters include harvesting area, harvesting order, and harvesting frequency; and / or, use the historical harvesting data to update the prediction model.

9. A harvesting method based on electrical signals and multispectral analysis, characterized in that, It is applied to the server side, which connects the control terminal, drone, edge computing node and robot; Accordingly, the harvesting method includes: The first maturity level is calculated based on the target spectral data transmitted by the edge computing node; wherein, the target spectral data is obtained by the edge computing node through denoising and stitching of a multispectral image; the multispectral image is obtained by the UAV responding to the harvesting command sent by the control terminal, using the multispectral camera of the UAV to scan the preset harvesting area in the harvesting command; If the first maturity meets the preset first condition, then the spectral similarity between the target spectral data and the mature spectral data is determined; If the spectral similarity meets a preset threshold, an electrical signal detection command is sent to the robot so that the robot collects plant electrical signals in the preset harvesting area according to the electrical signal detection command and transmits the plant electrical signals to the server. The second maturity level is calculated based on the plant's electrical signals; If the second maturity meets the preset second condition, a result on whether to harvest is generated based on the target spectral data and the plant electrical signal; If harvesting is confirmed based on the results, an execution command is sent to the robot so that the robot can harvest the preset harvesting area according to the execution command.

10. A non-volatile storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the method as described in claim 9.

Images