A fruit evaluation method, device and inspection equipment

Through deep camera and neural network technology, the fruit size and defect proportion are automatically estimated, which solves the problems of low fruit grading efficiency and insufficient accuracy in the existing technology, and achieves high-performance evaluation and market price prediction.

CN113393421BActive Publication Date: 2024-11-26SHENZHEN FENGNONG SHUZHI AGRI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110499315.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-08
Publication Date
2024-11-26
Estimated Expiration
2041-05-08

AI Technical Summary

Technical Problem

The existing fruit grading methods rely on manual operations, are inefficient and have insufficient accuracy, and require reference objects to estimate fruit size, which limits the application effect of image recognition.

Method used

Use a depth camera to collect the depth image to be measured, and pixel segmentation is performed through neural network to obtain the fruit boundary and defect part boundary. Combined with the coordinate conversion relationship of the depth camera, calculate the fruit size and defect part proportion, and realize automatic estimation of the fruit size and defect degree .

Benefits of technology

It realizes accurate estimation of the fruit size and defect proportion before fruit picking, improves identification accuracy and efficiency, saves manpower and time costs, and is suitable for market price and sales plan prediction before fruit picking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113393421B_ABST
    Figure CN113393421B_ABST
Patent Text Reader

Abstract

The present application is applicable to the field of image recognition technology, and provides a fruit evaluation method, device and patrol equipment. The method includes: collecting a depth image to be measured by a depth camera; performing pixel segmentation on the depth image to be measured to obtain the fruit boundary and the defective part boundary of the fruit; and calculating the fruit size and defective part ratio of each fruit based on the fruit boundary and the defective part boundary. The present application uses a depth camera to take pictures and collect images, and can accurately calculate the actual size of the fruit and its defective part without the need for a reference object, and uses pixel segmentation to distinguish the fruit and its defective area from other image areas. The size and defect ratio of the fruit can be estimated before the fruit is picked, which is convenient for determining the market price and sales plan, and predicting economic benefits; the recognition accuracy is high, the efficiency is high, and it saves manpower and time costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of image recognition technology, and in particular relates to a fruit evaluation method, device and patrol equipment. Background Art

[0002] The current fruit sorting system is mainly for sorting and fruit grading evaluation after harvesting. The actual economic benefits can only be obtained by counting the yield based on the sorting results. In terms of fruit grading, it mainly relies on manual labor, and the five-point sampling statistical estimation is adopted based on experience, which is very labor-intensive, inefficient and inaccurate.

[0003] With the development of image technology, some studies on in vivo fruit grading are currently being conducted through image recognition, which estimates the fruit diameter size based on the calibration of reference objects (wooden rulers, etc.). This has very strict restrictions on the reference object, such as: the reference object must have a measurable unit, that is, the width and height of the reference object must be known; the reference object can be easily distinguished based on the position of the target object or its appearance. This method is not only complicated to operate, but also when the target object and the reference object are not located at the same depth of field in the photo, the reference object loses its reference value, because the distance between the camera and the target object is different, and the size of the target object in the photo is very different. In practical applications, the effect is not ideal. Summary of the invention

[0004] The embodiments of the present application provide a fruit evaluation method, device and patrol equipment, which can obtain the size and yield of the fruit before the fruit is picked, facilitate the determination of market price and sales plan, and predict economic benefits.

[0005] In a first aspect, the present application provides a fruit evaluation method, comprising:

[0006] Collect the depth image to be measured through the depth camera;

[0007] Performing pixel segmentation on the depth image to be measured to obtain the boundary of the fruit and the boundary of the defective part of the fruit;

[0008] The fruit size and defective portion ratio of each fruit are calculated based on the fruit boundary and the defective portion boundary.

[0009] Furthermore, before performing pixel segmentation on the depth image to be measured, the method further includes:

[0010] Training the neural network through the depth image samples to obtain a trained neural network model;

[0011] The depth image samples include fruit image samples and defective part image samples;

[0012] The pixel segmentation of the depth image to be measured includes: performing pixel-level semantic segmentation on the depth image to be measured using the trained neural network model.

[0013] Specifically, the fruit size and defective portion ratio of each fruit are calculated based on the fruit boundary and the defective portion boundary, including:

[0014] Calibrate the depth camera to obtain a conversion relationship from a camera coordinate system of the depth camera to a world coordinate system;

[0015] Calculating the fruit size of each fruit according to the conversion relationship and the fruit boundary;

[0016] Calculating the size of the defective part of each fruit according to the conversion relationship and the defective part boundary;

[0017] The defective portion ratio is calculated based on the fruit size and the defective portion size of each fruit.

[0018] As a possible implementation, the neural network is trained using deep image samples, including:

[0019] Converting the depth image samples into RGB image samples by linear rescaling, and training a neural network by using the RGB image samples to obtain a first neural network model;

[0020] Depth channel data samples are extracted from the depth image samples, and a neural network is trained using the depth channel data samples to obtain a second neural network model.

[0021] Accordingly, using the trained neural network model to perform pixel-level semantic segmentation on the depth image to be measured includes:

[0022] The depth image to be measured is converted into an RGB image by linear rescaling; and the RGB image is semantically segmented at the pixel level by using the first neural network model;

[0023] Alternatively, extract depth channel data from the depth image to be measured, and use the second neural network model to perform pixel-level semantic segmentation on the depth channel data.

[0024] Alternatively, the trained neural network model is used to perform pixel-level semantic segmentation on the depth image to be measured to obtain the boundary of the fruit and the boundary of the defective part of the fruit, including:

[0025] The depth image to be measured is converted into an RGB image by linear rescaling; the RGB image is semantically segmented at the pixel level by using the first neural network model to obtain a first recognition result of a fruit boundary and a defective part boundary of the fruit;

[0026] And, extracting depth channel data from the depth image to be measured, performing pixel-level semantic segmentation on the depth channel data using the second neural network model, and obtaining a second recognition result of a fruit boundary and a defective part boundary of the fruit;

[0027] The first recognition result and the second recognition result are mutually verified, and / or the first recognition result and the second recognition result are merged.

[0028] Among them, the neural network is trained through deep image samples, including:

[0029] Annotating a fruit target frame and defective areas in the depth image sample;

[0030] The neural network model extracts and learns features of the fruit target frame and the defective area.

[0031] Furthermore, after calculating the fruit size and defective portion ratio of each fruit, it also includes:

[0032] determining a fruit grade based on the fruit size and a preset size threshold;

[0033] The yield is estimated based on the fruit grade and the percentage of defective parts.

[0034] In a second aspect, an embodiment of the present application provides a fruit evaluation device, comprising:

[0035] A depth camera module, used to collect the depth image to be measured;

[0036] A neural network module is used to perform pixel segmentation on the depth image to be measured to obtain the boundary of the fruit and the boundary of the defective part of the fruit;

[0037] The size calculation module is used to measure the fruit size and defective part ratio of each fruit according to the fruit boundary and the defective part boundary.

[0038] In a third aspect, an embodiment of the present application provides a patrol device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, and also comprising a depth camera and a mobile device;

[0039] The mobile device is used to drive the patrol device to the destination position;

[0040] The depth camera is used to obtain a depth image of the target location and send it to the processor;

[0041] When the processor executes the computer program, the fruit evaluation method as described above is implemented.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the fruit evaluation method as described above is implemented.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a patrol device, enables the patrol device to execute the fruit evaluation method described in any one of the first aspects above.

[0044] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0045] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by performing pixel segmentation on the depth image to be measured, the boundaries of the fruit and the boundaries of the defective parts of the fruit are obtained, and then the actual fruit size and the proportion of the defective parts of each fruit are calculated based on the boundaries of the fruit and the boundaries of the defective parts and the working principle of the depth camera. The present application uses a depth camera to take pictures and collect images, and can accurately calculate the actual size of the fruit and its defective parts without a reference object, and uses pixel segmentation to distinguish the fruit and its defective area from other image areas. The size and defective proportion of the fruit can be estimated before the fruit is picked, which is convenient for determining the market price and sales plan, and predicting economic benefits; the recognition accuracy is high, the efficiency is high, and the manpower and time costs are saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flowchart of a fruit evaluation method provided in one embodiment of the present application;

[0048] Figure 2 is a flow chart of a fruit evaluation method provided in another embodiment of the present application;

[0049] Figure 3 It is a schematic diagram of the architecture of two semantic segmentation methods provided in one embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of a network structure of separation aggregation gating provided in an embodiment of the present application;

[0051] Figure 5 is a structural schematic diagram of a fruit evaluation device provided in one embodiment of the present application;

[0052] Figure 6 It is a structural schematic diagram of a patrol device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0054] It should be understood that the term “and / or” used in the specification and appended claims of this application refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0055] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0056] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0057] The fruit evaluation method provided in the embodiment of the present application is suitable for scenarios in which fruit is graded and the yield of fruit trees is estimated before being picked in agriculture. The method is performed by a fruit evaluation device, which is composed of software and hardware and is wholly or partially integrated into equipment such as drones, unmanned vehicles, and patrol vehicles.

[0058] Figure 1 Schematic diagram of the fruit evaluation method provided in this embodiment. Figure 1As shown, the fruit evaluation method comprises the following steps:

[0059] S11, collecting a depth image to be measured through a depth camera.

[0060] The depth camera takes pictures of the fruit trees, and one or more depth images to be measured represent the fruit condition of a tree.

[0061] The depth camera is used to obtain an RGB-D depth image (RGB & Depth Map, an RGB image containing depth information). The depth image to be tested includes depth data, which reflects the distance from the object to the depth camera.

[0062] Methods for depth cameras to obtain depth data include structured light method, binocular stereo vision method and time of flight (TOF) method. Through any of these methods, the depth coordinates of each pixel on the depth image can be obtained.

[0063] The structured light method uses a near-infrared laser to project light with certain structural characteristics onto the object being photographed, and then a special infrared camera collects the light. This light with a certain structure will produce different image phase changes due to different depth areas of the object being photographed, and this structural change is converted into depth information. Structured light includes but is not limited to stripe structured light, Gray code structured light, speckle structured light, etc.

[0064] The time-of-flight method measures the time it takes light to fly to obtain distance. Specifically, it continuously emits laser pulses to the object, then uses a sensor to receive the light pulses reflected from the object, and detects the round-trip flight time of the light pulses to obtain the exact distance to the target.

[0065] Binocular stereo vision is an important form of machine vision. It is based on the principle of parallax and uses two cameras of an imaging device to obtain two images of the object under test from different positions. The three-dimensional geometric information of the object is obtained by calculating the position deviation between corresponding points in the images.

[0066] Before acquiring the depth image to be measured, the depth camera needs to be calibrated for internal and external parameters to obtain the transformation relationship from the camera coordinate system of the depth camera to the world coordinate system. The process is as follows:

[0067] A calibration object is set in front of the depth camera and a photo is taken. The calibration object includes multiple marking points. The (x, y) coordinates of each marking point on the image and the depth coordinates are combined to obtain the image coordinates of each marking point in the 3D camera coordinate system; the actual distances between the marking points and between the marking points and the depth camera are measured to obtain the actual coordinates of the marking points in the 3D world coordinate system; based on the image coordinates and the actual coordinates, the transformation relationship from the 3D camera coordinate system to the 3D world coordinate system is calculated, which is usually expressed as a transformation matrix.

[0068] After the calibration is completed, the depth camera can collect the depth image to be measured without the calibration object, and calculate the actual size of the object from the image size of the object on the depth image to be measured according to the conversion matrix.

[0069] S12, performing pixel segmentation on the depth image to be measured to obtain a fruit boundary and a boundary of a defective part of the fruit.

[0070] The neural network is trained by deep image samples to obtain a trained neural network model; the deep image samples include fruit image samples and defective part image samples. All deep image samples are strictly identified and classified, and duplicate and ambiguous situations are removed, so that the sample quality is guaranteed.

[0071] In order to improve the recognition effect of the neural network model on the image, the samples will be data enhanced before the model training to make the model more generalized and robust. The data enhancement methods include but are not limited to random jitter scaling of the depth image samples, changes in image brightness, HSV (Hue, Saturation, Value), random horizontal flipping and rotation, etc.

[0072] The fruit target frame and defective area are marked in the depth image sample; the defective area includes but is not limited to disease spots and surface damage caused by pests and diseases. The neural network model extracts and learns features of the fruit target frame and the defective area.

[0073] Neural network models have excellent feature extraction capabilities in images or videos. The extracted features can be used for image classification, target detection, and pixel-level segmentation of targets. Pixel-level segmentation can be semantic segmentation or instance segmentation. Commonly used neural network models include deep neural network models and convolutional neural network models.

[0074] The core of semantic segmentation is the encoder-decoder structure. The encoder refers to a feature extraction network based on deep learning, including but not limited to CSPDarknet53, EfficientNet_B0-B7, Resnest50, MobileNet_V3, GhostNet, ShuffleNet_V2, etc. The task of the decoder is to project the recognition feature semantics learned by the encoder to the pixel space through upsampling to obtain dense classification.

[0075] In this embodiment, the trained deep neural network model is used to perform pixel-level semantic segmentation on the depth image to be measured, so that the boundaries of the fruit and the boundaries of the defective parts of the fruit can be accurately identified.

[0076] S13, calculating the fruit size and defective portion ratio of each fruit according to the fruit boundary and the defective portion boundary.

[0077] The depth camera is calibrated through the aforementioned steps to obtain the conversion relationship from the camera coordinate system of the depth camera to the world coordinate system; the fruit size of each fruit is calculated based on the conversion relationship and the fruit boundary; the size of the defective part of each fruit is calculated based on the conversion relationship and the defective part boundary; the proportion of the defective part is calculated based on the fruit size and the defective part size of each fruit.

[0078] Specifically, the size of the fruit on the depth image to be measured can be obtained according to the fruit boundary, and the fruit size on the image is converted into the actual fruit size in combination with the conversion relationship; the defect part is similar. The defect part ratio is usually the area ratio.

[0079] This embodiment uses machine learning modeling to identify images containing fruits, and realizes in vivo evaluation of fruits, including fruit size, fruit surface damage and degree of defects; on this basis, the fruit size can be graded to assist orchard growers in quickly calculating the number of fruits on a single tree or even the entire orchard, guiding agricultural production practices, and can be used to accurately predict and calculate the economic benefits and yield of fruits in the planting area before harvesting, as well as to evaluate the level of orchard management, thereby improving work efficiency and avoiding the subjective differences in manual grading.

[0080] Based on the above embodiments, the image recognition process of the neural network model is optimized, such as Figure 2 As shown, the method steps are as follows:

[0081] S21, collecting a depth image to be measured through a depth camera.

[0082] S22, training the neural network through the depth image samples to obtain a trained neural network model.

[0083] The depth image samples are converted into RGB image samples through linear rescaling, and a neural network is trained through the RGB image samples to obtain a first neural network model.

[0084] The linear rescaling of the depth image into an RGB image is specifically as follows: the depth image is re-encoded using three channels, namely the horizontal disparity (H), the height above the ground (H) and the angle of the surface normal vector (A). This encoding method is called HHA, which is a representation of depth information; then the three-channel data obtained by HHA encoding is linearly rescaled to 0-255 and converted into an RGB image.

[0085] Alternatively, depth channel data samples may be extracted from the depth image samples, and a neural network may be trained using the depth channel data samples to obtain a second neural network model.

[0086] According to the need of recognition accuracy, one or both of the first neural network model and the second neural network model can be selected for training.

[0087] S23, using the trained neural network model to perform pixel-level semantic segmentation on the depth image to be measured, to obtain the boundary of the fruit and the boundary of the defective part of the fruit.

[0088] The HHA-encoded depth image to be measured is converted into an RGB image through linear recalibration; the RGB image is semantically segmented at the pixel level using the first neural network model to obtain a first recognition result of the fruit boundary and the defective part boundary of the fruit.

[0089] Depth channel data is extracted from the depth image to be measured, and pixel-level semantic segmentation is performed on the depth channel data using the second neural network model to obtain a second recognition result of a fruit boundary and a boundary of a defective part of the fruit.

[0090] If one of the first neural network model and the second neural network model is trained in step S22, a first recognition result or a second recognition result is generated accordingly; when the fruit characteristics are obvious or the recognition accuracy requirement is not high, one of the neural network models can be selected to obtain the corresponding recognition result.

[0091] If the first neural network model and the second neural network model are trained in step S22, such as Figure 3 As shown, the first recognition result and the second recognition result may be mutually verified, and / or the first recognition result and the second recognition result may be fused. The recognition result after verification and fusion is more accurate and has higher precision.

[0092] The verification and fusion of the first recognition result and the second recognition result can be achieved by using separation aggregation gating. Figure 4 As shown in , the separation and aggregation gate performs two operations to ensure that the feature information is propagated between the modes, namely the separation gate function and the aggregation gate function. Figure 4 The module (shown by the dotted box on the left) is a module that recalibrates the features of each single modality. Due to the physical characteristics of the depth sensor, the depth information will be mixed with a lot of noise signals. The noise signal is first filtered through the separation gate network, and then the high confidence information in the RGB information is used to filter out abnormal depth information of the same level. Specific operation: First, the global spatial information of the two modalities is fused, and then global average pooling is performed along the channel direction to realize the cross-modal attention feature vector. Finally, the cross-modal attention feature vector is multiplied by each single modality to obtain depth information with lower noise. Aggregation gate (such as Figure 4 The dotted box on the right is a cross-modal feature aggregation module. RGB information and depth information are highly complementary at various locations in space. The aggregation gate network can fuse these two modal features through the spatial attention mechanism to enhance the feature recognition of the target feature.

[0093] Separation aggregation gating can help better fuse the features between the two recognition results, effectively recalibrate the RGB features, and extract accurate depth information, thereby improving the performance of RGB-D semantic segmentation.

[0094] S24, calculating the fruit size and defective portion ratio of each fruit according to the fruit boundary and the defective portion boundary.

[0095] S25, performing fruit grading and yield estimation according to the fruit size and the proportion of defective parts.

[0096] Different fruits are pre-classified according to the threshold range of fruit diameter (such as extra-large fruit, large fruit, medium fruit and small fruit). The fruit size is judged to belong to which threshold range, and then classified into the corresponding level of the threshold range. The average weight of a single fruit of each level can be known through the statistics of previously harvested fruits.

[0097] Good and bad fruits are divided according to the threshold of the defective part ratio. Combining the grade, average weight and quality of the fruit, the yield of a single fruit tree or even the entire orchard can be estimated.

[0098] Corresponding to the fruit evaluation method described in the above embodiment, Figure 5 A schematic diagram of the structure of a fruit evaluation device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0099] Reference Figure 5 , the device comprises:

[0100] The depth camera module is used to acquire the depth image to be measured.

[0101] The neural network module is used to perform pixel segmentation on the depth image to be measured to obtain the boundary of the fruit and the boundary of the defective part of the fruit.

[0102] The size calculation module is used to measure the fruit size and defective part ratio of each fruit according to the fruit boundary and the defective part boundary.

[0103] The depth camera module is generally installed on movable patrol equipment such as drones, automatic guided vehicles (AGVs), and patrol cars. The neural network module and the size calculation module can be installed on the patrol equipment, or on smart terminals, servers, and other equipment that are connected to the patrol equipment for communication.

[0104] The device also includes a yield estimation module, which is used to grade the fruits according to the fruit size and the proportion of defective parts of each fruit, and further estimate the yield.

[0105] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0106] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0107] Figure 6 This is a schematic diagram of the structure of the patrol device provided in one embodiment of the present application. Figure 6 As shown, the patrol device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and also includes a depth camera and a mobile device.

[0108] The mobile device is used to drive the patrol device to the destination, such as the flight device of a drone and the power device of an unmanned vehicle.

[0109] The depth camera is used to obtain a depth image of a target location and send the image to the processor.

[0110] When the processor executes the computer program, the fruit evaluation method described in the above embodiment is implemented.

[0111] In other embodiments, the processor of the patrol device performs tasks such as path planning and drive control. The patrol device also includes a wireless communication device, which realizes the communication connection between the patrol device and the server. After the depth camera obtains the depth image of the destination location, it sends it to the server through the wireless communication device, and the server executes the fruit evaluation method described in the above embodiment.

[0112] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0113] In some embodiments, the memory may be an internal storage unit of the patrol device, such as a hard disk or a memory. In other embodiments, the memory may also be an external storage device of the patrol device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory may include both an internal storage unit of the patrol device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0114] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0115] An embodiment of the present application provides a computer program product. When the computer program product runs on a fruit evaluation device, the fruit evaluation device can implement the steps in the above-mentioned method embodiments when executed.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A fruit evaluation method, characterized in that: include: Collect the depth image to be measured through the depth camera; The depth image to be measured is segmented into pixels to obtain the boundary of the fruit and the boundary of the defective part of the fruit, including: converting the depth image to be measured into an RGB image by linear rescaling; performing pixel-level semantic segmentation on the RGB image using a first neural network model to obtain a first recognition result of the boundary of the fruit and the boundary of the defective part of the fruit; and extracting depth channel data from the depth image to be measured, performing pixel-level semantic segmentation on the depth channel data using a second neural network model to obtain a second recognition result of the boundary of the fruit and the boundary of the defective part of the fruit; verifying the first recognition result with the second recognition result, and / or fusing the first recognition result with the second recognition result; wherein the verification and fusion of the first recognition result with the second recognition result are achieved by using separation aggregation gating, including: fusing the global spatial information of the two modalities, performing global average pooling along the channel direction to achieve a cross-modal attention feature vector, and obtaining depth information by multiplying the cross-modal attention feature vector with each single modality; the aggregation gate is a cross-modal feature aggregation module; The fruit size and defective portion ratio of each fruit are calculated based on the fruit boundary and the defective portion boundary.

2. The fruit evaluation method according to claim 1, characterized in that Before performing pixel segmentation on the depth image to be measured, the method further includes: Training the neural network through the depth image samples to obtain a trained neural network model; The depth image samples include fruit image samples and defective part image samples; The pixel segmentation of the depth image to be measured includes: performing pixel-level semantic segmentation on the depth image to be measured using the trained neural network model.

3. The fruit evaluation method according to claim 1, characterized in that The fruit size and defective portion ratio of each fruit are calculated based on the fruit boundary and the defective portion boundary, including: Calibrate the depth camera to obtain a conversion relationship from a camera coordinate system of the depth camera to a world coordinate system; Calculating the fruit size of each fruit according to the conversion relationship and the fruit boundary; Calculating the size of the defective part of each fruit according to the conversion relationship and the defective part boundary; The defective portion ratio is calculated based on the fruit size and the defective portion size of each fruit.

4. The fruit evaluation method according to claim 2, characterized in that The neural network is trained using deep image samples, including: Converting the depth image samples into RGB image samples by linear rescaling, and training a neural network by using the RGB image samples to obtain a first neural network model; Depth channel data samples are extracted from the depth image samples, and a neural network is trained using the depth channel data samples to obtain a second neural network model.

5. The fruit evaluation method according to claim 4, characterized in that: Performing pixel-level semantic segmentation on the depth image to be measured using the trained neural network model includes: The depth image to be measured is converted into an RGB image by linear rescaling; and the RGB image is semantically segmented at the pixel level by using the first neural network model; Alternatively, extract depth channel data from the depth image to be measured, and use the second neural network model to perform pixel-level semantic segmentation on the depth channel data.

6. The fruit evaluation method according to claim 2, characterized in that: The neural network is trained using deep image samples, including: Annotating a fruit target frame and defective areas in the depth image sample; The neural network model extracts and learns features of the fruit target frame and the defective area.

7. The fruit evaluation method according to claim 1, characterized in that After calculating the fruit size and defect ratio of each fruit, it also includes: determining a fruit grade based on the fruit size and a preset size threshold; The yield is estimated based on the fruit grade and the percentage of defective parts.

8. A fruit evaluation device, characterized in that: include: A depth camera module, used to collect the depth image to be measured; A neural network module is used to perform pixel segmentation on the depth image to be measured to obtain the boundary of the fruit and the boundary of the defective part of the fruit, including: converting the depth image to be measured into an RGB image by linear rescaling; performing pixel-level semantic segmentation on the RGB image using a first neural network model to obtain a first recognition result of the boundary of the fruit and the boundary of the defective part of the fruit; and extracting depth channel data from the depth image to be measured, performing pixel-level semantic segmentation on the depth channel data using a second neural network model to obtain a second recognition result of the boundary of the fruit and the boundary of the defective part of the fruit; verifying the first recognition result with the second recognition result, and / or fusing the first recognition result with the second recognition result; wherein the verification and fusion of the first recognition result with the second recognition result are achieved by using separation aggregation gating, including: fusing the global spatial information of the two modalities, performing global average pooling along the channel direction to achieve a cross-modal attention feature vector, and obtaining depth information by multiplying the cross-modal attention feature vector with each single modality; the aggregation gate is a cross-modal feature aggregation module; The size calculation module is used to measure the fruit size and defective part ratio of each fruit according to the fruit boundary and the defective part boundary.

9. A patrol device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, further comprising a depth camera and a mobile device; The mobile device is used to drive the patrol device to the destination position; The depth camera is used to obtain a depth image of the target location and send it to the processor; When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.