Fruit sorting method, device and equipment and storage medium

Abstract

The invention provides a fruit sorting method, device and equipment and a storage medium. The method comprises the steps that the movement position of to-be-sorted fruits in a working area is detected based on a first visual unit; in response to the detection that the fruits to be sorted move to the preset position, multi-modal visual information of the fruits to be sorted is collected based on a second visual unit, and the multi-modal visual information comprises color images, infrared images and three-dimensional point cloud data of the fruits to be sorted; based on the multi-modal visual information, constructing an enhanced fruit digital model of the to-be-sorted fruits; based on the enhanced fruit digital model and a grabbing decision algorithm, a grabbing scheme for the to-be-sorted fruits is determined; and controlling a fruit sorting mechanism to grab and sort the to-be-sorted fruits based on the grabbing scheme.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a fruit sorting method, apparatus, equipment and storage medium. Background Technology

[0002] Fruit sorting is a core step in the post-harvest processing and commercialization of fruits. Its purpose is to quickly and accurately classify and grade fruits according to standards such as appearance, size, and internal quality. Currently, in actual production, manual sorting remains a basic and widely used method, mainly relying on sorting workers to complete the task by visual observation and manual selection. However, this method depends on personal experience, is highly subjective, and is inefficient. Summary of the Invention

[0003] This application provides a fruit sorting method, apparatus, equipment, and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this application, a fruit sorting method is provided, the method comprising: The movement position of the fruit to be sorted in the work area is detected based on the first visual unit; In response to detecting that the fruit to be sorted has moved to a preset position, the multimodal visual information of the fruit to be sorted is acquired based on the second vision unit. The multimodal visual information includes color images, infrared images and three-dimensional point cloud data of the fruit to be sorted. Based on the multimodal visual information, an enhanced digital model of the fruit to be sorted is constructed; Based on the enhanced fruit digital model and grasping decision algorithm, a grasping scheme is determined for the fruit to be sorted. The fruit sorting mechanism is controlled to grasp and sort the fruits to be sorted based on the grasping scheme.

[0005] In one possible implementation, the detection of the movement position of the fruit to be sorted in the work area based on the first visual unit includes: Based on the image of the working area acquired by the first visual unit, the fruits to be sorted entering the working area are identified and the movement position of the centroid of the fruits to be sorted is tracked by a lightweight foreground detection and target tracking algorithm.

[0006] In one possible implementation, the second vision unit includes a tunable light source, an industrial camera, and a 3D sensor; correspondingly, The process of acquiring multimodal visual information of the fruit to be sorted based on the second visual unit includes: The tunable light source is controlled to sequentially generate white light illumination and near-infrared illumination of a specific wavelength. The industrial camera is controlled to acquire color and infrared images of the fruit to be sorted under appropriate lighting conditions. The three-dimensional sensor is controlled to collect three-dimensional point cloud data of the fruit to be sorted.

[0007] In one possible implementation, constructing an enhanced digital model of the fruit to be sorted based on the multimodal visual information includes: Instance segmentation is performed based on the color image and the three-dimensional point cloud data to extract the three-dimensional contour of the fruit to be sorted, and the geometric parameters of the fruit to be sorted are calculated based on the three-dimensional contour. The color image is input into a pre-trained appearance recognition model, which outputs the color grade of the fruit to be sorted, the location information of preset biomarker points, and a semantic segmentation map of surface defects. The grayscale and texture features of the infrared image in the region corresponding to the three-dimensional contour are analyzed, and the extracted features are input into the internal quality analysis model corresponding to the fruit category to obtain the internal quality consistency index. The geometric parameters, color levels, location information of preset biomarker points, semantic segmentation map of surface defects, and internal quality consistency index are spatially associated and registered with the three-dimensional point cloud data to generate the enhanced fruit digital model.

[0008] In one possible implementation, the grasping scheme includes grasping pose information and force control parameters for controlling the gripping action of the fruit sorting mechanism; correspondingly, The step of determining a grasping scheme for the fruit to be sorted based on the enhanced fruit digital model and grasping decision algorithm includes: Based on the three-dimensional surface represented by the enhanced fruit digital model, multiple candidate grasping surfaces are generated; For each candidate grasping surface, the grasping decision algorithm is invoked to calculate a grasping suitability score. The grasping decision algorithm is constructed based on geometric stability constraints, biological damage avoidance constraints, and mechanical matching constraints. The target crawling surface is selected based on the crawling suitability score of each candidate crawling surface; The grasping pose information is determined based on the target grasping surface; Based on the enhanced fruit digital model, the force control parameters of the fruit to be sorted are obtained by querying the pre-built clamping force parameter library.

[0009] In one embodiment, the fruit sorting mechanism includes a robot and grippers mounted on the robot's end effector. The gripping pose information includes the gripping position, the gripper approach vector, and the gripper opening angle. Correspondingly, The controlled fruit sorting mechanism grasps and sorts the fruits to be sorted based on the grasping scheme, including: Based on the grasping position, the gripper approach vector, and the gripper opening angle, a motion trajectory is planned for the robot to move from the current position to the grasping position; The robot is controlled to move along the motion trajectory, and the gripper is controlled to perform a gripping action according to the force control parameters after reaching the gripping position; The robot is controlled to transport the successfully grabbed fruit to the corresponding sorting exit.

[0010] According to a second aspect of this application, a fruit sorting device is provided, the device comprising: The detection module is used to detect the movement position of the fruit to be sorted in the work area based on the first vision unit; The acquisition module is used to acquire multimodal visual information of the fruit to be sorted based on the second vision unit in response to detecting that the fruit to be sorted has moved to a preset position. The multimodal visual information includes color images, infrared images and three-dimensional point cloud data of the fruit to be sorted. A construction module is used to construct an enhanced digital model of the fruit to be sorted based on the multimodal visual information. The determination module is used to determine a grasping scheme for the fruit to be sorted based on the enhanced fruit digital model and the grasping decision algorithm. The control module is used to control the fruit sorting mechanism to grasp and sort the fruit to be sorted based on the grasping scheme.

[0011] In one possible implementation, the detection module is used for: Based on the image of the working area acquired by the first visual unit, a lightweight foreground detection and target tracking algorithm is used to identify the fruits to be sorted entering the working area and track the movement position of the centroid of the fruits to be sorted. According to a third aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0012] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this application.

[0013] The sorting method, apparatus, equipment, and storage medium of this application detect the movement position of the fruit to be sorted in the working area based on a first vision unit; in response to detecting that the fruit to be sorted has moved to a preset position, a second vision unit acquires multimodal visual information of the fruit to be sorted, including color images, infrared images, and three-dimensional point cloud data of the fruit; based on the multimodal visual information, an enhanced digital model of the fruit to be sorted is constructed; based on the enhanced digital model and a grasping decision algorithm, a grasping scheme is determined for the fruit; and the fruit sorting mechanism is controlled to grasp and sort the fruit based on the grasping scheme. By performing coarse detection of fruit position and acquisition of multimodal visual information of the fruit through different vision units, and analyzing the multimodal visual information to construct an enhanced digital model of the fruit, and combining the enhanced digital model of the fruit with the grasping decision algorithm to determine the grasping scheme, the fruit sorting mechanism is controlled to grasp and sort the fruit, achieving efficient, accurate, and near-zero-damage automated intelligent sorting of fruit.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0015] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0016] Figure 1 A schematic diagram illustrating the implementation flow of the fruit sorting method provided in this application embodiment is shown; Figure 2 This illustration shows a schematic diagram of the implementation process of the model building operation of the fruit sorting method provided in the embodiments of this application; Figure 3 This paper illustrates a schematic diagram of the implementation process of determining the grasping scheme in the fruit sorting method provided in an embodiment of this application. Figure 4 A schematic diagram of the composition structure of the fruit sorting device provided in the embodiments of this application is shown; Figure 5 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0017] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Figure 1 A schematic diagram illustrating the implementation flow of the fruit sorting method provided in this application embodiment is shown.

[0019] refer to Figure 1 This application provides a fruit sorting method, the method comprising: Operation 101: Detect the movement position of the fruit to be sorted in the work area based on the first visual unit.

[0020] The working area refers to the field of view of the first vision unit. After the fruit enters the working area via a conveying device, such as a vibrating feeding tray or a conveyor belt, it is photographed in real time by the first vision unit and assigned a unique ID to track the movement position of the fruit.

[0021] Operation 102, in response to detecting that the fruit to be sorted has moved to a preset position, collects multimodal visual information of the fruit to be sorted based on the second vision unit. The multimodal visual information includes color images, infrared images and three-dimensional point cloud data of the fruit to be sorted.

[0022] The first vision unit is mainly used to roughly screen the fruits and determine their arrival time, while the second vision unit is used to perform precise analysis on the fruits. The preset position can be understood as the location point where precise analysis can be performed. It can be configured according to actual needs. For example, based on the complexity of the fruit's shape, the more complex the fruit, the more advanced the preset position will be, so as to allow enough analysis time.

[0023] Once the first vision unit detects that the fruit to be sorted has reached the preset position, it sends a precise hardware trigger signal to the second vision unit. At this point, the second vision unit begins to work, acquiring multimodal visual information of the fruit to be sorted. To ensure sorting accuracy, the multimodal visual information must include color (RGB) images, infrared images, and 3D point cloud data of the fruit to be sorted.

[0024] Operation 103: Based on multimodal visual information, construct an enhanced digital model of the fruit to be sorted.

[0025] Color images, infrared images, and 3D point cloud data of fruits to be sorted are processed and fused to generate a feature model that characterizes the overall state of the fruits, namely the Enhanced Digital Fruit Model (EDFM). This model is based on three-dimensional morphology and integrates the appearance and internal quality information of the fruits.

[0026] Operation 104 determines the grasping scheme for the fruits to be sorted based on the enhanced fruit digital model and grasping decision algorithm.

[0027] The grasping decision algorithm is an intelligent decision-making model that integrates multiple physical constraints. Determining a grasping scheme based on this algorithm can be viewed as taking an enhanced digital model of the fruit as input, performing optimization calculations based on multiple physical constraints within the algorithm, and outputting a grasping scheme customized for the individual fruit to be sorted. The grasping scheme may include a series of operational parameters for the fruit sorting mechanism to grasp and sort, such as grasping position and grasping force.

[0028] Operation 105 controls the fruit sorting mechanism to grasp and sort the fruits to be sorted based on the grasping scheme.

[0029] Once the grabbing plan is determined, it is synchronized to the fruit sorting organization. The fruit sorting organization receives and executes the grabbing plan, completes the picking action of the fruit to be sorted, and transfers it to the corresponding sorting exit.

[0030] Thus, in this embodiment, different visual units perform coarse detection of fruit position and acquisition of multimodal visual information of fruit, and analyze the multimodal visual information to construct an enhanced digital model of fruit. At the same time, the enhanced digital model of fruit is combined with the grasping decision algorithm to determine the grasping scheme, and the fruit sorting mechanism is controlled to grasp and sort the fruit, thereby achieving efficient, accurate and near-zero damage automated intelligent fruit sorting.

[0031] In one embodiment of this application, the above operation 101, which detects the movement position of the fruit to be sorted in the work area based on the first visual unit, includes: acquiring an image of the work area based on the first visual unit, identifying the fruit to be sorted entering the work area and tracking the movement position of the centroid of the fruit to be sorted through a lightweight foreground detection and target tracking algorithm.

[0032] Specifically, the first vision unit can be configured as a high frame rate (≥500fps) global shutter monochrome or color industrial camera, equipped with uniform backlighting or diffuse illumination, to form a high-contrast binarized visual scene.

[0033] Once the fruit enters the working area, an extremely lightweight foreground detection and target tracking algorithm (e.g., background subtraction combined with Kalman filtering) is run to detect in real time whether any new fruit individuals have entered the field of view of the first visual unit. After a new fruit individual enters, a unique ID is assigned to each individual and its coarse centroid position is tracked.

[0034] Specifically, after the first vision unit detects that the fruit to be sorted has entered the optimal imaging window of the second vision unit, i.e., the preset position, it also sends a precise hardware trigger signal to the second vision unit.

[0035] Thus, by separating the target discovery and tracking task, which has high real-time requirements and low computational complexity, from the target discovery and tracking task and having it completed by dedicated hardware and lightweight algorithms, this application provides a sufficient processing time window for the subsequent analysis stage and achieves system-level time resource reuse.

[0036] In one embodiment of this application, the second vision unit includes a tunable light source, an industrial camera, and a three-dimensional sensor. The above-mentioned operation 102, which collects multimodal visual information of the fruit to be sorted based on the second vision unit, includes: controlling the tunable light source to sequentially generate white light illumination and near-infrared illumination of a specific wavelength; controlling the industrial camera to collect color images and infrared images of the fruit to be sorted under the corresponding illumination conditions; and controlling the three-dimensional sensor to collect three-dimensional point cloud data of the fruit to be sorted.

[0037] Specifically, the second vision unit can be understood as a multispectral imaging station equipped with a tunable LED light source (containing at least white light and specific near-infrared bands, such as 780nm and 850nm) and an industrial camera. The second vision unit may also optionally include a 3D sensor (e.g., a laser profile sensor) for 3D modeling.

[0038] After receiving a trigger signal, the second vision unit controls the tunable light source to quickly generate and switch between white light illumination and near-infrared illumination of a specific wavelength, simultaneously or quasi-synchronously acquiring RGB and near-infrared images of the same fruit. Simultaneously, a laser contour sensor or binocular vision is used to acquire high-precision 3D point cloud data of the fruit to be sorted.

[0039] Figure 2 This diagram illustrates the implementation flow of the model building operation for the fruit sorting method provided in this embodiment.

[0040] refer to Figure 2 In one embodiment of this application, the above-mentioned operation 103, based on multimodal visual information, constructs an enhanced digital model of the fruit to be sorted, including: Operation 201 involves instance segmentation based on color images and 3D point cloud data, extracting the 3D contours of the fruits to be sorted, and calculating the geometric parameters of the fruits to be sorted based on the 3D contours.

[0041] Based on RGB images and 3D point cloud data, instance segmentation is performed to accurately extract the contours of the fruits to be sorted. Geometric parameters such as volume, maximum and minimum diameters, and sphericity of the fruits to be sorted are calculated.

[0042] Operation 202: Input the color image into the pre-trained appearance recognition model, and the appearance recognition model outputs the color grade of the fruit to be sorted, the location information of the preset biomarker points, and the semantic segmentation map of surface defects.

[0043] This application pre-trains a fruit appearance recognition model, which is a convolutional neural network (CNN) model. The training process can refer to the training process of a conventional appearance recognition model, and will not be described in detail here.

[0044] After inputting the RGB image into the appearance recognition model, the model analyzes the RGB image and outputs the color grade of the fruit to be sorted, the location information of preset biomarker points, and a semantic segmentation map of surface defects. The preset biomarker points can refer to the location of the calyx and / or pedicel of the fruit to be sorted.

[0045] Operation 203 analyzes the grayscale and texture features of the infrared image within the region corresponding to the three-dimensional contour, and inputs the extracted features into the internal quality analysis model corresponding to the fruit category to obtain the internal quality consistency index.

[0046] The uniformity of grayscale distribution and texture features in the target region of the near-infrared image are analyzed. The target region can be understood as the area in the infrared image corresponding to the three-dimensional contour extracted from the RGB image. An internal quality consistency index is calculated by comparing the index with pre-established simplified spectral-quality correlation models for different fruit categories, i.e., internal quality analysis models. This internal quality consistency index is used to identify fruits that may have internal browning, hollowness, or uneven ripening.

[0047] Operation 204 involves spatially associating and registering geometric parameters, color levels, location information of preset biomarker points, semantic segmentation maps of surface defects, and internal quality consistency indices with 3D point cloud data to generate an enhanced digital model of the fruit.

[0048] Specifically, all the above features (geometric parameters, color grades, location information of preset biomarker points, semantic segmentation map of surface defects) are spatially associated and registered with the 3D point cloud data of the fruit to be sorted, forming an enhanced digital model of the fruit carrying multi-dimensional attributes.

[0049] Figure 3 This diagram illustrates the implementation flow of the grasping scheme determination operation in the fruit sorting method provided in this embodiment.

[0050] refer to Figure 3 In one embodiment of this application, the grasping scheme includes grasping pose information and force control parameters for controlling the gripping action of the fruit sorting mechanism. Operation 104, based on an enhanced fruit digital model and a grasping decision algorithm, determines a grasping scheme for the fruit to be sorted, including: Operation 301 generates multiple candidate grasping surfaces based on the three-dimensional surface represented by the enhanced fruit digital model.

[0051] On the surface of the enhanced fruit digital model, multiple stable candidate grasping surfaces are automatically generated based on normal direction and curvature constraints.

[0052] Operation 302: For each candidate grasping surface, call the grasping decision algorithm to calculate a grasping suitability score. The grasping decision algorithm is constructed based on geometric stability constraints, biological damage avoidance constraints, and mechanical matching constraints.

[0053] Based on the grasping decision algorithm, a comprehensive Grasping Suitability Score (GSS) is calculated for each candidate grasping surface. The algorithm comprehensively considers: 1) Geometric stability constraints: curvature and flatness of the contact surface. 2) Biological damage avoidance constraints: the three-dimensional spatial distance between the contact surface and the fruit stalk, calyx, and any surface defect areas marked in the EDFM. The closer the distance, the greater the penalty. 3) Mechanical matching constraints: based on the mechanical model of the fruit sorting mechanism, the uniformity of the clamping force distribution under the candidate grasping surface and whether the maximum pressure exceeds the safety threshold of the fruit peel for this type of fruit are evaluated. The safety threshold can be configured based on experience.

[0054] For example, the specific process of scoring candidate gripping surfaces based on the gripping decision algorithm may include: 1) Calculating the geometric stability score: This is calculated based on the average curvature and surface smoothness of the candidate gripping surface. The lower the curvature and the smoother the surface, the higher the score. 2) Calculating the biological damage avoidance score: This calculates the nearest spatial distance from the candidate gripping surface to the marked fruit stalk, calyx, or surface defect area in the enhanced fruit digital model. The closer the distance, the lower the score, i.e., the greater the penalty; when the distance exceeds the safety threshold, the score is close to full. 3) Calculating the mechanical matching score: Based on the mechanical model of the flexible gripper of the fruit sorting mechanism, the pressure distribution when applying clamping force to the candidate gripping surface is simulated. If the maximum pressure exceeds the safety threshold of the fruit peel for this type of fruit, the score is zero; if it does not exceed the threshold, the score is calculated based on the uniformity of the pressure distribution and the safety margin of the maximum pressure. The more uniform the distribution and the larger the margin, the higher the score. 4) Calculate the comprehensive score: The above three scores are weighted and summed according to the pre-configured weight ratio to obtain the final crawling suitability score of the candidate crawling surface.

[0055] Operation 303: Select the target crawling surface based on the crawling suitability score of each candidate crawling surface.

[0056] The candidate surface with the highest score is selected as the target surface for crawling.

[0057] Operation 304: Determine the grasping pose information based on the target grasping surface.

[0058] Based on the three-dimensional spatial position and normal direction of the target grasping surface, the grasping pose information is automatically calculated, such as the precise coordinates of the grasping center point, the vector direction that the gripper of the fruit sorting mechanism should follow when approaching the plane, and the gripper opening angle adapted to the fruit size.

[0059] Operation 305 uses an enhanced digital fruit model to query a pre-built clamping force parameter library and matches the force control parameters of the fruit to be sorted.

[0060] Based on the ripeness information inferred from the fruit category, size specifications, and internal quality contained in the enhanced fruit digital model, a pre-set clamping force parameter database is queried. This database stores experimentally verified safe clamping force ranges or pressure thresholds for different categories, sizes, and ripeness levels of fruit. Through matching, appropriate force control parameters (such as maximum permissible clamping force or pressure value) are assigned to the fruit to be sorted, thereby ensuring stable gripping while ensuring that the applied force does not exceed the fruit skin's tolerance limit and avoiding damage.

[0061] In one embodiment of this application, the fruit sorting mechanism includes a robot and a gripper installed at the end of the robot. The gripping posture information includes the gripping position, the gripper approach vector, and the gripper opening angle. The above-mentioned operation 105 controls the fruit sorting mechanism to grip and sort the fruit to be sorted based on the gripping scheme, including: planning a motion trajectory for the robot from the current position to the gripping position based on the gripping position, the gripper approach vector, and the gripper opening angle; controlling the robot to move along the motion trajectory, and controlling the gripper to perform a gripping action according to the force control parameters after reaching the gripping position; and controlling the robot to transport the successfully gripped fruit to be sorted to the corresponding sorting exit.

[0062] The fruit sorting mechanism includes a robot (a three-axis Cartesian robot) and grippers (adaptive pneumatic / electric flexible grippers) mounted at the robot's end effector. Grasping pose information includes the gripping position, gripper approach vector, and gripper opening angle. The gripping center point is the coordinate (X, Y, Z) of the geometric center of the target gripping surface in a 3D point cloud. The approach vector is the average normal direction of the target gripping surface. The opening angle is determined based on the local diameter of the fruit corresponding to the target gripping surface and the physical opening and closing range of the gripper.

[0063] Specifically, based on the grasping position (3D coordinates), the gripper approach vector (spatial orientation), and the gripper opening angle (pre-opening amount), a path search algorithm (such as RRT or trajectory interpolation) is used to calculate the optimal trajectory for the robot to move from any current position to the grasping position without collision. The robot then tracks the planned trajectory, driving the end effector gripper to approach the fruit to be sorted in a specified posture. Once the spatial positioning meets the accuracy requirements, the gripper controller switches to force control or position-force hybrid control mode based on the force control parameters matched in the scheme (such as current-force mapping or pressure threshold), performing adaptive compliant grasping to ensure that the gripping force overcomes gravity without exceeding the fruit peel safety threshold. After successful grasping, based on the quality grade determined by the enhanced fruit digital model, the corresponding sorting exit coordinates are called to plan the transport trajectory, and the robot accurately transfers and releases the fruit to the designated exit.

[0064] In one embodiment of this application, there are multiple fruits to be sorted. After calculating the EDFM and grasping scheme of all fruits to be sorted, the EDFM and grasping scheme of all fruits to be sorted are integrated to plan the globally optimal grasping sequence and motion trajectory for the robot, and to perform closed-loop synchronization with the conveyor belt speed to ensure accurate grasping in dynamics.

[0065] In one embodiment of this application, during the process of the robot grasping and placing according to the grasping scheme, a low-cost review point (such as a weighing sensor or simple visual review) can be set up to provide feedback on the very few cases of misjudgment.

[0066] In one embodiment of this application, a sorting log is also maintained. When the statistical characteristics (such as average size and color distribution) of the same type of fruit drift slowly, unsupervised or semi-supervised fine-tuning of the parameters of each model and algorithm can be triggered to achieve adaptive optimization in long-term operation.

[0067] Based on the above-described fruit sorting method, this application also provides a fruit sorting system, comprising: a vibrating feeding or conveyor belt, a first vision unit (high-speed global trigger camera), a second vision unit (multispectral fine imaging station), a three-axis Cartesian coordinate robot, an adaptive pneumatic / electric flexible gripper, and a main control computer. The main control computer is configured to execute the fruit sorting method of this application.

[0068] Figure 4 A schematic diagram of the composition structure of the fruit sorting device provided in the embodiments of this application is shown.

[0069] refer to Figure 4 This application also provides a fruit sorting device, which includes: Detection module 401 is used to detect the movement position of the fruit to be sorted in the working area based on the first vision unit; The acquisition module 402 is used to acquire multimodal visual information of the fruit to be sorted based on the second vision unit in response to detecting that the fruit to be sorted has moved to a preset position. The multimodal visual information includes color images, infrared images and three-dimensional point cloud data of the fruit to be sorted. Module 403 is used to build an enhanced digital model of the fruit to be sorted based on multimodal visual information. The determination module 404 is used to determine the grasping scheme for the fruit to be sorted based on the enhanced fruit digital model and grasping decision algorithm. The control module 405 is used to control the fruit sorting mechanism to grasp and sort the fruit to be sorted based on the grasping scheme.

[0070] In one embodiment of this application, the detection module 401 is used for: The image of the working area is acquired based on the first visual unit, and the fruits to be sorted entering the working area are identified and the movement position of the centroid of the fruits to be sorted is tracked through a lightweight foreground detection and target tracking algorithm.

[0071] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For any technical details not covered in the fruit sorting apparatus provided in this application embodiment, please refer to... Figures 1 to 3 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0072] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0073] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0074] like Figure 5As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0075] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a fruit sorting method. For example, in some embodiments, the fruit sorting method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the fruit sorting method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the fruit sorting method by any other suitable means (e.g., by means of firmware).

[0077] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0078] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0079] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0080] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0081] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0082] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0083] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0084] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for sorting fruit, characterized in that, The method includes: The movement position of the fruit to be sorted in the work area is detected based on the first visual unit; In response to detecting that the fruit to be sorted has moved to a preset position, the multimodal visual information of the fruit to be sorted is acquired based on the second vision unit. The multimodal visual information includes color images, infrared images and three-dimensional point cloud data of the fruit to be sorted. Based on the multimodal visual information, an enhanced digital model of the fruit to be sorted is constructed; Based on the enhanced fruit digital model and grasping decision algorithm, a grasping scheme is determined for the fruit to be sorted. The fruit sorting mechanism is controlled to grasp and sort the fruits to be sorted based on the grasping scheme.

2. The method according to claim 1, characterized in that, The detection of the movement position of the fruit to be sorted in the work area based on the first visual unit includes: Based on the image of the working area acquired by the first visual unit, the fruits to be sorted entering the working area are identified and the movement position of the centroid of the fruits to be sorted is tracked by a lightweight foreground detection and target tracking algorithm.

3. The method according to claim 1, characterized in that, The second vision unit includes a tunable light source, an industrial camera, and a 3D sensor; correspondingly, The process of acquiring multimodal visual information of the fruit to be sorted based on the second visual unit includes: The tunable light source is controlled to sequentially generate white light illumination and near-infrared illumination of a specific wavelength. The industrial camera is controlled to acquire color and infrared images of the fruit to be sorted under appropriate lighting conditions. The three-dimensional sensor is controlled to collect three-dimensional point cloud data of the fruit to be sorted.

4. The method according to claim 1, characterized in that, The step of constructing an enhanced digital model of the fruit to be sorted based on the multimodal visual information includes: Instance segmentation is performed based on the color image and the three-dimensional point cloud data to extract the three-dimensional contour of the fruit to be sorted, and the geometric parameters of the fruit to be sorted are calculated based on the three-dimensional contour. The color image is input into a pre-trained appearance recognition model, which outputs the color grade of the fruit to be sorted, the location information of preset biomarker points, and a semantic segmentation map of surface defects. The grayscale and texture features of the infrared image in the region corresponding to the three-dimensional contour are analyzed, and the extracted features are input into the internal quality analysis model corresponding to the fruit category to obtain the internal quality consistency index. The geometric parameters, color levels, location information of preset biomarker points, semantic segmentation map of surface defects, and internal quality consistency index are spatially associated and registered with the three-dimensional point cloud data to generate the enhanced fruit digital model.

5. The method according to claim 1, characterized in that, The grasping scheme includes grasping pose information and force control parameters for controlling the gripping action of the fruit sorting mechanism; correspondingly, The step of determining a grasping scheme for the fruit to be sorted based on the enhanced fruit digital model and grasping decision algorithm includes: Based on the three-dimensional surface represented by the enhanced fruit digital model, multiple candidate grasping surfaces are generated; For each candidate grasping surface, the grasping decision algorithm is invoked to calculate a grasping suitability score. The grasping decision algorithm is constructed based on geometric stability constraints, biological damage avoidance constraints, and mechanical matching constraints. The target crawling surface is selected based on the crawling suitability score of each candidate crawling surface; The grasping pose information is determined based on the target grasping surface; Based on the enhanced fruit digital model, the force control parameters of the fruit to be sorted are obtained by querying the pre-built clamping force parameter library.

6. The method according to claim 5, characterized in that, The fruit sorting mechanism includes a robot and grippers mounted on the robot's end effector. The gripping pose information includes the gripping position, gripper approach vector, and gripper opening angle. Correspondingly, The controlled fruit sorting mechanism grasps and sorts the fruits to be sorted based on the grasping scheme, including: Based on the grasping position, the gripper approach vector, and the gripper opening angle, a motion trajectory is planned for the robot to move from the current position to the grasping position; The robot is controlled to move along the motion trajectory, and the gripper is controlled to perform a gripping action according to the force control parameters after reaching the gripping position; The robot is controlled to transport the successfully grabbed fruit to the corresponding sorting exit.

7. A fruit sorting device, characterized in that, The device includes: The detection module is used to detect the movement position of the fruit to be sorted in the work area based on the first vision unit; The acquisition module is used to acquire multimodal visual information of the fruit to be sorted based on the second vision unit in response to detecting that the fruit to be sorted has moved to a preset position. The multimodal visual information includes color images, infrared images and three-dimensional point cloud data of the fruit to be sorted. A construction module is used to construct an enhanced digital model of the fruit to be sorted based on the multimodal visual information. The determination module is used to determine a grasping scheme for the fruit to be sorted based on the enhanced fruit digital model and the grasping decision algorithm. The control module is used to control the fruit sorting mechanism to grasp and sort the fruit to be sorted based on the grasping scheme.

8. The apparatus according to claim 7, characterized in that, The detection module is used for: Based on the image of the working area acquired by the first visual unit, the fruits to be sorted entering the working area are identified and the movement position of the centroid of the fruits to be sorted is tracked by a lightweight foreground detection and target tracking algorithm.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images