Image processing methods, apparatus, equipment, and storage media for food sorting

Abstract

This disclosure provides an image processing method, apparatus, device, and storage medium for food sorting, relating to the field of artificial intelligence technology, particularly computer vision, image processing, and deep learning. The specific implementation involves: segmenting at least one target region in a food image using an image segmentation model; determining a region of impurities to be sorted within the at least one target region; and obtaining the sorting position information of the impurities in the food based on the centroid coordinates of the region of impurities to be sorted; wherein the sorting position information is used to instruct the sorting execution device to sort the impurities. The technical solution of this disclosure can improve the accuracy of the sorting position information. Furthermore, by using this sorting position information to sort impurities, the sorting execution device can avoid food breakage caused by a shift in the sorting center of gravity, thereby improving sorting efficiency and accuracy while reducing labor costs.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, image processing, and deep learning. Background Technology

[0002] In traditional food processing, raw material handling often involves manual handling and sorting, with sorting including removing impurities and grading. Some foods are lightweight but often fragile and easily deformed. Manual sorting of food suffers from high labor costs, low efficiency, and inconsistent accuracy due to varying evaluation standards. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, device, and storage medium for food sorting.

[0004] According to one aspect of this disclosure, an image processing method for food sorting is provided, comprising:

[0005] At least one target region is segmented from a food image using an image segmentation model;

[0006] Determine the area of ​​impurities to be picked out in the at least one target area;

[0007] Based on the centroid coordinates of the region of impurities to be sorted, the sorting location information of the impurities in the food is obtained; wherein, the sorting location information is used to instruct the sorting execution equipment to sort the impurities.

[0008] According to another aspect of this disclosure, an image processing apparatus for food sorting is provided, comprising:

[0009] The image segmentation module is used to segment at least one target region in a food image using an image segmentation model.

[0010] A region determination module is used to determine the region of impurities to be picked out in the at least one target region;

[0011] The position determination module is used to obtain the picking position information of impurities in the food based on the centroid coordinates of the impurity area to be picked; wherein the picking position information is used to instruct the sorting execution equipment to pick the impurities.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0018] According to another aspect of this disclosure, a food sorting system is provided, including an image acquisition device, a sorting execution device, and an electronic device according to an embodiment of this disclosure; wherein the image acquisition device is used to acquire food images and send the food images to the electronic device; the sorting execution device is used to receive sorting instruction information from the electronic device and pick out impurities from the food based on the sorting instruction information.

[0019] The technical solution adopted in this disclosure has the following beneficial effects:

[0020] Since image segmentation models can segment irregularly shaped target regions, the centroid coordinates of the impurity region to be sorted identified within at least one target region can accurately characterize the centroid position of the irregularly shaped food impurity. Using these centroid coordinates to obtain sorting position information can improve the accuracy of the sorting position information. Furthermore, when the sorting equipment sorts impurities based on this sorting position information, it can also avoid food breakage caused by a shift in the sorting center of gravity, thereby improving sorting efficiency and accuracy while reducing labor costs.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a schematic diagram of an exemplary application scenario according to an embodiment of the present disclosure;

[0024] Figure 2 This is a schematic flowchart of an image processing method for food sorting according to an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic diagram of the target area in an embodiment of this disclosure;

[0026] Figure 4 This is a schematic flowchart of an image processing method for food sorting according to another embodiment of the present disclosure;

[0027] Figure 5 This is a schematic diagram of another exemplary application scenario according to an embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram illustrating yet another exemplary application scenario according to an embodiment of the present disclosure;

[0029] Figure 7 This is a schematic flowchart of an image processing method for food sorting according to yet another embodiment of the present disclosure;

[0030] Figure 8 This is a schematic block diagram of an image processing apparatus for food sorting provided in an embodiment of the present disclosure;

[0031] Figure 9 This is a schematic block diagram of an image processing apparatus for food sorting provided in another embodiment of this disclosure;

[0032] Figure 10 This is a schematic block diagram of an image processing apparatus for food sorting provided in another embodiment of this disclosure;

[0033] Figure 11 This is a schematic block diagram of an image processing apparatus for food sorting provided in another embodiment of this disclosure;

[0034] Figure 12 This is a schematic block diagram of an image processing apparatus for food sorting provided in another embodiment of this disclosure;

[0035] Figure 13 This is a schematic block diagram of an image processing apparatus for food sorting provided in another embodiment of this disclosure;

[0036] Figure 14 This is a schematic block diagram of an image processing apparatus for food sorting provided in another embodiment of this disclosure;

[0037] Figure 15 This is a block diagram of an electronic device used to implement the image processing method for food sorting according to the embodiments of this disclosure;

[0038] Figure 16 This is a schematic block diagram of a food sorting system according to an embodiment of the present disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] To facilitate understanding of the image processing method for food sorting in the embodiments of this disclosure, the application scenarios of the embodiments of this disclosure will be introduced below. Figure 1 This is a schematic diagram illustrating an exemplary application scenario of an embodiment of this disclosure. For example... Figure 1 As shown, in this application scenario, an electronic device 101 is included. The electronic device 101 can be, for example, a terminal, server, or other processing device in a standalone, multi-machine, or cluster system. The terminal can be a UE (User Equipment), mobile device, PDA (Personal Digital Assistant), handheld device, computing device, in-vehicle device, wearable device, etc. In some optional implementations, the electronic device 101 can implement the image processing method for food sorting according to embodiments of this disclosure by having a processor call computer-readable instructions stored in memory. This method can determine sorting location information based on food images.

[0041] like Figure 1 As shown, this application scenario also includes an image acquisition device 102 and a sorting execution device 103.

[0042] The image acquisition device 102 is used to acquire images of the food. Specifically, the image acquisition device 102 takes a picture of the food by pointing it toward the tray 104 on which the food is placed. The image acquisition device 102 is also used to send the food image to the electronic device 101, so that the electronic device 101 can determine the picking position information based on the food image. In practical applications, the image acquisition device 102 can be used as follows: Figure 1 For example, it can be fixed to the end of the sorting execution device 103, or it can be separated from the sorting execution device 103.

[0043] The sorting execution device 103 is used to pick out impurities from food according to the picking position information sent by the electronic device 101. Exemplarily, the sorting execution device 103 can be an automated robot or a robotic arm. Exemplarily, the end of the sorting execution device 103 can be equipped with a suction pen to pick out impurities from the food by suction.

[0044] The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0045] Figure 2 This is a schematic flowchart of an image processing method for food sorting according to an embodiment of the present disclosure. The method can be applied to an image processing apparatus. Exemplarily, the apparatus can be deployed in the aforementioned electronic device, but is not limited thereto. Figure 2 As shown, the method may include:

[0046] Step S210: Use an image segmentation model to segment at least one target region in the food image;

[0047] Step S220: Determine the area of ​​impurities to be picked out in at least one target area;

[0048] Step S230: Based on the centroid coordinates of the area to be sorted, obtain the sorting location information of the impurities in the food; wherein, the sorting location information is used to instruct the sorting execution equipment to sort the impurities.

[0049] In step S210 above, the image segmentation model can refer to a neural network model based on deep learning. Optionally, the image segmentation model can be a model using network frameworks such as OCRnet (Object Contextual Representation Network), U-net (U-shaped network), or Fast-SCNN (Fast Segmentation Convolutional Neural Network).

[0050] For example, a food image is input into an image segmentation model, which can semantically segment the food image into multiple image regions, including at least one target region. Here, the target region can refer to a segmented image region of a specific shape, such as a dot-like region or a strip-like region within a predetermined scale range. Figure 3 A schematic diagram of the target area in an embodiment of this disclosure is shown. For example... Figure 3 As shown, the dot-like region 302 and the strip-like regions 301 and 303 segmented from the food image are both target regions. In practical applications, the dot-like regions, strip-like regions, and other target regions obtained from segmenting food images are often the image regions where impurities are located.

[0051] Training an image segmentation model with a large number of sample images enables the model to recognize and segment regions of a specific shape. Optionally, before step S210, multiple sample images can be obtained through data augmentation, and the image segmentation model can be trained to convergence using these multiple sample images. Obtaining multiple sample images through data augmentation can include performing data augmentation on multiple food images containing impurities of different scales and categories to obtain multiple sample images. The data augmentation method can be to perform operations such as flipping, rotating, cropping, scaling, and translating on each food image to obtain one or more augmented images corresponding to each food image. Both the food image and the corresponding augmented image can be used as sample images.

[0052] In this embodiment of the disclosure, the impurity region to be picked out can refer to the image region where the impurities that need to be picked out in the food are located. In practical applications, all impurities can be picked out, or only some impurities can be picked out. Accordingly, all or part of the segmented target region can be used as the impurity region to be picked out. Exemplarily, in the above step S220, at least one target region can be filtered or screened according to pre-configured rules to obtain the impurity region to be picked out. The number of impurity regions to be picked out can be one or more.

[0053] In step S230 above, the centroid coordinates of the impurity area to be picked are the pixel coordinates of the centroid point of that area in the image coordinate system, determined based on the coordinates of each pixel in the impurity area to be picked. It can be understood that these coordinates represent the position of the centroid of the impurity to be picked. By transforming these coordinates, sorting position information that the sorting execution device can recognize and understand can be obtained. For example, the sorting position information can be the coordinates of the impurity's centroid in the world coordinate system; the electronic device can convert the centroid coordinates of the impurity area to be picked into coordinates in the world coordinate system and use these coordinates as the sorting position information, then the sorting execution device can move the suction pen to the point corresponding to these world coordinates. As another example, the sorting position information can be the displacement information of the suction pen moving to the centroid of the impurity; the electronic device can convert the centroid coordinates of the impurity area to be picked into coordinates in the world coordinate system and determine the displacement information of the suction pen moving to these coordinates based on these coordinates, then the sorting execution device can move the suction pen according to this displacement information.

[0054] According to the method provided in this disclosure, since the image segmentation model can segment irregularly shaped target regions, the centroid coordinates of the impurity region to be picked out determined in at least one target region can accurately characterize the centroid position of the irregularly shaped food impurity. Using these centroid coordinates to obtain picking position information can improve the accuracy of the picking position information. Furthermore, the sorting execution equipment can avoid food breakage caused by the shift in the picking center of gravity by picking impurities based on this picking position information, thereby improving picking efficiency and accuracy while reducing labor costs.

[0055] Optionally, in some embodiments of this disclosure, the image processing method for food sorting may further include:

[0056] Based on the area of ​​the impurity to be picked out, determine the model of the suction pen used for picking out impurities;

[0057] Send sorting instruction information to the sorting execution equipment; wherein, the sorting instruction information is used to indicate the picking location information and the model of the suction pen.

[0058] For example, area thresholds, such as upper and lower area limits, can be preset for each model of the suction pen. Then, the appropriate suction pen model for the impurity area to be picked out can be determined based on the size of the area to be picked out and the area thresholds corresponding to each model.

[0059] For example, if impurities with a diameter of 2 mm or more need to be picked up using a large-sized suction pen, then first convert 2 mm into the pixel area in the image, for example, 1500 pixels. If the area of ​​the impurity to be picked up is greater than 1500 pixels, then the suction pen is determined to be a large-sized model.

[0060] According to the above optional embodiments, when the electronic device instructs the sorting execution device on the picking position information, it also instructs the sorting execution device on the model of the suction pen, so that the sorting execution device can use the corresponding model of suction pen to pick up impurities, avoiding picking failure or food breakage due to unsuitable suction pen size, thereby ensuring picking accuracy.

[0061] For example, the electronic device can indicate the model of the suction pen by carrying the model number in the sorting instruction information. Alternatively, the electronic device can indicate the model of the suction pen by carrying a suction pen switching mark in the sorting instruction information. For example, a small-sized suction pen is used by default for impurity picking, and when the sorting instruction information carries a suction pen switching mark, a large-sized suction pen is used for impurity picking.

[0062] Optionally, in an exemplary embodiment, step S210, segmenting at least one target region in the food image using an image segmentation model, may include: dividing the food image into M image slices, where M is an integer greater than or equal to 2; inputting each of the M image slices into the image segmentation model to obtain the target region output by the image segmentation model for each image slice; and obtaining at least one target region based on the target region output by the image segmentation model for each image slice.

[0063] In other words, a large food image is divided into multiple image slices, which are then input into an image segmentation model, allowing the model to process smaller image slices. By processing these smaller image slices using the image segmentation model, the target region in each slice is obtained, and these slices are then combined to obtain at least one target region in the food image.

[0064] For example, if the food image acquired by the image acquisition device is 5472*3648 pixels, the food image can be evenly divided into 36 slices, each slice having a resolution of 912*608 pixels. Then, the input image size of the image segmentation model can be set to 912*608 pixels.

[0065] Since the size of the input image affects the computational parameters and model complexity in the image segmentation model, thus impacting the accuracy and speed of the model, this embodiment of the disclosure segments the food image, allowing the image segmentation model to process these image slices, thereby optimizing the computational parameters and model complexity in image segmentation. By processing small image slices multiple times instead of processing a large image slice once, a significant improvement in processing speed can be achieved.

[0066] Optionally, in practical applications, deep learning inference optimizers such as TensorRT can be used to accelerate the image segmentation model to further improve the speed of obtaining at least one target region, thereby improving food sorting efficiency.

[0067] Optionally, in an exemplary embodiment, step S230, determining the impurity area to be picked in at least one target area, includes: determining the impurity area to be picked in at least one target area based on the attribute information of each target area in at least one target area; wherein the attribute information includes at least one of area, spacing, and brightness.

[0068] In other words, the target region can be filtered or screened based on at least one of its area, spacing, and brightness to obtain the impurity areas to be removed. The spacing of the target regions refers to the distance between this target region and other target regions, specifically the distance between their centroids. The brightness of the target region can be calculated based on the pixel values ​​of each pixel in the target region of the food image.

[0069] According to this implementation method, the requirements for the area to be sorted impurities, such as area, spacing, or brightness, can be flexibly set according to the actual situation of the food sorting scenario. This allows for the optimization of the selection criteria for the area to be sorted impurities based on the actual situation, which is beneficial to improve sorting efficiency by selecting appropriate areas to be sorted impurities.

[0070] For example, based on the attribute information of each target region in at least one target region, determining the impurity region to be picked in at least one target region includes: in the case that at least one target region contains N regions with a spacing less than a preset distance threshold, determining the K regions with the largest area among the N regions, where N is an integer greater than or equal to 2 and K is a positive integer less than N; and determining the impurity region to be picked in the K regions.

[0071] In other words, for multiple target areas that are close together, only the largest one or more areas are retained. In food sorting scenarios, for impurity clusters formed by impurities that are close together, picking the largest one or more impurities can remove all impurities from the cluster. Therefore, implementing the above example can also improve sorting efficiency.

[0072] For example, based on the attribute information of each target region in at least one target region, determining the impurity region to be picked in at least one target region includes: filtering out regions in at least one target region whose area is smaller than a preset area threshold and / or whose brightness is smaller than a preset brightness threshold, thereby obtaining the impurity region to be picked.

[0073] In other words, impurities that are too small in area, too dim in color (light-colored), or both too small in area and too dim in color can be filtered out. In practical applications, some small and / or light-colored impurities can be left unsorted, thereby improving sorting efficiency.

[0074] Optionally, the above example of determining the areas to be picked based on the attribute information of the target area can be implemented in combination. For example, after determining the K largest areas among N areas with a spacing less than a preset distance threshold, areas with an area less than a preset area threshold and / or a brightness less than a preset brightness threshold are filtered out from the K areas to obtain the impurity areas to be picked. As another example, after filtering out areas with an area less than a preset area threshold and / or a brightness less than a preset brightness threshold from at least one target area, if at least one target area still contains N areas with a spacing less than a preset distance threshold, then the K largest areas are determined from the N areas, and these K areas are taken as the K impurity areas to be picked.

[0075] Optionally, in some embodiments of this disclosure, step S230, obtaining the picking location information of impurities in the food based on the centroid coordinates of the impurity region to be picked, includes: obtaining the picking location information of impurities in the food based on a predetermined coordinate transformation relationship and the centroid coordinates of the impurity region to be picked.

[0076] Figure 4 A flowchart illustrating the determination of coordinate transformation relationships in an embodiment of this disclosure is shown.

[0077] like Figure 4 As shown, the methods for determining this coordinate transformation relationship include:

[0078] Step S410: Acquire multiple images corresponding to multiple points; wherein each of the multiple images is acquired by an image acquisition device at the corresponding point facing the calibration plate; the calibration plate is set on the food tray;

[0079] Step S420: Detect feature points on the calibration board in each image and obtain multiple pixel coordinates of the feature points;

[0080] Step S430: Based on the multiple world coordinates and multiple pixel coordinates corresponding to multiple points in the world coordinate system, obtain the camera coordinate system of the image acquisition device and the perspective transformation matrix between the world coordinate system;

[0081] Step S440: Based on the perspective transformation matrix, obtain the coordinate transformation relationship.

[0082] This coordinate transformation relationship is used to obtain sorting position information that the sorting execution device (equivalent to the "hand") can understand based on the pixel coordinates in the image acquired by the image acquisition device (equivalent to the "eye"). Therefore, the method of determining the coordinate transformation relationship can also be called hand-eye calibration.

[0083] For example, during the hand-eye alignment process, the image acquisition device can be fixed to the sorting execution equipment, i.e., in a position that remains unchanged relative to the component in the sorting execution equipment used to pick out impurities (e.g., a suction pen). Figure 5 As shown, the image acquisition device 510 is installed at the end of the sorting execution device 520. The sorting execution device 520 includes a suction pen 521, and the relative position of the suction pen 521 and the image acquisition device 510 remains unchanged during the movement of the sorting execution device 520. The camera coordinate system of the image acquisition device can be referenced... Figure 5 Example (x) c y c , z c The world coordinate system uses the plane containing the food tray 540 as the xy plane. A calibration plate 530 is installed on the food tray. The world coordinate system can be referenced here. Figure 5 Example (x) w y w , z w ).

[0084] In specific implementation, such as Figure 6 As shown, the calibration plate 610 can be fixed on the food tray 620, and the image acquisition device can be moved to multiple points (e.g., Figure 6 (Nine points above the nine regions in the calibration plate). At each point, the world coordinates of the image acquisition device are recorded, and an image is acquired. The pixel coordinates of feature point 611 on the calibration plate are detected using a feature point detection algorithm. In this way, multiple world coordinates and multiple pixel coordinates can be obtained for multiple points. Among them, feature point 611 can be any corner point on the calibration plate.

[0085] Using the multiple world coordinates and multiple pixel coordinates obtained in the above process, the camera coordinate system (x) can be obtained. c y c , z c ) and world coordinate system (x) w y w , z w The perspective transformation matrix between the camera coordinate system and the world coordinate system is used to obtain the coordinate transformation relationship between pixel coordinates and world coordinates. In practical applications, the above coordinate transformation relationship can be obtained by combining the perspective transformation matrix between the camera coordinate system and the world coordinate system, as well as the intrinsic parameters and distortion parameters of the image acquisition device.

[0086] Specifically, the intrinsic parameters and distortion parameters of the image acquisition device can be predetermined. Then, based on the perspective transformation matrix and the camera's intrinsic and distortion parameters, a transformation formula between pixel coordinates and coordinates in the world coordinate system is established. Using this transformation formula, intrinsic parameters, distortion parameters, multiple world coordinates, and multiple pixel coordinates, a simultaneous equation is solved to obtain the perspective transformation matrix in the formula. After obtaining the perspective transformation matrix, the coordinate transformation relationship between pixel coordinates and world coordinates is obtained using the perspective transformation matrix, intrinsic parameters, and distortion parameters. This coordinate transformation relationship can be represented by a matrix as follows:

[0087]

[0088] Where, matrix f world_cam This indicates the coordinate transformation relationship, a ij This represents the element in the i-th row and j-th column of the matrix. It can be understood that using matrix f... world_cam Any coordinate in a food image (including the centroid coordinates of the area containing impurities to be picked) can be converted into corresponding world coordinates. Since the world coordinates of the image acquisition device are used as the world coordinates corresponding to the points during hand-eye calibration, matrix f can be used... world_cam The centroid coordinates can be converted into the displacement of the image acquisition device from the impurity. Combined with the relative position between the image acquisition device and the suction pen, the displacement required for the suction pen to move to the center of the impurity can be obtained. Refer to the following formula for specific calculations:

[0089]

[0090] Among them, (x c ,y c (x) represents the pixel coordinates where the impurity is located (i.e., the centroid coordinates mentioned above); p ,y p (x) represents the world coordinates of the pen's location. q ,y q ) represents the world coordinates of the image acquisition device, i.e. The relative position between the image acquisition device and the pen; (x o ,y o This represents the displacement required for the suction pen center to move to the impurity center (i.e., the picking position information).

[0091] In the hand-eye calibration process during food sorting, the sorting equipment needs to move above the machine, making it impossible to guarantee that the image acquisition device remains parallel to the machine and the food trays on it. Therefore, displacement along the z-axis occurs during this movement. Common hand-eye calibration methods use affine transformation matrices to represent the transformation between the camera coordinate system and the world coordinate system, resulting in low calibration accuracy and affecting the accuracy of impurity picking position information. However, in this embodiment, a perspective transformation matrix is ​​used to represent the transformation between the camera coordinate system and the world coordinate system. This effectively eliminates the error caused by z-axis offset, making the final coordinate transformation relationship more robust and improving the accuracy of the picking position information.

[0092] Based on the above explanation, it can be understood that in a food sorting scenario, the intrinsic parameters and distortion parameters of the image acquisition device are first determined, followed by hand-eye calibration to obtain the coordinate transformation relationship. During actual sorting, the image acquisition device captures images of the food on the tray. Electronic equipment then uses these images to obtain the picking position information and the model of the suction pen, and further uses this information to obtain sorting instruction information, enabling the sorting execution equipment to operate accordingly. In practical applications, PLC (Programmable Logic Controller) signals can be used to control the operation of the image acquisition device and the sorting execution equipment.

[0093] In practical applications, after a sorting action is completed, a new food image can be acquired and image segmentation performed to confirm whether the sorting process for the current food has been completed. Specifically, in another embodiment of this disclosure, the above method may further include: after the sorting execution device sorts the impurities from the food based on the sorting location information, acquiring a new food image, and returning to the step of segmenting at least one target region in the food image using an image segmentation model; if the number of impurity regions to be sorted determined based on the at least one target region is less than or equal to a preset number threshold, confirming that the sorting process for the food has been completed.

[0094] Figure 7 A schematic diagram illustrating the control flow of the camera (image acquisition device) and sorting execution equipment according to this embodiment is shown. Figure 7 As shown, according to this embodiment, the control flow for the camera and sorting execution equipment includes the following steps:

[0095] Step S710: Take a picture with the camera to obtain an image of the food.

[0096] Step S720: Model prediction and post-processing. Here, model prediction refers to using an image segmentation model to segment at least one target region in the food image. Post-processing refers to filtering and screening based on at least one target region to obtain the impurity region to be picked out.

[0097] In step S730, the PLC sends the picking position information corresponding to the processing result of step S720 to the sorting execution device, and the sorting execution device performs the picking.

[0098] Step S740: Take a picture with the camera to obtain a new food image.

[0099] Step S750, model prediction and post-processing.

[0100] Step S760: Based on the processing result of step S750, determine whether the processing result meets the requirements. If the requirements are not met, the PLC sends the picking position information corresponding to the processing result of step S750 to the sorting execution device and returns to step S730. If the requirements are met, proceed to step S770.

[0101] Step S770: Camera moves. Specifically, it moves to the next disk and returns to step S710.

[0102] As can be seen from the above embodiments, the food can be repeatedly sorted for impurities through multiple iterations until the number of areas with impurities to be sorted in the food meets the requirements. This ensures the accuracy and effectiveness of food sorting through automated control.

[0103] According to embodiments of this disclosure, this disclosure also provides an image processing apparatus for food sorting. Figure 8 A schematic block diagram of an image processing apparatus for food sorting according to an embodiment of this disclosure is shown. Figure 8 As shown, the image processing device for food sorting may include:

[0104] Image segmentation module 810 is used to segment at least one target region in a food image using an image segmentation model;

[0105] The region determination module 820 is used to determine the region of impurities to be picked out in the at least one target region;

[0106] The position determination module 830 is used to obtain the picking position information of impurities in the food based on the centroid coordinates of the impurity area to be picked; wherein the picking position information is used to instruct the sorting execution equipment to pick the impurities.

[0107] In some embodiments of this disclosure, such as Figure 9 As shown, the image processing device for food sorting further includes:

[0108] Model determination module 910 is used to determine the model of the suction pen used to pick up the impurities based on the area of ​​the impurity area to be picked up;

[0109] The information sending module 920 is used to send sorting instruction information to the sorting execution device; wherein the sorting instruction information is used to indicate the picking position information and the model of the suction pen.

[0110] In some embodiments of this disclosure, such as Figure 10 As shown, the image segmentation module 810 includes:

[0111] The slicing unit 1010 is used to slice the food image into M image slices; where M is an integer greater than or equal to 2;

[0112] The model processing unit 1020 is used to input each of the M image slices into the image segmentation model to obtain the target region output by the image segmentation model for each image slice.

[0113] The segmentation result summarization unit 1030 is used to obtain the at least one target region based on the target region output by the image segmentation model for each image slice.

[0114] In some embodiments of this disclosure, the region determination module 820 is specifically used for:

[0115] Based on the attribute information of each target area in the at least one target area, a region of impurities to be picked out is determined in the at least one target area; wherein the attribute information includes at least one of area, spacing and brightness.

[0116] In some embodiments of this disclosure, such as Figure 11 As shown, the region determination module 820 includes:

[0117] The region filtering unit 1110 is used to determine the K regions with the largest area among the N regions when the at least one target region contains N regions with a spacing less than a preset distance threshold; where N is an integer greater than or equal to 2, and K is a positive integer less than N.

[0118] The region determination unit 1120 is used to determine the region of impurities to be picked out in the K regions.

[0119] In some embodiments of this disclosure, such as Figure 12 As shown, the region determination module 820 includes:

[0120] The area filtering unit 1210 is used to filter out areas with an area smaller than a preset area threshold and / or a brightness smaller than a preset brightness threshold in the at least one target area, thereby obtaining the area of ​​impurities to be picked.

[0121] In some embodiments of this disclosure, the position determination module 830 is specifically used for:

[0122] Based on the predetermined coordinate transformation relationship and the centroid coordinates of the region of impurities to be picked, the picking location information of impurities in the food is obtained.

[0123] Among them, such as Figure 13 As shown, the device further includes:

[0124] The calibration image acquisition module 1310 is used to acquire multiple images corresponding to multiple points respectively; wherein, each of the multiple images is captured by an image acquisition device at the corresponding point facing the calibration plate; the calibration plate is set on the food tray;

[0125] The pixel coordinate determination module 1320 is used to detect feature points on the calibration plate in each image and obtain multiple pixel coordinates of the feature points;

[0126] The transformation matrix determination module 1330 is used to obtain the camera coordinate system of the image acquisition device and the perspective transformation matrix between the world coordinate system and the world coordinate system based on the multiple world coordinates corresponding to the multiple points and the multiple pixel coordinates.

[0127] The transformation relationship determination module 1340 is used to obtain the coordinate transformation relationship based on the perspective transformation matrix.

[0128] In some embodiments of this disclosure, such as Figure 14 As shown, it also includes:

[0129] The iteration module 1410 is used to acquire a new food image after the sorting execution device picks out the impurities in the food based on the picking position information, and return to the step of segmenting at least one target region in the food image using the image segmentation model.

[0130] The end confirmation module 1420 is used to confirm the completion of the sorting process of the food when the number of impurity areas to be sorted determined based on the at least one target area is less than or equal to a preset quantity threshold.

[0131] The specific functions and examples of each module and sub-unit block of the apparatus in this disclosure embodiment can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0132] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0133] Figure 15A schematic block diagram of an electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0134] like Figure 15 As shown, device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1502 or a computer program loaded from storage unit 1508 into random access memory (RAM) 1504. The RAM 1504 may also store various programs and data required for the operation of device 1500. The computing unit 1501, ROM 1502, and RAM 1504 are interconnected via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.

[0135] Multiple components in device 1500 are connected to I / O interface 1505, including: input unit 1506, such as keyboard, mouse, etc.; output unit 1507, such as various types of monitors, speakers, etc.; storage unit 1508, such as disk, optical disk, etc.; and communication unit 1509, such as network card, modem, wireless transceiver, etc. Communication unit 1509 allows device 1500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] The computing unit 1501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1501 performs the various methods and processes described above, such as an image processing method for food sorting. For example, in some embodiments, an image processing method for food sorting may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1500 via ROM 1502 and / or communication unit 1509. When the computer program is loaded into RAM 1504 and executed by the computing unit 1501, one or more steps of the image processing method for food sorting described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured by any other suitable means (e.g., by means of firmware) to perform an image processing method for food sorting.

[0137] According to embodiments of this disclosure, this disclosure also provides a food sorting system. Figure 16 This is a schematic block diagram of a food sorting system according to an embodiment of the present disclosure. Figure 16 As shown, the food sorting system includes an image acquisition device 1610, a sorting execution device 1630, and an electronic device 1620 as described in the above embodiments of this disclosure. The image acquisition device 1610 is used to acquire food images and send them to the electronic device 1620; the sorting execution device 1630 is used to receive sorting instruction information from the electronic device 1620 and, based on the sorting instruction information, pick out impurities from the food.

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0143] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method for food sorting, comprising: segmenting at least one target region in a food image using an image segmentation model; wherein the image segmentation model is configured to semantically segment the food image into a plurality of image regions, the plurality of image regions containing the at least one target region; the target region includes a point-like region and / or a strip-like region within a predetermined scale range; in a case where the at least one target region contains N regions with a distance less than a preset distance threshold, determining K regions with the largest area among the N regions; wherein N is an integer greater than or equal to 2, and K is a positive integer less than N; determining a to-be-picked impurity region among the K regions; obtaining picking position information of an impurity in the food based on a centroid coordinate of the to-be-picked impurity region; wherein the picking position information is used to instruct a sorting execution device to pick the impurity; the picking position information is a coordinate of a centroid of the impurity in a world coordinate system; wherein the segmenting at least one target region in a food image using an image segmentation model comprises: segmenting the food image into M image slices; wherein M is an integer greater than or equal to 2; inputting each of the M image slices into the image segmentation model respectively to obtain a target region output by the image segmentation model for each of the image slices; obtaining the at least one target region based on the target region output by the image segmentation model for each of the image slices.

2. The method of claim 1, further comprising: determining a model of a suction pen used to pick the impurity based on an area of the to-be-picked impurity region; sending sorting instruction information to the sorting execution device; wherein the sorting instruction information is used to instruct the picking position information and the model of the suction pen.

3. The method of claim 1, wherein, The determining the to-be-picked impurity region among the K regions comprises: filtering out regions with an area less than a preset area threshold and / or a brightness less than a preset brightness threshold among the K regions to obtain the to-be-picked impurity region.

4. The method of any one of claims 1-3, wherein, The obtaining picking position information of an impurity in the food based on a centroid coordinate of the to-be-picked impurity region comprises: obtaining the picking position information of the impurity in the food based on a predetermined coordinate conversion relationship and the centroid coordinate of the to-be-picked impurity region; wherein the coordinate conversion relationship is determined in the following manner: obtaining a plurality of images corresponding to a plurality of points respectively; wherein each of the plurality of images is obtained by an image acquisition device at a corresponding point towards a calibration board; the calibration board is arranged on a food tray; detecting a feature point on the calibration board in each of the images to obtain a plurality of pixel coordinates of the feature point; obtaining a perspective transformation matrix between a camera coordinate system of the image acquisition device and a world coordinate system based on a plurality of world coordinates in the world coordinate system corresponding to the plurality of points and the plurality of pixel coordinates; obtaining the coordinate conversion relationship based on the perspective transformation matrix.

5. The method of any one of claims 1-3, further comprising: obtaining a new food image after the sorting execution device picks the impurities in the food based on the picking position information, and returning to the step of segmenting at least one target region in the food image by using an image segmentation model; in a case where a number of impurity regions to be picked determined based on the at least one target region is less than or equal to a preset number threshold, confirming that the picking process of the food is completed.

6. An image processing device for food sorting, comprising: an image segmentation module configured to segment at least one target region in a food image by using an image segmentation model; wherein the image segmentation model is configured to semantically segment the food image into a plurality of image regions, and the plurality of image regions contain the at least one target region; the target region includes a point-like region and / or a strip-like region within a predetermined scale range; a region screening unit configured to, in a case where the at least one target region contains N regions with a distance less than a preset distance threshold, determine K regions with the largest area in the N regions; wherein N is an integer greater than or equal to 2, and K is a positive integer less than N; a region determination unit configured to determine an impurity region to be picked in the K regions; a position determination module configured to obtain picking position information of an impurity in the food based on a centroid coordinate of the impurity region to be picked; wherein the picking position information is used to instruct a sorting execution device to pick the impurity; and the picking position information is a coordinate of a centroid of the impurity in a world coordinate system; wherein the image segmentation module comprises: a slicing unit configured to slice the food image into M image slices; wherein M is an integer greater than or equal to 2; a model processing unit configured to input each of the M image slices into the image segmentation model to obtain a target region output by the image segmentation model for each of the image slices; a segmentation result aggregation unit configured to obtain the at least one target region based on the target region output by the image segmentation model for each of the image slices.

7. The device of claim 6, further comprising: a model determination module configured to determine a model of a suction pen used to pick the impurity based on an area of the impurity region to be picked; an information sending module configured to send picking instruction information to the sorting execution device; wherein the picking instruction information is used to instruct the picking position information and the model of the suction pen.

8. The apparatus of claim 6, wherein, The region determination unit comprises: a region filtering unit configured to filter out regions with an area less than a preset area threshold and / or a brightness less than a preset brightness threshold in the K regions to obtain the impurity region to be picked.

9. The apparatus of any one of claims 6-8, wherein, The position determination module is specifically configured to: obtain the picking position information of the impurity in the food based on a pre-determined coordinate conversion relationship and the centroid coordinate of the impurity region to be picked; wherein the device further comprises: An image acquisition module is configured to acquire a plurality of images corresponding to a plurality of positions respectively; each of the plurality of images is obtained by using an image acquisition device to capture a calibration board at a corresponding position; the calibration board is arranged on a food tray; A pixel coordinate determination module is configured to detect a feature point on the calibration board in each of the images to obtain a plurality of pixel coordinates of the feature point; A transformation matrix determination module is configured to obtain a perspective transformation matrix between a camera coordinate system of the image acquisition device and a world coordinate system based on a plurality of world coordinates corresponding to the plurality of positions in the world coordinate system and the plurality of pixel coordinates; A conversion relationship determination module is configured to obtain the coordinate conversion relationship based on the perspective transformation matrix.

10. The apparatus according to any one of claims 6-8, further comprising: An iteration module is configured to, after the sorting execution device picks the impurities in the food based on the picking position information, acquire a new food image, and return to the step of segmenting at least one target region in the food image by using the image segmentation model; An end confirmation module is configured to, in a case where a number of impurity regions to be picked determined based on the at least one target region is less than or equal to a preset number threshold, confirm that the picking process of the food is completed.

11. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method according to any one of claims 1-5.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-5.

14. A food sorting system comprising an image acquisition device, a sorting execution device, and the electronic device according to claim 11; wherein the image acquisition device is configured to acquire a food image and send the food image to the electronic device; the sorting execution device is configured to receive sorting instruction information from the electronic device, and pick impurities in food based on the sorting instruction information.

Images