Online detection method and device for apple surface defects

Abstract

The present invention provides an online detection method and device for apple surface defects, the method comprising: obtaining images of a detection target passing through a plurality of different detection stations, and splicing to generate a composite image of the detection area; inputting the composite image of the detection area into a trained target detection model, and outputting the detection result of the defect area; wherein the target detection model is trained based on training samples of marked defect areas, stalk areas, and calyx areas, and then obtained after sparse training and channel pruning. The method performs detection through the target detection model obtained after sparse training and channel pruning, and has the characteristics of small calculation amount and high speed while ensuring accuracy, ensuring that defective apples are quickly and accurately detected and sorted without being interfered by the apple stalk / calyx area. Furthermore, a composite image of the detection area is generated by splicing images of a plurality of different detection stations, which not only improves the detection speed but also improves the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of fruit and vegetable detection, and in particular to an online detection method and device for apple surface defects. Background Art

[0002] At present, most apple growers do not have enough knowledge about apple grading. They only remove apples that cannot be put on the market and roughly grade the apples by size. The grading method mainly relies on manual grading, and the grading results are not accurate enough. The apple external quality sorting machine based on machine vision can comprehensively detect the main external quality indicators of apples such as size, color, shape and surface defects. It has the advantages of high precision, high degree of automation and non-destructive detection, and has been widely used in apple post-harvest grading.

[0003] The automatic detection methods of apple size, shape, and color indicators are relatively mature, but the rapid identification of defects has always been an obstacle to real-time apple grading. The main difficulty is that the defective area on the apple surface and the stalk / calyx area have very similar grayscale values ​​in the image. Both appear as dark spots in the image, making it difficult to distinguish between defects and stalks / calyx. Therefore, whether the stalk / calyx area in the apple image can be accurately detected becomes a prerequisite for correctly identifying apple surface defects. The existing methods for detecting apple stalks / calyx mainly include:

[0004] The YOLO target detection model is an object positioning and recognition algorithm based on a deep neural network. It combines the two stages of candidate area selection and object recognition into one, which greatly improves the detection speed and is suitable for real-time detection systems. However, this method cannot meet the needs of real-time detection of apple surface defects. Because apple sorting must ensure both high detection accuracy and the sorting output to meet actual production needs, and to minimize the hardware cost of the sorting system, it is necessary to collect full-surface images of apples in a short time and process them in real time. The method of reducing hardware costs by using one camera to detect two sorting channels further increases the requirements for detection speed. In actual production, the working method of using two near-infrared cameras to collect apple images in two sorting channels for detection and grading requires the defect detection of 12 apple images to be completed within 30 milliseconds, which cannot be achieved by simply using the YOLO target detection method. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides an online detection method and device for apple surface defects.

[0006] The present invention provides an online detection method for apple surface defects, comprising: acquiring images of a detection target passing through a plurality of different detection stations, and splicing and generating a composite image of a detection area; inputting the composite image of the detection area into a trained target detection model, and outputting a detection result of the defect area; wherein the target detection model is trained based on training samples of marked defect areas, fruit stalk areas and calyx areas, and then obtained through sparse training and channel pruning.

[0007] According to an online method for detecting surface defects of apples according to an embodiment of the present invention, the method of acquiring images of the detection target passing through multiple different detection stations and stitching them together to generate a composite image of the detection area includes: acquiring images of the detection target passing through multiple different detection stations through multiple cameras, respectively, and stitching together to generate composite images of the detection area corresponding to each camera; accordingly, inputting each composite image of the detection area into a trained target detection model, and outputting a defect area detection result of each composite image of the detection area; and determining the defect result of the detection target according to the defect area detection results of all the composite images of the detection area.

[0008] According to an embodiment of the present invention, an online detection method for apple surface defects, wherein a plurality of cameras are used to respectively acquire images of a detection target passing through a plurality of different detection stations, and respectively stitching together a composite image of the detection area corresponding to each camera, comprises: acquiring near-infrared light images of a detection target passing through a plurality of different detection stations through a plurality of near-infrared cameras, and respectively stitching together a composite image of the detection area corresponding to each camera; or acquiring near-infrared light images and RGB images of a detection target passing through a plurality of different detection stations through a plurality of near-infrared cameras and a plurality of RGB cameras, and respectively stitching together a composite image of the detection area corresponding to each camera.

[0009] According to an online detection method for apple surface defects according to an embodiment of the present invention, the defect result of the detection target is determined based on the defect area detection results of the composite images of all detection areas, including: determining the number of defect areas of the detection target in each composite image of the detection area, and storing the number of defect areas in each composite image of the detection area in a circular coding queue; when the detection target passes the last inspection station, if the number of defects in the circular coding queue of the detection target is zero, it is judged as a normal result, and if the number of defects is not zero, it is judged as a defective result, and the defect number information recorded by the detection target in the circular coding queue is cleared.

[0010] According to an embodiment of the present invention, the online detection method for apple surface defects, before obtaining images of the detection target passing through multiple different detection stations, also includes: obtaining images of multiple training samples passing through multiple different detection stations to generate a composite image of the detection area of ​​the samples; marking the defect area, stalk area and calyx area in the composite image of the sample detection area, and training the original model to obtain a trained detection model.

[0011] According to an embodiment of the present invention, the online detection method for apple surface defects, after obtaining the trained detection model, further includes: adding a BN (Batch Normalization) layer scale factor coefficient as a constraint term to the loss function of the detection model, using the scale factor to perform sparse training on the detection model, and pruning the detection model according to the sparse training result to obtain a pruned detection model.

[0012] According to an embodiment of the present invention, the online detection method for apple surface defects further comprises, after obtaining the pruned detection model: calculating the ratio of the L1 norm of any two prediction boxes of the pruned detection model in the same direction in the horizontal and vertical directions to the minimum width and height of the two prediction boxes, respectively; summing the ratios in the two directions, comparing them with a set threshold, and judging the redundant detection box according to the comparison result; and correcting the pruned detection model according to the judgment result of the redundant detection box to obtain the target detection model.

[0013] The present invention also provides an online detection device for apple surface defects, comprising: an acquisition module, used to obtain images of a detection target passing through multiple different detection stations, and splicing them to generate a composite image of the detection area; a processing module, used to input the composite image of the detection area into a trained target detection model, and output the detection result of the defective area; wherein the target detection model is trained based on training samples of marked defective areas, stalk areas and calyx areas, and then obtained after sparse training and channel pruning.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for online detection of apple surface defects are implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for online detection of apple surface defects.

[0016] The method and device for online detection of apple surface defects provided by the present invention perform detection through a target detection model obtained through sparse training and channel pruning, which has the characteristics of small calculation amount and fast speed while ensuring accuracy, and ensures that defective apples can be quickly and accurately detected and sorted without interference from the apple stalk / calyx area. Furthermore, a composite image of the detection area is generated by splicing images of multiple different detection stations, which not only improves the detection speed but also improves the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic flow chart of the online detection method for apple surface defects provided by the present invention;

[0019] Figure 2 It is a schematic diagram of the structure of the multi-spectral machine vision apple online detection system provided by the present invention;

[0020] Figure 3 It is a schematic diagram of generating a composite image of a detection area provided by the present invention;

[0021] Figure 4 It is a structural schematic diagram of an online detection device for apple surface defects provided by the present invention;

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Combine the following Figure 1-Figure 5 The invention describes an online method and device for detecting apple surface defects. Figure 1 FIG. 1 is a flow chart of an online detection method for apple surface defects provided by the present invention. Figure 1 As shown, the present invention provides an online detection method for apple surface defects, comprising:

[0025] 101. Acquire images of the inspection target passing through a plurality of different inspection stations, and stitch them together to generate a composite image of the inspection area.

[0026] Figure 2 This is a schematic diagram of the structure of the multi-spectral machine vision apple online inspection system provided by the present invention. The machine vision inspection system is composed of 4 industrial cameras, including 2 infrared cameras and 2 RGB cameras. Each camera is responsible for simultaneously collecting apple images in two channels. When each apple passes through three consecutive workstations, the 4 cameras continuously collect 9 images. A total of 36 images are collected for each apple, including 18 RGB images and 18 near-infrared images. The inspection speed of each channel of the sorting line is 5 fruits / second.

[0027] Specifically, in order to improve the online detection speed of apple surface defects, the present invention composites the images of three consecutive detection stations within the field of view of a camera, such as a near-infrared industrial camera, to generate a grayscale image, and uses the image as the input image of the model. Correspondingly, during model training, a large number of composite images of apple samples are collected by this method, and defect areas, sunken areas, fruit stem areas and calyx areas are marked to generate a near-infrared apple image database for model training.

[0028] like Figure 3 As shown, the generation process of the composite image can be expressed by the following formula:

[0029]

[0030] Where I is the generated composite image, I1, I2, and I3 are images of three consecutive workstations within the field of view, and the image grayscale values ​​of the remaining positions of the composite image are set to 0.

[0031] The embodiment of the present invention can improve the detection accuracy by repeatedly passing through the assembly line, that is, each apple passes through the assembly line multiple times, such as 9 times, in the process of passing through three consecutive workstations, and multiple images are collected.

[0032] 102. Input the composite image of the detection area into a trained target detection model, and output the detection result of the defect area. The target detection model is trained based on training samples of the marked defect area, peduncle area and calyx area, and then obtained after sparse training and channel pruning.

[0033] The target detection model of the present invention can be obtained based on YOLOv4 model training, and the trained detection model is subjected to sparse processing. Moreover, based on the network after sparse processing, the network lines are layer pruned according to set parameters, and then channel pruning is performed to achieve compression of the network structure.

[0034] Through the network after training and channel pruning, the composite image of the detection area of ​​the detection target is processed, and the recognition results of the defect area, fruit stalk area and calyx area can be obtained accordingly.

[0035] The online detection method for apple surface defects of the present invention performs detection through a target detection model obtained after sparse training and channel pruning, and has the characteristics of small calculation amount and fast speed while ensuring accuracy, ensuring that defective apples are quickly and accurately detected and sorted without interference from the apple stalk / calyx area. Furthermore, a composite image of the detection area is generated by splicing images of multiple different detection stations, which not only improves the detection speed but also improves the detection accuracy.

[0036] In one embodiment, the method of acquiring images of the inspection target passing through multiple different inspection stations and stitching them together to generate a composite image of the inspection area includes: acquiring images of the inspection target passing through multiple different inspection stations through multiple cameras, and stitching them together to generate composite images of the inspection area corresponding to each camera; accordingly, inputting each composite image of the inspection area into a trained target detection model, and outputting defect area detection results of each composite image of the inspection area; and determining the defect results of the inspection target based on the defect area detection results of all the composite images of the inspection area.

[0037] Specifically, there are multiple composite images of the detection area of ​​each scene. For example, through two visible light cameras and two near-infrared cameras of specified bands, four cameras are used to perform online inspection of apples in the dual-channel sorting line. 18 RGB images and 18 near-infrared images are collected for each apple. The inspection speed of each channel of the sorting line can be set to 5 fruits / second.

[0038] The embodiment of the present invention uses multiple cameras to perform detection, which can achieve image acquisition at multiple angles and further improve detection accuracy.

[0039] In one embodiment, the method of acquiring images of the detection target passing through multiple different detection stations through multiple cameras, and stitching them together to generate composite images of the detection area corresponding to each camera, includes: acquiring near-infrared light images of the detection target passing through multiple different detection stations through multiple near-infrared cameras, and stitching them together to generate composite images of the detection area corresponding to each camera; or acquiring near-infrared light images and RGB images of the detection target passing through multiple different detection stations through multiple near-infrared cameras and multiple RGB cameras, and stitching them together to generate composite images of the detection area corresponding to each camera.

[0040] Due to the wide variety of apples, the surface color distribution of each apple has its own characteristics. For example, the surface color of the green apple is green, the surface color of the yellow apple is yellow, the surface color of the red apple is all red, and the surface color of the striped apple is red stripes. These changing color distributions further make it difficult to detect defective areas.

[0041] The embodiment of the present invention uses a near-infrared camera to capture apple images, which not only utilizes the characteristics of near-infrared light to ensure obvious defect areas, but also is not affected by surface color. The model trained by one variety of apples can be directly used to detect surface defects of other varieties of apples, greatly improving production efficiency.

[0042] In one embodiment, before obtaining images of the inspection target passing through multiple different inspection stations, the method further includes: obtaining images of multiple training samples passing through multiple different inspection stations to generate a composite image of the inspection area of ​​the samples; marking defective areas, fruit stalk areas, and calyx areas in the composite image of the sample inspection area, and training the original model to obtain a trained inspection model.

[0043] For example, extract images of apples in the near-infrared band, generate a composite image of the detection area, mark the defective areas, stalk areas and calyx areas in the generated image, and train the original model to achieve accurate detection of defective areas in near-infrared apple images.

[0044] In one embodiment, after obtaining the trained target detection model, the method further includes: adding a BN layer scale factor coefficient as a constraint term to the loss function of the detection model, using the scale factor to perform sparse training on the detection model, and pruning the detection model according to the sparse training result to obtain a pruned detection model.

[0045] After completing the model training, since the detection speed cannot meet the real-time detection requirements, it is necessary to further prune and compress the model while ensuring the detection accuracy. In order to compress the model, it is necessary to select the network channels that play an important role in the detection and retain them, while removing the unimportant network channels from the model, so as to achieve the purpose of reducing model parameters and improving the model detection speed.

[0046] Therefore, the BN layer scale factor coefficient is added to the loss function of the target detection model as a constraint term, and the scale factor γ is used to perform sparse training on the detection model, so as to clarify the size of the effects of different channels in the network model.

[0047] In one embodiment, the loss function in the sparse training is determined based on the position error, the error of the presence or absence of an object, the category error, and a regularization constraint term including a BN layer scale factor.

[0048] Specifically, the loss function when the model is sparse is as follows:

[0049] Loss = L CIoU +L conf +L cls +λ∑g(γ) (2)

[0050] Among them, L CIoU is the position error, L conf is the error of whether there is an object, L cls is the category error, λ∑g(γ) is the L1 regularization constraint; λ is the coefficient, and γ is the BN layer scale factor.

[0051] After the sparse training is completed, the model can be pruned according to the specified parameters. According to the set threshold, the specified proportion of channels can be pruned, and then the layer pruning can be performed to pruned the specified number of layers with the smallest effect.

[0052] In one embodiment, after obtaining the pruned detection model, the method further includes: calculating the ratio of the L1 norm of any two prediction boxes of the pruned detection model in the same direction in the horizontal and vertical directions to the minimum width and minimum height of the two prediction boxes respectively; summing the ratios in the two directions, comparing them with a set threshold, and judging the redundant detection box according to the comparison result; and correcting the pruned detection model according to the judgment result of the redundant detection box to obtain the target detection model.

[0053] After pruning is completed, the model needs to be fine-tuned to ensure that the model recovers the detection accuracy before pruning. Since redundant prediction frames will appear after fine-tuning, the L1 norm non-maximum suppression method based on the minimum bounding box is used to quickly delete redundant defect detection frames and restore defect detection accuracy. It is necessary to calculate the ratio of the L1 norm of the same direction vertices of any two prediction frames (such as the upper left vertex) in the horizontal and vertical directions to the minimum width and minimum height of the two prediction frames, and then compare them with the set threshold to determine the redundant detection frame. When the value d after the sum of the ratios is less than the set threshold, it is considered that there is a redundant frame and any prediction frame is removed. Otherwise, both prediction frames must be retained. The calculation process of d is shown in the following formula:

[0054] d=|x2-x1| / W+|y2-y1| / H (3)

[0055] Where W = min(w1,w2) and H = min(h1,h2), respectively represent the minimum width and height of the two prediction boxes. (x1,y1) and (x2,y2) represent the coordinates of the upper left corners of the two prediction boxes.

[0056] The embodiment of the present invention corrects the pruned detection model through a redundant detection frame, so that the detection model can still maintain a high detection accuracy after pruning.

[0057] In one embodiment, the defect result of the detection target is determined based on the defect area detection results of the composite images of all detection areas, including: determining the number of defect areas of the detection target in each composite image of the detection area, and storing the number of defect areas in each composite image of the detection area in a circular coding queue; when the detection target passes through the last detection station, if the number of defects in the circular coding queue of the detection target is zero, it is judged as a normal result, and if the sum of the number of defects is not zero, it is judged as a defective result, and the defect number information recorded in the circular coding queue of the detection target is cleared.

[0058] Specifically, after completing the model fine-tuning, the compressed target detection model is deployed on the online detection computer, and each time the monitoring target image, such as a near-infrared image, is collected to generate a composite input image, and the image size can be set to 416 × 416. The output results of the target detection model include the position coordinates of three types of areas: the fruit stalk area, the calyx area, and the defect area.

[0059] The generated composite near-infrared image is restored and decomposed, and the position coordinates are used to determine the number of defective areas in the apple image at each station, and the number of defective areas in each near-infrared apple image is stored in the circular coding queue. When the apple passes the last inspection station, check whether the sum of the number of defects in the circular coding queue of the apple is zero. If the sum of the number of defects is zero, it means that the apple is normal. If the sum of the number of defects is not zero, it means that the apple is defective. Finally, the number of defects recorded in the circular coding queue of the apple is cleared.

[0060] In order to further test the effectiveness of the method of the present invention, 238 Striped Fuji apples, 120 Red Fuji apples, 116 Green apples, and 136 Yellow Marshal apples were tested on the detection system using the YOLO model trained and pruned with Striped Fuji apples. The detection accuracy of the method proposed in the present invention was as high as 93.9%, reaching a leading level.

[0061] The online detection device for apple surface defects provided by the present invention is described below. The online detection device for apple surface defects described below and the online detection method for apple surface defects described above can be referenced to each other.

[0062] Figure 4 FIG. 1 is a schematic diagram of the structure of the online detection device for apple surface defects provided by the present invention. Figure 4As shown, the online detection device for apple surface defects includes: an acquisition module 401 and a processing module 402. The acquisition module 401 is used to obtain images of the detection target passing through multiple different detection stations, and splice to generate a composite image of the detection area; the processing module 402 is used to input the composite image of the detection area into a trained target detection model, and output the detection result of the defect area; wherein the target detection model is trained based on training samples of marked defect areas, fruit stem areas and calyx areas, and then obtained after sparse training and channel pruning.

[0063] In one device embodiment, the acquisition module 401 is used to: obtain images of the inspection target passing through multiple different inspection stations through multiple cameras, and respectively stitch together to generate composite images of the inspection area corresponding to each camera; accordingly, the processing module 402 is used to input each inspection area composite image into a trained target detection model, and output the defect area detection result of each inspection area composite image; determine the defect result of the inspection target based on the defect area detection results of all inspection area composite images.

[0064] In one device embodiment, the acquisition module 401 is used to: respectively acquire near-infrared light images of the detection target passing through multiple different detection stations through multiple near-infrared cameras, and respectively splice them to generate composite images of the detection area corresponding to each camera; or, respectively acquire near-infrared light images and RGB images of the detection target passing through multiple different detection stations through multiple near-infrared cameras and multiple RGB cameras, and respectively splice them to generate composite images of the detection area corresponding to each camera.

[0065] In one device embodiment, the processing module 402 is used to: determine the number of defective areas of the detection target in each detection area composite image, and store the number of defective areas in each detection area composite image in a circular coding queue; when the detection target passes through the last detection station, if the number of defects in the circular coding queue of the detection target is zero, it is judged as a normal result, and if the number of defects is not zero, it is judged as a defective result, and the defect number information recorded in the circular coding queue of the detection target is cleared.

[0066] In one embodiment of the device, it also includes a model generation module, which is used to: obtain images of multiple training samples passing through multiple different inspection stations to generate a composite image of the inspection area of ​​the sample; mark the defect area, fruit stalk area and calyx area in the composite image of the sample inspection area, and train the original model to obtain a trained detection model.

[0067] In one device embodiment, the model generation module is also used to: after obtaining the trained detection model, add the batch normalization BN layer scale factor coefficient as a constraint term to the loss function of the detection model, use the scale factor to perform sparse training on the detection model, and prune the detection model based on the sparse training results to obtain the pruned detection model.

[0068] In one device embodiment, the model generation module is also used to: after obtaining the pruned detection model, calculate the L1 norm of any two prediction boxes of the pruned detection model in the same direction in the horizontal and vertical directions, and the ratio of the minimum width and minimum height of the two prediction boxes respectively; after summing the ratios in the two directions, compare them with the set threshold, and judge the redundant detection box according to the comparison result; according to the judgment result of the redundant detection box, correct the pruned detection model to obtain the target detection model.

[0069] The device embodiments provided in the embodiments of the present invention are for implementing the above-mentioned method embodiments. For specific processes and detailed contents, please refer to the above-mentioned method embodiments, which will not be repeated here.

[0070] The apple surface defect online detection device provided by the embodiment of the present invention performs detection through the target detection model obtained after sparse training and channel pruning, and has the characteristics of small calculation amount and fast speed while ensuring accuracy, ensuring that defective apples are quickly and accurately detected and sorted without interference from the apple stalk / calyx area. Furthermore, a composite image of the detection area is generated by splicing images of multiple different detection stations, which not only improves the detection speed but also improves the detection accuracy.

[0071] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communications interface 502 and the memory 503 communicate with each other through the communication bus 504. The processor 501 may call the logic instructions in the memory 503 to execute an online detection method for apple surface defects, the method comprising: obtaining images of the detection target passing through a plurality of different detection stations, and splicing and generating a composite image of the detection area; inputting the composite image of the detection area into a trained target detection model, and outputting the detection result of the defect area; wherein the target detection model is obtained after training based on training samples of the marked defect area, the peduncle area and the calyx area, and then undergoing sparse training and channel pruning.

[0072] In addition, the logic instructions in the above-mentioned memory 503 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the online detection method for apple surface defects provided by the above-mentioned methods, and the method includes: acquiring images of the detection target passing through multiple different detection stations, and splicing them to generate a composite image of the detection area; inputting the composite image of the detection area into a trained target detection model, and outputting the detection result of the defective area; wherein the target detection model is trained based on training samples of marked defective areas, stalk areas and calyx areas, and then obtained after sparse training and channel pruning.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the online detection method for apple surface defects provided in the above-mentioned embodiments, the method comprising: acquiring images of the detection target passing through multiple different detection stations, and splicing them to generate a composite image of the detection area; inputting the composite image of the detection area into a trained target detection model, and outputting the detection result of the defective area; wherein the target detection model is trained based on training samples of marked defective areas, stalk areas and calyx areas, and then obtained after sparse training and channel pruning.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An online detection method for apple surface defects, characterized in that: include: Acquire images of the inspection target passing through multiple different inspection stations, and stitch them together to generate a composite image of the inspection area; Inputting the composite image of the inspection area into a trained target detection model, and outputting the inspection result of the defect area; The target detection model is obtained after training with training samples of marked defective areas, fruit stalk areas and calyx areas, and then undergoing sparse training and channel pruning; Before acquiring the images of the inspection target passing through a plurality of different inspection stations, the method further includes: Acquire images of multiple training samples passing through multiple different inspection stations to generate a composite image of the inspection area of ​​the samples; Marking the defective area, the fruit stalk area and the calyx area in the composite image of the sample detection area, and training the original model to obtain a trained detection model; After obtaining the trained detection model, the method further includes: Add the scale factor coefficient of the batch normalization (BN) layer as a constraint term in the loss function of the detection model, use the scale factor to perform sparse training on the detection model, and prune the detection model according to the sparse training results to obtain a pruned detection model; After obtaining the pruned detection model, the method further includes: Calculate the ratio of the L1 norm of any two predicted boxes in the same direction of the pruned detection model in the horizontal and vertical directions to the minimum width and height of the two predicted boxes respectively; After summing the ratios of the two directions, the ratios are compared with the set threshold, and the redundant detection frame is determined based on the comparison result; According to the judgment result of the redundant detection frame, the pruned detection model is corrected to obtain the target detection model.

2. The method for online detection of apple surface defects according to claim 1, characterized in that: The method of acquiring images of the detection target passing through a plurality of different detection stations and stitching them together to generate a composite image of the detection area comprises: acquiring images of the detection target passing through a plurality of different detection stations through a plurality of cameras respectively, and stitching them together to generate a composite image of the detection area corresponding to each camera respectively; Accordingly, each inspection area composite image is input into the trained target detection model, and the defect area detection result of each inspection area composite image is output; The defect result of the inspection target is determined based on the defect area inspection results of the composite image of all inspection areas.

3. The method for online detection of apple surface defects according to claim 2, characterized in that: The method of acquiring images of the detection target passing through multiple detection stations through multiple cameras and splicing them to generate composite images of the detection area corresponding to each camera includes: By using multiple near-infrared cameras, near-infrared images of the inspection target passing through multiple different inspection stations are respectively acquired, and composite images of the inspection area corresponding to each camera are generated by splicing them respectively; Alternatively, multiple near-infrared cameras and multiple RGB cameras are used to obtain near-infrared images and RGB images of the detection target passing through multiple different detection stations, and these images are stitched together to generate composite images of the detection area corresponding to each camera.

4. The method for online detection of apple surface defects according to claim 2, characterized in that: The step of determining the defect result of the detection target according to the defect area detection results of the composite image of all detection areas includes: Determine the number of defective areas of the inspection target in each inspection area composite image, and store the number of defective areas in each inspection area composite image in a cyclic encoding queue; When the inspection target passes the last inspection station, if the number of defects in the circular coding queue of the inspection target is zero, it is judged as a normal result. If the number of defects is not zero, it is judged as a defective result, and the defect number information recorded in the circular coding queue of the inspection target is cleared.

5. An online detection device for apple surface defects, characterized in that: include: The acquisition module is used to obtain images of the inspection target passing through multiple different inspection stations and stitch them together to generate a composite image of the inspection area; A processing module, used for inputting the composite image of the detection area into a trained target detection model and outputting the detection result of the defect area; The target detection model is obtained after training with training samples of marked defective areas, fruit stalk areas and calyx areas, and then undergoing sparse training and channel pruning; Before acquiring the images of the inspection target passing through a plurality of different inspection stations, the method further includes: Acquire images of multiple training samples passing through multiple different inspection stations to generate a composite image of the inspection area of ​​the samples; Marking the defective area, the fruit stalk area and the calyx area in the composite image of the sample detection area, and training the original model to obtain a trained detection model; After obtaining the trained detection model, the method further includes: Add the scale factor coefficient of the batch normalization (BN) layer as a constraint term in the loss function of the detection model, use the scale factor to perform sparse training on the detection model, and prune the detection model according to the sparse training results to obtain a pruned detection model; After obtaining the pruned detection model, the method further includes: Calculate the ratio of the L1 norm of any two predicted boxes in the same direction of the pruned detection model in the horizontal and vertical directions to the minimum width and height of the two predicted boxes respectively; After summing the ratios of the two directions, the ratios are compared with the set threshold, and the redundant detection frame is determined based on the comparison result; According to the judgment result of the redundant detection frame, the pruned detection model is corrected to obtain the target detection model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the online detection method for apple surface defects as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the online method for detecting apple surface defects as described in any one of claims 1 to 4 are implemented.

Images