A quasi-planar foreign matter detection method, device and medium
By acquiring images of the surface of large objects and using location feature points and machine learning techniques to eliminate noise before foreign object detection, the problems of low detection accuracy and high false alarm rate in existing technologies are solved, and high-precision foreign object identification is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG SYNTHESIS ELECTRONICS TECH
- Filing Date
- 2022-12-30
- Publication Date
- 2026-05-22
AI Technical Summary
Existing technologies struggle to effectively detect small objects larger than 5 millimeters on the surface of large objects, and are easily affected by damage, paint peeling, contamination, and surface texture patterns, resulting in a high false alarm rate.
The system collects quasi-planar surface images of objects in different directions, performs image transformation processing through preset positioning feature points, eliminates noise, and uses machine learning techniques for foreign object detection, including SIFT feature extraction, Codebook background model and clustering algorithm, to generate a list of suspicious targets and collect close-up images for accurate detection.
It improves the accuracy of foreign object detection, reduces the number of false alarms, and reduces interference caused by surface damage, paint peeling, and contamination of objects.
Smart Images
Figure CN115953379B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image analysis, specifically to a quasi-planar foreign object detection method, device, and medium. Background Technology
[0002] Many applications require the detection of tiny objects on the surface of large objects. For example, foreign object detection on the surface of shipping containers requires the ability to detect small objects larger than 5 millimeters scattered on the top surface of the container.
[0003] Traditional detection methods include image-based detection, but this approach struggles to handle interference from surface damage, paint peeling, contamination, and surface texture patterns on large objects, easily leading to false alarms. Methods based on lidar and structured light are also available, but due to accuracy limitations, they struggle to detect small targets as small as 5 millimeters, such as those on large objects like shipping containers. Summary of the Invention
[0004] To address the aforementioned problems, this application proposes a quasi-planar foreign object detection method, comprising:
[0005] Collect quasi-planar surface images of the object in different directions and generate multiple corresponding first digital images;
[0006] Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points.
[0007] After extracting graphic feature points from the second digital image and removing noise, a set of outliers is obtained, and suspicious targets are identified based on the set of outliers.
[0008] Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
[0009] In one example, based on a number of pre-defined positioning feature points on a quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points, specifically including:
[0010] The quasi-plane surface of the object is determined to be rectangular. The four sides of the rectangle are detected according to the Hough transform, and the coordinates of the four corner vertices of the rectangle are obtained based on the four sides. The four corner points of the rectangle are used as the positioning feature points of the quasi-plane surface.
[0011] Based on the positioning feature points, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points. The image transformation processing includes perspective transformation, displacement, and scaling transformation.
[0012] In one example, after extracting graphic feature points from the second digital image and removing noise, a set of outliers is obtained, specifically including:
[0013] Graphic feature points are extracted from the second digital image using the SIFT feature method;
[0014] Exclude graphic feature points with the same coordinates from multiple second digital images;
[0015] Determine the codebook background model based on historical records, and eliminate regular interference points based on the codebook background model;
[0016] Isolated feature points within a specified radius are excluded based on clustering methods;
[0017] The set of outliers is obtained from the remaining graphic feature points after exclusion.
[0018] In one example, the identification of suspicious targets based on the set of outliers specifically includes:
[0019] Based on the set of outliers, a clustering algorithm is used to obtain the number of suspicious targets and the coordinates and size of each suspicious target on the quasi-planar surface of the object, thereby generating a list of suspicious targets.
[0020] In one example, foreign object detection of the suspicious target based on the close-up digital image specifically includes:
[0021] For the close-up digital image, the machine learning technique is used to detect and identify whether the suspicious target is a foreign object. The machine learning technique uses at least one of SVM, Bayesian inference, and convolutional neural network. The detection target of the machine learning technique includes at least one of target classification, target detection, and image segmentation.
[0022] In one example, the method further includes:
[0023] Based on the coordinates of the foreign object in the second digital image and the coordinate transformation relationship between the first digital image and the second digital image, the foreign object information in the first digital image is determined, and the foreign object information includes the type, location, and size of the foreign object;
[0024] The foreign object information is sent to the relevant equipment for further processing.
[0025] In one example, the foreign object information is sent to a relevant device for further processing, specifically including:
[0026] A panoramic image of the object is acquired, and the foreign object is marked in the panoramic image according to the foreign object information.
[0027] In one example, before acquiring quasi-planar surface images of the object in different directions, the method further includes:
[0028] A foreign object detection device is defined, comprising a first image acquisition device, a second image acquisition device, and a computing module. Multiple first image acquisition devices are mounted in fixed positions above the side of the object's quasi-plane surface to acquire images of the quasi-plane surface. The second image acquisition device carries a gimbal and a zoom lens to acquire close-up and panoramic images of the object. The computing module is connected to the first and second image acquisition devices to perform corresponding image data processing.
[0029] On the other hand, this application also proposes a quasi-planar foreign object detection device, comprising:
[0030] At least one processor; and,
[0031] A memory communicatively connected to the at least one processor; wherein,
[0032] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:
[0033] Collect quasi-planar surface images of the object in different directions and generate multiple corresponding first digital images;
[0034] Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points.
[0035] After extracting graphic feature points from the second digital image and removing noise, a set of outliers is obtained, and suspicious targets are identified based on the set of outliers.
[0036] Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
[0037] On the other hand, this application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:
[0038] Collect quasi-planar surface images of the object in different directions and generate multiple corresponding first digital images;
[0039] Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points.
[0040] After extracting graphic feature points from the second digital image and removing noise, a set of outliers is obtained, and suspicious targets are identified based on the set of outliers.
[0041] Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
[0042] The quasi-planar foreign object detection method proposed in this application can bring the following beneficial effects:
[0043] In the foreign object detection process, a preliminary screening is performed using digital images. Then, close-up images of suspicious targets are acquired for precise foreign object detection. Compared to detection methods such as LiDAR and structured light, this approach effectively improves detection accuracy. Furthermore, noise is eliminated during the preliminary screening process, reducing interference from surface damage, paint peeling, contamination, and surface texture patterns, thus lowering the number of false alarms. Attached Figure Description
[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0045] Figure 1 This is a flowchart illustrating the quasi-planar foreign object detection method in the embodiments of this application;
[0046] Figure 2 This is a schematic diagram of the quasi-planar foreign object detection device in the embodiments of this application. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0048] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0049] like Figure 1 As shown, this application provides a quasi-planar foreign object detection method, including:
[0050] S101: Acquire quasi-planar surface images of the object in different directions, and generate multiple corresponding first digital images.
[0051] An object typically refers to a large object, specifically an object whose dimensions exceed a preset size, such as a shipping container. A quasi-plane refers to an object whose surface is essentially flat, but may also have a certain degree of concavity or convexity, with the concave or convex surfaces below a preset thickness.
[0052] The acquired quasi-planar surface image is processed by a computer to obtain a digital image that can be processed by a computer. A digital image is an image represented in the form of a two-dimensional digital group, and its digital unit is a pixel.
[0053] A pre-configured foreign object detection device can be used to execute the quasi-planar foreign object detection method described in this application. The foreign object detection device includes a first image acquisition device, a second image acquisition device, and a computing module. Multiple first image acquisition devices (e.g., two) are mounted in fixed positions above the sides of the quasi-planar surface of the object (e.g., above both sides) to acquire images of the quasi-planar surface. The second image acquisition device carries a gimbal and a zoom lens to acquire close-up and panoramic images of the object for accurate analysis and subsequent processing. The computing module is connected to the first and second image acquisition devices and performs corresponding image data processing.
[0054] S102: Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, perform image transformation processing on the first digital image to obtain a second digital image without perspective distortion and with the same positioning feature points.
[0055] When the quasi-planar surface of an object is rectangular, such as when the object is a container, all its surfaces are rectangular. In this case, the four sides of the rectangle are detected according to the Hough transform, and the coordinates of the four corner vertices of the rectangle are obtained based on the four sides. The four corner points of the rectangle are then used as the positioning feature points of the quasi-planar surface.
[0056] At this point, the first digital image can be transformed based on the positioning feature points to obtain a second digital image without perspective distortion and with the same positioning feature points. The image transformation process includes perspective transformation, displacement, and scaling transformation. When there are multiple first digital images, there are also multiple second digital images. The positioning feature points of these multiple second digital images can be guaranteed to be the same.
[0057] S103: Extract graphic feature points from the second digital image, remove noise from it to obtain a set of outliers, and determine suspicious targets based on the set of outliers.
[0058] Graphical feature points are extracted from the second digital image using the SIFT feature method. Feature points with identical coordinates from multiple second digital images are excluded, thus eliminating duplicate feature points across multiple images. A codebook background model based on historical records is established, and regular interference points, such as those formed by regular lighting patterns, are excluded based on this model. Isolated feature points within a specified radius are excluded using clustering methods, performing three rounds of noise removal. A set of outliers is obtained based on the remaining graphic feature points after the exclusion process.
[0059] At this point, based on the set of outliers, a clustering algorithm is used to obtain the number of suspicious targets and the coordinates and size of each suspicious target on the quasi-planar surface of the object, generating a list of suspicious targets.
[0060] S104: Acquire a close-up digital image corresponding to the suspicious target, and perform foreign object detection on the suspicious target based on the close-up digital image.
[0061] The second image acquisition device can adjust the pan-tilt direction and lens focal length according to the target coordinates and size provided by the list of suspicious targets, and acquire close-up digital images of each suspicious target.
[0062] For close-up digital images, machine learning techniques can be used to detect and identify whether suspicious targets are foreign objects. These machine learning techniques can be trained using tools such as SVM, Bayesian inference, and convolutional neural networks. The detection objectives of machine learning techniques can include target classification, target detection, and image segmentation. For example, after detecting the target, the type of foreign object can be identified and classified through target classification, and image segmentation can be performed based on the classification results.
[0063] In the foreign object detection process, a preliminary screening is performed using digital images. Then, close-up images of suspicious targets are acquired for precise foreign object detection. Compared to detection methods such as LiDAR and structured light, this approach effectively improves detection accuracy. Furthermore, noise is eliminated during the preliminary screening process, reducing interference from surface damage, paint peeling, contamination, and surface texture patterns, thus lowering the number of false alarms.
[0064] In one embodiment, after foreign object detection is completed, the foreign object information in the first digital image can be determined based on the coordinates of the foreign object in the second digital image and the coordinate transformation relationship between the first and second digital images. The foreign object information includes the type, location, and size of the foreign object. This foreign object information is then sent to relevant devices (e.g., a server, a smart terminal, etc.) for further processing.
[0065] Specifically, during subsequent processing, the lens of the second imaging device can be adjusted to a wide-angle lens to capture a panoramic image of the object. Based on the foreign object information, the foreign object can be marked in the panoramic image, such as its location and size, so that the user can manually check it.
[0066] like Figure 2 As shown in the embodiments of this application, a quasi-planar foreign object detection device is also proposed, comprising:
[0067] At least one processor; and,
[0068] A memory communicatively connected to the at least one processor; wherein,
[0069] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:
[0070] Collect quasi-planar surface images of the object in different directions and generate multiple corresponding first digital images;
[0071] Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points.
[0072] After extracting graphic feature points from the second digital image and removing noise, a set of outliers is obtained, and suspicious targets are identified based on the set of outliers.
[0073] Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
[0074] This application also proposes a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:
[0075] Collect quasi-planar surface images of the object in different directions and generate multiple corresponding first digital images;
[0076] Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points.
[0077] After extracting graphic feature points from the second digital image and removing noise, a set of outliers is obtained, and suspicious targets are identified based on the set of outliers.
[0078] Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
[0079] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.
[0080] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.
[0081] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0082] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0083] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0084] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0086] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0087] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0088] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0089] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for detecting quasi-planar foreign objects, characterized in that, include: Collect quasi-planar surface images of the object in different directions and generate multiple corresponding first digital images; The object is a large object; Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points. The process involves extracting graphic feature points from the second digital image, removing noise, and obtaining a set of outliers. Suspicious targets are then identified based on this set of outliers. Specifically, this includes: extracting graphic feature points from the second digital image using the SIFT feature method; removing graphic feature points with identical coordinates from multiple second digital images; determining a codebook background model based on historical records and removing regular interference points based on this codebook background model; removing isolated feature points within a specified radius using a clustering method; and obtaining a set of outliers based on the remaining graphic feature points after the removal process. Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
2. The method according to claim 1, characterized in that, Based on a number of pre-defined positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points, specifically including: The quasi-plane surface of the object is determined to be rectangular. The four sides of the rectangle are detected according to the Hough transform, and the coordinates of the four corner vertices of the rectangle are obtained based on the four sides. The four corner points of the rectangle are used as the positioning feature points of the quasi-plane surface. Based on the positioning feature points, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points. The image transformation processing includes perspective transformation, displacement, and scaling transformation.
3. The method according to claim 1, characterized in that, Suspicious targets are identified based on the set of outliers, specifically including: Based on the set of outliers, a clustering algorithm is used to obtain the number of suspicious targets and the coordinates and size of each suspicious target on the quasi-planar surface of the object, thereby generating a list of suspicious targets.
4. The method according to claim 1, characterized in that, Foreign object detection of the suspicious target based on the close-up digital image specifically includes: For the close-up digital image, the machine learning technique is used to detect and identify whether the suspicious target is a foreign object. The machine learning technique uses at least one of SVM, Bayesian inference, and convolutional neural network. The detection target of the machine learning technique includes at least one of target classification, target detection, and image segmentation.
5. The method according to claim 1, characterized in that, The method further includes: Based on the coordinates of the foreign object in the second digital image and the coordinate transformation relationship between the first digital image and the second digital image, the foreign object information in the first digital image is determined, and the foreign object information includes the type, location, and size of the foreign object; The foreign object information is sent to the relevant equipment for further processing.
6. The method according to claim 5, characterized in that, The foreign object information is sent to relevant devices for further processing, specifically including: A panoramic image of the object is acquired, and the foreign object is marked in the panoramic image according to the foreign object information.
7. The method according to claim 1, characterized in that, Before acquiring quasi-planar surface images of the object in different directions, the method further includes: A foreign object detection device is defined, comprising a first image acquisition device, a second image acquisition device, and a computing module. Multiple first image acquisition devices are mounted in fixed positions above the side of the object's quasi-plane surface to acquire images of the quasi-plane surface. The second image acquisition device carries a gimbal and a zoom lens to acquire close-up and panoramic images of the object. The computing module is connected to the first and second image acquisition devices to perform corresponding image data processing.
8. A quasi-planar foreign object detection device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Images of the quasi-planar surface of an object are acquired from different directions, and multiple corresponding first digital images are generated; the object is a large object. Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points. The process involves extracting graphic feature points from the second digital image, removing noise, and obtaining a set of outliers. Suspicious targets are then identified based on this set of outliers. Specifically, this includes: extracting graphic feature points from the second digital image using the SIFT feature method; removing graphic feature points with identical coordinates from multiple second digital images; determining a codebook background model based on historical records and removing regular interference points based on this codebook background model; removing isolated feature points within a specified radius using a clustering method; and obtaining a set of outliers based on the remaining graphic feature points after the removal process. Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.
9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Images of the quasi-planar surface of an object are acquired from different directions, and multiple corresponding first digital images are generated; the object is a large object. Based on a number of pre-set positioning feature points on the quasi-planar surface of the object, the first digital image is subjected to image transformation processing to obtain a second digital image without perspective distortion and with the same positioning feature points. The process involves extracting graphic feature points from the second digital image, removing noise, and obtaining a set of outliers. Suspicious targets are then identified based on this set of outliers. Specifically, this includes: extracting graphic feature points from the second digital image using the SIFT feature method; removing graphic feature points with identical coordinates from multiple second digital images; determining a codebook background model based on historical records and removing regular interference points based on this codebook background model; removing isolated feature points within a specified radius using a clustering method; and obtaining a set of outliers based on the remaining graphic feature points after the removal process. Acquire close-up digital images of the suspected target and perform foreign object detection on the suspected target based on the close-up digital images.