A flexible object detection method, apparatus, device and medium

Abstract

The application discloses a flexible object detection method, device and equipment and medium, and relates to the technical field of intelligent manufacturing. The method comprehensively uses multiple traditional image processing methods, can detect flexible objects with small calculation amount and high calculation efficiency, can obtain instance segmentation results of an image plane and object surface space coordinate information of a focal plane, and can effectively and efficiently comprehensively detect projection planes and three-dimensional postures of various flexible objects.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing technology, and in particular to a flexible object detection method, device, equipment and medium based on machine vision. Background Art

[0002] Machine vision-based inspection methods are commonly used in industrial production. Machine vision generally refers to methods that use cameras to capture images and process them using image algorithms to measure, identify, and detect objects. Machine vision-based inspection methods are non-contact, automated, and highly versatile, making them widely used in industrial production.

[0003] However, most detection methods based on machine vision are aimed at rigid objects. Since flexible objects have non-fixed shapes and sizes, image algorithms often have poor robustness and low reliability, especially for the spatial relationship and posture detection of flexible objects. Currently, there is no effective, cost-effective and efficient detection method.

[0004] One of the existing technologies is a detection method based on traditional machine vision image algorithms.

[0005] For example, patent CN109766802: Flexible Object Recognition Method, Apparatus, Computer Device, and Storage Medium. It obtains template images of the object under test in different configurations, then collects multiple original object images, segments and combines these images, performs similarity matching with the template image, and determines the closest object shape based on the similarity with the template image.

[0006] Another example is patent CN114494169: Industrial Flexible Object Detection Method Based on Machine Vision. This method acquires an image of the object to be detected and uses algorithms such as edge detection and Hough transform to detect and extract the object's contour curve, thereby calculating the width of the flexible object.

[0007] Flexible object recognition methods rely on pre-acquired template images of product shapes, making them inapplicable to flexible objects with fluid shapes. Furthermore, template matching relies on product edges, making it difficult to generalize to more common flexible objects with fluid edges. Industrial flexible object detection is limited to measuring the width of flat flexible objects and cannot be extended to other aspects of flexible object posture.

[0008] Another existing technology is a posture estimation method based on deep learning neural network.

[0009] For example, patent CN116494244: A flexible object grasping method, apparatus, computer device, and storage medium. This method constructs a position evaluation strategy model and uses deep learning methods to train and optimize the model to obtain a neural network model that can predict the grasping point of flexible objects.

[0010] For example, patent CN115272741: an elongated flexible object detection method, terminal device and storage medium. The method constructs an instance segmentation neural network model, trains the model using human body, fishing net, safety belt and other image data sets, and obtains a model that can be used for flexible object detection and segmentation.

[0011] The deep learning method relies on a specific data set, and it takes a lot of time to build and train the data set. The stability of objects outside the data set is poor, and the output format of the neural network is fixed. The target detection network and the instance segmentation network are used to detect the bounding box and the foreground area of the object respectively, and cannot further detect the object pose. SUMMARY

[0012] In view of the above problems, the present application provides a flexible object detection method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. The method uses a coarse and fine granularity combined detection method designed based on a focus evaluation algorithm for the collected flexible object image, realizes comprehensive detection of object spatial position relationship, pose and the like, and has high adaptability and high computational efficiency.

[0013] The present application provides the following solutions:

[0014] A flexible object detection method, comprising:

[0015] Obtaining an original image of a target object collected by a camera, the target object including a flexible object with variable shape and size;

[0016] Pretreating the original image to obtain a to-be-processed image;

[0017] Processing the to-be-processed image using different granularity focus evaluation algorithms to obtain a coarse granularity focus spectrum and a fine granularity focus spectrum, respectively;

[0018] Respectively performing threshold segmentation on the coarse granularity focus spectrum and the fine granularity focus spectrum to obtain a coarse granularity foreground segmentation map and a fine granularity foreground segmentation map of the target object;

[0019] Performing erosion and / or dilation operation on the coarse granularity foreground segmentation map to obtain a region segmentation map;

[0020] Performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection pose detection of the target object;

[0021] Using calibration information of the camera to perform coordinate conversion on the fine granularity foreground segmentation map to obtain surface space coordinates of the target object; and using the surface space coordinates to perform three-dimensional pose and / or spatial relationship detection of the target object.

[0022] Preferably: multiple surface space coordinates corresponding to multiple fine-grained foreground segmentation maps are focal plane synthesized to obtain a complete spatial point cloud of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the spatial point cloud.

[0023] Preferably, the coarse-grained foreground segmentation map includes a complete object foreground area, and the fine-grained foreground segmentation map includes object image points in a current focal plane.

[0024] Preferably, the focus evaluation algorithm includes a coarse-grained focus evaluation algorithm and a fine-grained focus evaluation algorithm.

[0025] Preferably, the coarse-grained focus evaluation algorithm includes any one of a tenegrad function method and a Laplace gradient function method; the fine-grained focus evaluation algorithm includes any one of an energy gradient function method and a Brenner gradient method.

[0026] Preferably, the preprocessing includes region of interest extraction and contrast enhancement.

[0027] A flexible object detection device, used to perform the flexible object detection method described above, comprising:

[0028] An original image acquisition unit, configured to acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0029] An image preprocessing unit, configured to preprocess the original image to obtain an image to be processed;

[0030] A focus evaluation unit, configured to process the image to be processed using focus evaluation algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum;

[0031] a foreground segmentation unit, configured to perform threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0032] a region segmentation map acquisition unit, configured to perform erosion and / or dilation operations on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0033] an object instance segmentation result acquisition unit, configured to perform connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and perform projection posture detection of the target object using the object instance segmentation result;

[0034] A surface space coordinate acquisition unit is configured to perform coordinate conversion on the fine-grained foreground segmentation map by using calibration information of the camera to obtain surface space coordinates of the target object, and perform three-dimensional pose and / or spatial relationship detection of the target object by using the surface space coordinates.

[0035] A flexible object detection device, the device comprising a processor and a memory:

[0036] The memory is configured to store program code and transmit the program code to the processor.

[0037] The processor is configured to execute the above flexible object detection method according to instructions in the program code.

[0038] A computer-readable storage medium for storing program code for executing the above flexible object detection method.

[0039] According to the embodiments of the present application, the following technical effects are provided:

[0040] The embodiments of the present application provide a flexible object detection method, device, equipment and medium, which comprehensively uses various traditional image processing methods, can detect flexible objects with small calculation amount and high calculation efficiency, can obtain instance segmentation results of an image plane, can obtain object surface space coordinate information of a focal plane, and can effectively and efficiently detect projection planes and three-dimensional poses of various flexible objects.

[0041] Of course, implementing any product of the present application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0043] Figure 1 is a flowchart of a flexible object detection method provided by the embodiments of the present application;

[0044] Figure 2 is an implementation flowchart of a flexible object detection method provided by the embodiments of the present application;

[0045] Figure 3 is a schematic diagram of a flexible object detection device provided by the embodiments of the present application;

[0046] Figure 4 Schematic diagram of a flexible object detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0048] See also Figure 1 , is a flexible object detection method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0049] S101: Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0050] S102: Preprocessing the original image to obtain an image to be processed; the preprocessing includes region of interest extraction and contrast enhancement.

[0051] S103: The image to be processed is processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively; in specific implementation, the embodiment of the present application can provide that the coarse-grained foreground segmentation map includes the complete object foreground area, and the fine-grained foreground segmentation map includes the object image points of the current focal plane.

[0052] Furthermore, the focus evaluation algorithm includes a coarse-grained focus evaluation algorithm and a fine-grained focus evaluation algorithm. The coarse-grained focus evaluation algorithm includes any one of the tenegrad function method and the Laplace gradient function method; the fine-grained focus evaluation algorithm includes any one of the energy gradient function method and the Brenner gradient method.

[0053] S104: performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0054] S105: performing an erosion and / or dilation operation on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0055] S106: performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and performing projection posture detection of the target object using the object instance segmentation result;

[0056] S107: performing coordinate transformation on the fine-grained foreground segmentation map using the camera calibration information to obtain surface space coordinates of the target object; and performing three-dimensional posture and / or spatial relationship detection on the target object using the surface space coordinates.

[0057] If there are multiple original images, the results of other images are added to perform three-dimensional posture and / or spatial relationship detection. In specific implementation, the embodiment of the present application can provide a focal plane synthesis of multiple surface space coordinates corresponding to multiple fine-grained foreground segmentation maps to obtain a complete spatial point cloud of the target object; and use the spatial point cloud to perform three-dimensional posture and / or spatial relationship detection of the target object.

[0058] The flexible object detection method provided in the embodiments of the present application evaluates preprocessed object images. Focus assessment methods of different granularities are used on the object surface images. The resulting focus spectra are then threshold-segmented to produce an object foreground segmentation map. The results obtained by the coarse-grained focus assessment method are post-processed with corrosion and dilation to produce instance segmentation results. The results obtained by the fine-grained focus assessment method are then coordinate-transformed to produce a spatial point cloud of the current focal plane. The results of the coarse-grained and fine-grained methods are combined to detect the pose and spatial relationships of flexible objects.

[0059] This method uses the focus assessment algorithm's result image to segment the object image, effectively segmenting and detecting flexible objects. It also uses the focus assessment algorithm's result image to calculate the spatial position of the object's current focus plane, enabling rapid and accurate construction of the object's spatial coordinates. This method, which integrates focus assessment algorithms of varying granularity to detect flexible objects, demonstrates excellent adaptability and accuracy.

[0060] The implementation process of the method provided in the embodiment of the present application is described in detail below. Figure 2 As shown,

[0061] 1. First, the original image of the target object is collected and preprocessed to obtain the image to be processed. The image preprocessing methods include ROI (Region of Interest) extraction and contrast enhancement.

[0062] 2. Process the processed image using focus assessment algorithms of different granularities to obtain coarse-grained focus spectra and fine-grained focus spectra. Coarse-grained focus assessment methods include but are not limited to the Tenegrad function method and the Laplace gradient function method; fine-grained focus assessment methods include but are not limited to the energy gradient function method and the Brenner gradient method.

[0063] Where I represents the image matrix, f(I) represents the focus spectrum obtained by evaluation, and the formulas of several focus evaluation algorithms are as follows:

[0064] Tenegrad function method, gradient-based image clarity evaluation function.

[0065]

[0066] Laplace gradient function method, an image processing function that uses the Laplace operator for convolution calculation.

[0067]

[0068] Energy gradient function method, an image processing function that uses the difference between adjacent points to calculate the gradient value of a point.

[0069]

[0070] The Brenner gradient method, also known as the gradient filter method, evaluates the contribution of edge information to the image by calculating the square of the second-order gradient of adjacent points.

[0071]

[0072] 3. Threshold segmentation is performed on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain the foreground segmentation map of the object. The coarse-grained foreground segmentation map contains a relatively complete foreground area of ​​the object, while the fine-grained foreground segmentation map contains the object image points in the current focal plane.

[0073] 4. Perform operations such as erosion and expansion on the coarse-grained foreground segmentation map to obtain a region segmentation map, which is then subjected to connectivity analysis to obtain the object instance segmentation result.

[0074] 5. For the fine-grained foreground segmentation map, use the camera calibration information to perform coordinate transformation to obtain the surface space coordinates of the object. If there are multiple images, add other image results and perform focal plane synthesis on these results to obtain a complete object space point cloud.

[0075] 6. The object instance segmentation results obtained by the above coarse-grained detection are used for object projection posture detection, and the surface space coordinates or spatial point cloud obtained by fine-grained detection are used for three-dimensional posture, spatial relationship and other detections.

[0076] In summary, the flexible object detection method provided in this application uses a combination of multiple traditional image processing methods, and can detect flexible objects with very small computational complexity and extremely high computational efficiency. It can not only obtain instance segmentation results of the image plane, but also obtain the spatial coordinate information of the object surface of the focal plane. It can effectively and efficiently perform comprehensive detection of the projection surface and three-dimensional posture of various flexible objects.

[0077] See also Figure 3 , the embodiment of the present application can also provide a flexible object detection device, such as Figure 3 As shown, for performing the above-mentioned flexible object detection method, the device may include:

[0078] The original image acquisition unit 301 is used to acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0079] An image preprocessing unit 302 is configured to preprocess the original image to obtain an image to be processed;

[0080] A focus evaluation unit 303 is configured to process the image to be processed using focus evaluation algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively;

[0081] a foreground segmentation unit 304 for performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0082] A region segmentation map acquisition unit 305 is configured to perform an erosion and / or dilation operation on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0083] An object instance segmentation result acquisition unit 306 is configured to perform connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and use the object instance segmentation result to perform projection posture detection on the target object.

[0084] The surface space coordinate acquisition unit 307 is used to perform coordinate transformation on the fine-grained foreground segmentation map using the camera calibration information to obtain the surface space coordinates of the target object; and use the surface space coordinates to detect the three-dimensional posture and / or spatial relationship of the target object.

[0085] An embodiment of the present application may further provide a flexible object detection device, the device comprising a processor and a memory:

[0086] The memory is used to store program code and transmit the program code to the processor;

[0087] The processor is configured to execute the steps of the flexible object detection method according to the instructions in the program code.

[0088] like Figure 4 As shown, a flexible object detection device provided by an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other via the communication bus 13.

[0089] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.

[0090] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute operations in the embodiment of the flexible object detection method.

[0091] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0092] Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0093] Preprocessing the original image to obtain an image to be processed;

[0094] Processing the image to be processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively;

[0095] Performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0096] Performing erosion and / or dilation operations on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0097] Performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection posture detection of the target object;

[0098] The fine-grained foreground segmentation map is subjected to coordinate transformation using the camera calibration information to obtain the surface space coordinates of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the surface space coordinates.

[0099] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0100] In addition, the memory 11 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0101] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0102] Of course, it needs to be explained that Figure 4 The structure shown does not constitute a limitation on the flexible object detection device in the embodiment of the present application. In actual applications, the flexible object detection device may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0103] An embodiment of the present application may further provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned flexible object detection method.

[0104] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0105] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0106] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A flexible object detection method, characterized in that: include: Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size; Preprocessing the original image to obtain an image to be processed; Processing the image to be processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively; Performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object; Performing erosion and / or dilation operations on the coarse-grained foreground segmentation map to obtain a region segmentation map; Performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection posture detection of the target object; The fine-grained foreground segmentation map is subjected to coordinate transformation using the camera calibration information to obtain the surface space coordinates of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the surface space coordinates.

2. The flexible object detection method according to claim 1, characterized in that: Perform focal plane synthesis on the multiple surface space coordinates corresponding to the multiple fine-grained foreground segmentation maps to obtain a complete spatial point cloud of the target object; and use the spatial point cloud to detect the three-dimensional posture and / or spatial relationship of the target object.

3. The flexible object detection method according to claim 1, characterized in that: The coarse-grained foreground segmentation map includes a complete object foreground area, and the fine-grained foreground segmentation map includes object image points in a current focal plane.

4. The flexible object detection method according to claim 1, characterized in that: The focus evaluation algorithm includes a coarse-grained focus evaluation algorithm and a fine-grained focus evaluation algorithm.

5. The flexible object detection method according to claim 4, characterized in that: The coarse-grained focus evaluation algorithm includes any one of the tenegrad function method and the Laplace gradient function method; the fine-grained focus evaluation algorithm includes any one of the energy gradient function method and the Brenner gradient method.

6. The flexible object detection method according to claim 1, characterized in that: The preprocessing includes region of interest extraction and contrast enhancement.

7. A flexible object detection device, characterized in that: For executing the flexible object detection method according to any one of claims 1 to 6, the device comprises: An original image acquisition unit, configured to acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size; An image preprocessing unit, configured to preprocess the original image to obtain an image to be processed; A focus evaluation unit, configured to process the image to be processed using focus evaluation algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum; a foreground segmentation unit, configured to perform threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object; a region segmentation map acquisition unit, configured to perform erosion and / or dilation operations on the coarse-grained foreground segmentation map to obtain a region segmentation map; an object instance segmentation result acquisition unit, configured to perform connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and perform projection posture detection of the target object using the object instance segmentation result; A surface space coordinate acquisition unit is used to perform coordinate transformation on the fine-grained foreground segmentation map using the camera calibration information to obtain the surface space coordinates of the target object; and to perform three-dimensional posture and / or spatial relationship detection of the target object using the surface space coordinates.

8. A flexible object detection device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the flexible object detection method according to any one of claims 1 to 6 according to instructions in the program code.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program codes, and the program codes are used to execute the flexible object detection method according to any one of claims 1 to 6.

Images