Flexible object detection method, device, equipment and medium

Abstract

The invention discloses a flexible object detection method, device and equipment and a medium, and relates to the technical field of intelligent manufacturing, the method comprehensively uses multiple traditional image processing methods, can detect a flexible object with a small calculation amount and extremely high calculation efficiency, can obtain an instance segmentation result of an image plane, and also can detect the flexible object with a small calculation amount and extremely high calculation efficiency. And object surface space coordinate information of a focal plane can be obtained, and comprehensive detection of projection planes and three-dimensional postures of various flexible objects can be effectively and efficiently carried out.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing technology, and in particular to a flexible object detection method, device, equipment and medium based on machine vision. Background Art

[0002] Machine vision-based detection methods are commonly used in industrial production. Machine vision generally refers to the method of using a camera to collect images and image algorithms to process images to measure, identify, and detect objects. Machine vision-based detection methods have the characteristics of non-contact, automation, and high versatility, and have been widely used in industrial production.

[0003] However, most detection methods based on machine vision are aimed at rigid objects. Since flexible objects have irregular shapes and sizes, image algorithms often have poor robustness and low reliability. In particular, there is currently no effective, cost-effective and efficient detection method for the spatial relationship and posture detection of flexible objects.

[0004] One of the existing technologies is a detection method based on traditional machine vision image algorithm.

[0005] For example, patent CN109766802: Flexible object recognition method, device, computer equipment and storage medium. It obtains template images of different forms of the object to be tested, then collects multiple original object images, and divides and combines these images, performs similarity matching with the template image, and determines the closest object shape based on the similarity with the template image.

[0006] Another example is patent CN114494169: Industrial flexible object detection method based on machine vision. It acquires the image of the object to be detected, uses edge detection, Hough transform and other algorithms to detect and extract the contour curve of the object, and thus calculates the width of the flexible object.

[0007] Flexible object recognition methods rely on pre-acquired product form template images, which are not universal for flexible objects with unstable shapes. In addition, the template matching relies on product edges and cannot be extended to more common flexible objects with unstable edges. Industrial flexible object detection is only applicable to the width detection of planar flexible objects, and cannot be extended to the posture detection of flexible objects.

[0008] Another prior art is a posture estimation method based on a deep learning neural network.

[0009] For example, patent CN116494244: A method, device, computer equipment and storage medium for grasping a flexible object. The method constructs a position evaluation strategy model, trains and optimizes the model using a deep learning method, and obtains a neural network model that can predict the grasping point of a flexible object.

[0010] Another example is patent CN115272741: A method, terminal device and storage medium for detecting slender and flexible objects. The method constructs an instance segmentation neural network model, trains the model using image data sets such as human bodies, fishing nets, and seat belts, and obtains a model that can be used for flexible object detection and segmentation.

[0011] Deep learning methods rely on specific data sets. It takes a lot of time to build and train data sets. They are not stable for objects outside the data sets, and the output format of the neural network is fixed. The target detection network and instance segmentation network are used to detect the bounding box and foreground area of ​​the object respectively, and cannot further detect the object's posture. Summary of the invention

[0012] In view of the above problems, the present invention provides a flexible object detection method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems. The method uses a coarse and fine granularity combination detection method based on a focus evaluation algorithm to collect flexible object images, so as to achieve comprehensive detection of the spatial position relationship, posture, etc. of the object, and has high adaptability and high computational efficiency.

[0013] The present invention provides the following scheme:

[0014] A flexible object detection method, comprising:

[0015] Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0016] Preprocessing the original image to obtain an image to be processed;

[0017] Processing the image to be processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively;

[0018] Performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0019] Performing corrosion and / or expansion operations on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0020] Performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection posture detection on the target object;

[0021] The fine-grained foreground segmentation map is subjected to coordinate transformation using the calibration information of the camera to obtain the surface space coordinates of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the surface space coordinates.

[0022] Preferably: multiple surface space coordinates corresponding to multiple fine-grained foreground segmentation maps are focal plane synthesized to obtain a complete space point cloud of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the space point cloud.

[0023] Preferably, the coarse-grained foreground segmentation map includes a complete object foreground area, and the fine-grained foreground segmentation map includes object image points in a current focal plane.

[0024] Preferably, the focus evaluation algorithm includes a coarse-grained focus evaluation algorithm and a fine-grained focus evaluation algorithm.

[0025] Preferably: the coarse-grained focus evaluation algorithm includes any one of a tenegrad function method and a Laplace gradient function method; the fine-grained focus evaluation algorithm includes any one of an energy gradient function method and a Brenner gradient method.

[0026] Preferably, the preprocessing includes region of interest extraction and contrast enhancement.

[0027] A flexible object detection device, used to perform the flexible object detection method, the device comprising:

[0028] An original image acquisition unit, used to acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0029] An image preprocessing unit, used for preprocessing the original image to obtain an image to be processed;

[0030] A focus evaluation unit, used to process the to-be-processed image using focus evaluation algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively;

[0031] a foreground segmentation unit, configured to perform threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0032] A region segmentation map acquisition unit, used for performing erosion and / or dilation operations on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0033] An object instance segmentation result acquisition unit, configured to perform connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and use the object instance segmentation result to perform projection posture detection on the target object;

[0034] A surface space coordinate acquisition unit is used to perform coordinate transformation on the fine-grained foreground segmentation map using the calibration information of the camera to obtain the surface space coordinates of the target object; and to perform three-dimensional posture and / or spatial relationship detection of the target object using the surface space coordinates.

[0035] A flexible object detection device, the device comprising a processor and a memory:

[0036] The memory is used to store program code and transmit the program code to the processor;

[0037] The processor is used to execute the flexible object detection method according to the instructions in the program code.

[0038] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the flexible object detection method.

[0039] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The embodiments of the present application provide a flexible object detection method, device, equipment and medium. The method uses a combination of multiple traditional image processing methods and can detect flexible objects with very small computational complexity and extremely high computational efficiency. It can not only obtain instance segmentation results of the image plane, but also obtain the spatial coordinate information of the object surface of the focal plane, and can effectively and efficiently perform comprehensive detection of the projection surface and three-dimensional posture of various flexible objects.

[0041] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 is a flow chart of a flexible object detection method provided by an embodiment of the present invention;

[0044] Figure 2 It is a framework diagram of an implementation process of a flexible object detection method provided by an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of a flexible object detection device provided by an embodiment of the present invention;

[0046] Figure 4 It is a schematic diagram of a flexible object detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0048] See also Figure 1 , is a flexible object detection method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0049] S101: Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0050] S102: Preprocessing the original image to obtain an image to be processed; the preprocessing includes extracting a region of interest and enhancing contrast.

[0051] S103: The image to be processed is processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively; in specific implementation, the embodiment of the present application can provide that the coarse-grained foreground segmentation map includes a complete object foreground area, and the fine-grained foreground segmentation map includes object image points in the current focal plane.

[0052] Furthermore, the focus evaluation algorithm includes a coarse-grained focus evaluation algorithm and a fine-grained focus evaluation algorithm. The coarse-grained focus evaluation algorithm includes any one of the tenegrad function method and the Laplace gradient function method; the fine-grained focus evaluation algorithm includes any one of the energy gradient function method and the Brenner gradient method.

[0053] S104: performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object respectively;

[0054] S105: performing corrosion and / or expansion operations on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0055] S106: performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection posture detection on the target object;

[0056] S107: performing coordinate transformation on the fine-grained foreground segmentation map using the calibration information of the camera to obtain the surface space coordinates of the target object; and performing three-dimensional posture and / or spatial relationship detection of the target object using the surface space coordinates.

[0057] If there are multiple original images, the results of other images are added to perform three-dimensional posture and / or spatial relationship detection. In specific implementation, the embodiment of the present application can provide focal plane synthesis of multiple surface space coordinates corresponding to multiple fine-grained foreground segmentation maps to obtain a complete spatial point cloud of the target object; and use the spatial point cloud to perform three-dimensional posture and / or spatial relationship detection of the target object.

[0058] The flexible object detection method provided in the embodiment of the present application evaluates the object image after preprocessing, and uses different focus evaluation methods with different granularities for the surface image of the object, and performs threshold segmentation on the obtained focus spectrum to obtain the object foreground segmentation map, wherein the result obtained by the coarse-grained focus evaluation method is post-processed by corrosion and expansion to obtain the instance segmentation result; the result obtained by the fine-grained focus evaluation method is coordinate-converted to obtain the spatial point cloud of the current focal plane. The results of the coarse-grained and fine-grained methods are combined to detect the object posture and spatial relationship of the flexible object.

[0059] This method uses the focus evaluation algorithm result image to segment the object image, which can effectively segment and detect flexible objects. The method uses the focus evaluation algorithm result image to calculate the spatial point position of the object's current focus plane. This method can quickly and accurately construct the object's spatial coordinates. The method uses focus evaluation algorithms of different granularities to detect flexible objects. This method can detect flexible objects well and has high adaptability and accuracy.

[0060] The implementation process of the method provided in the embodiment of the present application is described in detail below. Figure 2 As shown,

[0061] 1. First, the original image of the target object is collected and preprocessed to obtain the image to be processed. The image preprocessing methods include ROI (Region of Interest) extraction, contrast enhancement, etc.

[0062] 2. Use different granularity focus assessment algorithms to process the processed image to obtain coarse-grained focus spectra and fine-grained focus spectra. The coarse-grained focus assessment method includes but is not limited to the tenegrad function method and the Laplace gradient function method; the fine-grained focus assessment method includes but is not limited to the energy gradient function method and the Brenner gradient method.

[0063] Where I represents the image matrix, f(I) represents the focus spectrum obtained by evaluation, and the formulas of several focus evaluation algorithms are as follows:

[0064] Tenegrad function method, gradient-based image clarity evaluation function.

[0065]

[0066] Laplace gradient function method, an image processing function that uses the Laplace operator for convolution calculation.

[0067]

[0068] Energy gradient function method, an image processing function that uses the difference between adjacent points to calculate the gradient value of a point.

[0069]

[0070] The Brenner gradient method, also known as the gradient filter method, evaluates the contribution of edge information to the image by calculating the square of the second-order gradient of adjacent points.

[0071]

[0072] 3. Perform threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain the foreground segmentation map of the object, and obtain the coarse-grained foreground segmentation map and the fine-grained foreground segmentation map. Among them, the coarse-grained foreground segmentation map contains a relatively complete object foreground area, while the fine-grained foreground segmentation map contains the object image points of the current focal plane.

[0073] 4. Perform operations such as erosion and expansion on the coarse-grained foreground segmentation map to obtain a region segmentation map, which is then subjected to connectivity analysis to obtain the object instance segmentation result.

[0074] 5. For the fine-grained foreground segmentation map, use the camera calibration information to perform coordinate transformation to obtain the surface space coordinates of the object. If there are multiple images, add other image results and perform focal plane synthesis on these results to obtain a complete object space point cloud.

[0075] 6. The object instance segmentation results obtained by the above coarse-grained detection are used for object projection posture detection, and the surface space coordinates or space point cloud obtained by fine-grained detection are used for three-dimensional posture, spatial relationship and other detections.

[0076] In summary, the flexible object detection method provided by the present application uses a combination of various traditional image processing methods, and can detect flexible objects with very small computational complexity and extremely high computational efficiency. It can not only obtain instance segmentation results of the image plane, but also obtain the spatial coordinate information of the object surface of the focal plane. It can effectively and efficiently perform comprehensive detection of the projection surface and three-dimensional posture of various flexible objects.

[0077] See also Figure 3 , the embodiment of the present application can also provide a flexible object detection device, such as Figure 3 As shown, for executing the above-mentioned flexible object detection method, the device may include:

[0078] The original image acquisition unit 301 is used to acquire the original image of the target object captured by the camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0079] An image preprocessing unit 302 is used to preprocess the original image to obtain an image to be processed;

[0080] A focus evaluation unit 303 is used to process the image to be processed using focus evaluation algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively;

[0081] A foreground segmentation unit 304 is used to perform threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0082] A region segmentation map acquisition unit 305 is used to perform an erosion and / or dilation operation on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0083] An object instance segmentation result acquisition unit 306 is used to perform connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and use the object instance segmentation result to perform projection posture detection of the target object;

[0084] The surface space coordinate acquisition unit 307 is used to perform coordinate transformation on the fine-grained foreground segmentation map using the camera calibration information to obtain the surface space coordinates of the target object; and use the surface space coordinates to perform three-dimensional posture and / or spatial relationship detection of the target object.

[0085] The embodiment of the present application may also provide a flexible object detection device, the device comprising a processor and a memory:

[0086] The memory is used to store program code and transmit the program code to the processor;

[0087] The processor is used to execute the steps of the flexible object detection method according to the instructions in the program code.

[0088] like Figure 4 As shown, a flexible object detection device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0089] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0090] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the flexible object detection method.

[0091] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0092] Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size;

[0093] Preprocessing the original image to obtain an image to be processed;

[0094] Processing the image to be processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively;

[0095] Performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object;

[0096] Performing corrosion and / or expansion operations on the coarse-grained foreground segmentation map to obtain a region segmentation map;

[0097] Performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection posture detection on the target object;

[0098] The fine-grained foreground segmentation map is subjected to coordinate transformation using the calibration information of the camera to obtain the surface space coordinates of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the surface space coordinates.

[0099] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0100] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0101] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0102] Of course, it should be noted that Figure 4 The structure shown does not constitute a limitation on the flexible object detection device in the embodiment of the present application. In actual applications, the flexible object detection device may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0103] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the steps of the above-mentioned flexible object detection method.

[0104] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0105] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A flexible object detection method, characterized in that: include: Acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size; Preprocessing the original image to obtain an image to be processed; Processing the image to be processed using focus assessment algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively; Performing threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object; Performing corrosion and / or expansion operations on the coarse-grained foreground segmentation map to obtain a region segmentation map; Performing connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and using the object instance segmentation result to perform projection posture detection on the target object; The fine-grained foreground segmentation map is subjected to coordinate transformation using the calibration information of the camera to obtain the surface space coordinates of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the surface space coordinates.

2. The flexible object detection method according to claim 1, characterized in that: The plurality of surface space coordinates corresponding to the plurality of fine-grained foreground segmentation maps are focal plane synthesized to obtain a complete space point cloud of the target object; and the three-dimensional posture and / or spatial relationship detection of the target object is performed using the space point cloud.

3. The flexible object detection method according to claim 1, characterized in that: The coarse-grained foreground segmentation map includes a complete object foreground area, and the fine-grained foreground segmentation map includes object image points in a current focal plane.

4. The flexible object detection method according to claim 1, characterized in that: The focus evaluation algorithm includes a coarse-grained focus evaluation algorithm and a fine-grained focus evaluation algorithm.

5. The flexible object detection method according to claim 4, characterized in that: The coarse-grained focus evaluation algorithm includes any one of the tenegrad function method and the Laplace gradient function method; the fine-grained focus evaluation algorithm includes any one of the energy gradient function method and the Brenner gradient method.

6. The flexible object detection method according to claim 1, characterized in that: The preprocessing includes region of interest extraction and contrast enhancement.

7. A flexible object detection device, characterized in that: Used to perform the flexible object detection method according to any one of claims 1 to 6, the device comprises: An original image acquisition unit, used to acquire an original image of a target object captured by a camera, wherein the target object includes a flexible object with a non-fixed shape and size; An image preprocessing unit, used for preprocessing the original image to obtain an image to be processed; A focus evaluation unit, used to process the to-be-processed image using focus evaluation algorithms of different granularities to obtain a coarse-grained focus spectrum and a fine-grained focus spectrum respectively; a foreground segmentation unit, configured to perform threshold segmentation on the coarse-grained focus spectrum and the fine-grained focus spectrum to obtain a coarse-grained foreground segmentation map and a fine-grained foreground segmentation map of the target object; A region segmentation map acquisition unit, used for performing erosion and / or dilation operations on the coarse-grained foreground segmentation map to obtain a region segmentation map; An object instance segmentation result acquisition unit, configured to perform connectivity analysis on the region segmentation map to obtain an object instance segmentation result, and use the object instance segmentation result to perform projection posture detection on the target object; A surface space coordinate acquisition unit is used to perform coordinate transformation on the fine-grained foreground segmentation map using the calibration information of the camera to obtain the surface space coordinates of the target object; and to perform three-dimensional posture and / or spatial relationship detection of the target object using the surface space coordinates.

8. A flexible object detection device, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the flexible object detection method according to any one of claims 1 to 6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the flexible object detection method according to any one of claims 1 to 6.

Images