Target segmentation method, electronic device and storage medium
By converting the depth image into a point cloud and determining the minimum cut set, the color image is segmented, which solves the accuracy problems of traditional methods in complex backgrounds and the high cost and complexity problems of deep learning methods, and achieves efficient and accurate target segmentation.
Patent Information
- Application Number
- CN202411676554.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-04-08
AI Technical Summary
Traditional goal segmentation methods are difficult to accurately distinguish target goods from backgrounds in complex contexts. Deep learning methods require a large amount of annotated data and multiple model training, which increases cost and complexity.
By acquiring color images and depth images, converting the depth images into point clouds, filtering out the target point clouds, determining the minimum cut set, and segmenting the color images.
It improves the accuracy and efficiency of target segmentation, reduces the cost of data acquisition and model training, simplifies model deployment, and adapts to industrial automation production needs in complex scenarios.
Smart Images
Figure CN119784768A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and more specifically, to a target segmentation method, electronic equipment and storage medium. Background Art
[0002] In the industrial production process, real-time monitoring, target segmentation and classification of goods on the conveyor belt are key links to improve production efficiency and quality control. Target segmentation refers to the process of accurately separating the target goods from the complex background, which is crucial for subsequent automated processing and analysis.
[0003] However, traditional target segmentation methods often rely on fixed image processing algorithms, such as threshold segmentation and edge detection. These algorithms are difficult to accurately distinguish target goods from the background when facing the complex background of the conveyor belt environment (such as lighting changes, the diversity of product shapes and colors, and the unevenness of the background, etc.), resulting in poor segmentation results. Target segmentation methods based on deep learning (such as convolutional neural networks (CNN)) can be used to deal with target segmentation problems under complex backgrounds, but a large amount of labeled data needs to be collected for model training, which is not only time-consuming and labor-intensive, but also costly. In addition, deep learning models usually need to be retrained for different products to adapt to different segmentation tasks, which increases the complexity and time cost of model deployment. Summary of the invention
[0004] One purpose of an embodiment of the present invention is to provide a new technical solution for target segmentation to address the limitations of traditional target segmentation methods and existing deep learning target segmentation methods, improve the accuracy and efficiency of target segmentation, and achieve more efficient and accurate target segmentation to meet the needs of industrial automation production.
[0005] According to a first aspect of the present invention, there is provided a method for object segmentation, comprising:
[0006] Acquire a first image captured by a camera; wherein the first image includes a pixel-aligned color image and a depth image, and the color image includes target objects and non-target objects;
[0007] According to the camera intrinsic parameters of the camera, the depth image is converted into a point cloud in a camera coordinate system;
[0008] The point cloud is screened and processed according to an estimated distance range for the target object to obtain a target point cloud; wherein the estimated distance range is an estimated distance range between the target object and the camera;
[0009] Determine a minimum cut set according to the target point cloud;
[0010] Perform image segmentation processing on the color image according to the minimum cut set to obtain and output a second image.
[0011] Optionally, converting the depth image into a point cloud in a camera coordinate system according to the camera intrinsic parameters of the camera includes:
[0012] For each pixel point in the depth image, the pixel coordinates of the pixel point are converted into the camera coordinates of the pixel point according to the camera intrinsic parameters to obtain the point cloud.
[0013] Optionally, when the first image is a monitoring image of a set monitoring area for a conveyor belt, the estimated distance range is determined according to the installation height of the camera, the laying height of the conveyor belt, and the noise of the shooting scene.
[0014] Optionally, determining a minimum cut set according to the target point cloud includes:
[0015] Determine a source point and a sink point according to the target point cloud; wherein the source point corresponds to the target object, and the sink point corresponds to the non-target object;
[0016] Constructing a point cloud map according to the target point cloud, the source point and the sink point;
[0017] Assigning a corresponding weight to each edge in the point cloud graph;
[0018] According to the weights, a minimum cut set is determined in the point cloud image by using a minimum cut algorithm.
[0019] Optionally, the point cloud graph includes a first type of edge, a second type of edge, and a third type of edge, the first type of edge is a line between any two adjacent points in the target point cloud, the second type of edge is a line between the source point and a point in the target point cloud, and the third type of edge is a line between the sink point and a point in the target point cloud, and the assigning of a corresponding weight to each edge in the point cloud graph includes:
[0020] Assigning a first weight to the first type of edge in the point cloud graph;
[0021] assigning a second weight to the second type of edge in the point cloud graph;
[0022] A third weight is assigned to the third type of edge in the point cloud graph.
[0023] Optionally, the first weight is calculated based on the Euclidean distance and scale parameter of any two adjacent points in the target point cloud, the second weight is a preset weight, and the third weight is calculated based on the Euclidean distance between the point in the target point cloud and the source point and a preset radius of the target object.
[0024] Optionally, the source point is the centroid of the target point cloud, and the sink point is the point farthest from the centroid.
[0025] Optionally, before determining the minimum cut set according to the target point cloud, the method further includes:
[0026] The target point cloud is downsampled to obtain a downsampled target point cloud as a new target point cloud.
[0027] According to a second aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store executable instructions; and the processor is used to operate according to the control of the instructions to execute the method as described in the first aspect.
[0028] According to a third aspect of the present disclosure, a readable storage medium is provided, on which computer executable instructions are stored. When the executable instructions are executed by a processor, the target segmentation method as described in the first aspect is implemented.
[0029] One of the beneficial effects of the present invention is that by determining the minimum cut set based on the depth image, the color image is segmented to obtain a second image. Compared with the traditional target segmentation method, it can cope with target segmentation processing in complex scenes. Compared with the existing deep learning target segmentation method, it saves a lot of data collection and model training, improves the efficiency of target segmentation, and achieves more efficient and accurate target segmentation to meet the needs of industrial automation production. In addition, by screening the point cloud according to the estimated distance range of the target object to obtain the target point cloud, abnormal points in the point cloud can be removed, simplifying the complexity of subsequent processing and further improving the efficiency of target segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0031] Figure 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention;
[0032] Figure 2 is a schematic flow chart of a target segmentation method according to an embodiment of the present invention;
[0033] Figure 3 is a schematic diagram of a shooting scene according to an embodiment of the present invention;
[0034] Figure 4 is a schematic diagram of a point cloud image according to an embodiment of the present invention;
[0035] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.
[0037] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0038] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0039] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0041] <Hardware Configuration>
[0042] Figure 1 is a block diagram of a hardware configuration of an electronic device 100 according to an embodiment of the present invention.
[0043] like Figure 1 As shown, the electronic device 100 may be a tablet computer, a desktop computer, a small computer, a workstation, a laptop computer, a mobile phone, etc., which is not limited here.
[0044] In this embodiment, refer to Figure 1 As shown, the electronic device 100 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like.
[0045] The processor 1100 may be a mobile version processor. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, a USB interface, a headphone interface, etc. The communication device 1400 may, for example, be capable of wired or wireless communication. The communication device 1400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1500 may be, for example, a liquid crystal display, a touch display, etc., and the display device 1500 is used to display a second image. The input device 1600 may include, for example, a touch screen, a keyboard, etc. The user may input voice information through a microphone and output voice information through a speaker 1700.
[0046] In this embodiment, the memory 1200 of the electronic device is used to store instructions, and the instructions are used to control the processor 1100 to operate to at least perform the target segmentation method implemented by the electronic device according to any embodiment of the present invention. The technician can design the instructions according to the scheme disclosed in the present invention. How the instructions control the processor to operate is well known in the art, so it will not be described in detail here.
[0047] Despite Figure 1 , multiple devices of the electronic device 100 are shown; however, the present invention may only involve some of the devices, for example, the electronic device 100 only involves the memory 1200 and the processor 1100 .
[0048] <Method Example>
[0049] Figure 2 is a flowchart of a target segmentation method according to an embodiment of the present invention, which can be implemented by the electronic device 100 .
[0050] according to Figure 2 As shown, the target segmentation method of this embodiment may include the following steps S2100 to S2500:
[0051] Step S2100, obtaining a first image captured by a camera.
[0052] In this embodiment, the camera may be a 3D camera or a time-of-flight (ToF) camera, etc., which is not limited here.
[0053] The camera captures a first image in a shooting scene, wherein the shooting scene may be a scene of goods being transported by a conveyor belt, or other shooting scenes, which are not limited here.
[0054] The first image includes a color image and a depth image. The depth image contains the distance information from each pixel in the shooting scene to the camera. In addition, the distance is often represented by grayscale values in the depth image, where different grayscale levels correspond to different distances. The color image contains the color information of each pixel in the shooting scene.
[0055] The pixels of the color image and the depth image in the first image are aligned. That is, for each pixel in the color image, a corresponding pixel can be found in the depth image, and for each pixel in the depth image, a corresponding pixel can be found in the color image.
[0056] Since the shooting scene includes not only the target object, ie, the object that the user wants to shoot, but also the objects that the user does not need to shoot, ie, the non-target objects, the captured color image also includes the target object and the non-target objects.
[0057] For example, in a scene corresponding to a conveyor belt transporting goods, the target object is the goods, and the non-target objects are the conveyor belt and other objects other than the goods.
[0058] Step S2200: converting the depth image into a point cloud in a camera coordinate system according to the camera intrinsic parameters of the camera.
[0059] In this embodiment, the camera intrinsic parameters may be parameters describing the internal characteristics of the camera, including focal length, principal point position (image center), pixel size, etc. Using the camera intrinsic parameters, a depth map in a two-dimensional space may be converted into a point cloud in a three-dimensional space.
[0060] In some embodiments, in step S2200, converting the depth image into a point cloud in a camera coordinate system according to the camera intrinsic parameters of the camera includes:
[0061] For each pixel point in the depth image, the pixel coordinates of the pixel point are converted into the camera coordinates of the pixel point according to the camera intrinsic parameters to obtain the point cloud.
[0062] In this embodiment, the pixel coordinates (i.e., two-dimensional image coordinates) (u, v) of each pixel point in the depth map can be converted into the three-dimensional coordinates (X, Y, Z) of the pixel point in the camera coordinate system, i.e., the camera coordinates of the pixel point, according to the following formula (1), thereby obtaining a point cloud.
[0063]
[0064] Among them, f x ,f y is the focal length of the camera in the x and y directions, c x ,c y are the principal point coordinates of the camera, which are all camera internal parameters. d is the depth value of the depth map.
[0065] According to an embodiment of the present application, by converting the pixel coordinates of the pixel points in the depth map into the camera coordinates of the pixel points to construct a point cloud, it is convenient to perform preliminary screening of the point cloud based on the distance between the points in the point cloud and the camera, so as to simplify the processing amount of subsequent image segmentation and improve the efficiency of image segmentation.
[0066] Step S2300: screening the point cloud according to the estimated distance range of the target object to obtain a target point cloud.
[0067] In this embodiment, the estimated distance range is the estimated distance range between the target object and the camera.
[0068] The estimated distance range can be determined based on the distance between the camera and non-target objects in the captured scene.
[0069] In one example, if Figure 3 As shown, when the camera is mounted above the conveyor belt and at a certain height from the ground, the estimated distance range can be determined according to the height of the conveyor belt relative to the camera.
[0070] In another example, when the camera is mounted on the left side of the conveyor belt and is at a first distance from the conveyor belt, the estimated distance range may be determined based on the distance of the conveyor belt relative to the camera in the horizontal direction.
[0071] In some embodiments, when the first image is a monitoring image of a set monitoring area for a conveyor belt, the estimated distance range is determined according to the installation height of the camera, the laying height of the conveyor belt, and the shooting scene noise.
[0072] In this embodiment, the camera is mounted on Figure 3 The above of the conveyor belt shown in the figure is used to monitor the set monitoring area on the conveyor belt. The set monitoring area is Figure 3 The first image captured is an image corresponding to the set monitoring area, and the first image at least includes the conveyor belt (non-target object) and the goods on the conveyor belt (ie, target object).
[0073] In this scenario, the estimated distance range can be calculated using formula (2):
[0074] 0≤Cloud z≤HhZ thresh (2)
[0075] Among them, H is the camera installation height, h is the height of the conveyor belt surface from the ground, that is, the laying height of the conveyor belt, and Z thresh is the noise threshold in this shooting scene, Cloud z is the z coordinate of the point cloud, that is, the height of the pixel relative to the camera.
[0076] The noise threshold in a shooting scene may be an empirical value set in the shooting scene, or may be a value determined after multiple adjustments in the current shooting scene, which is not limited here.
[0077] The noise in the shooting scene may cause the first image to be offset from the real scene, so increasing the noise threshold in the shooting scene can compensate for the offset to improve the accuracy of target point cloud screening.
[0078] For example, the camera is installed at a height of H = 2000 mm, the height of the conveyor belt surface from the ground is h = 850 mm, and Z thresh =50mm., then the estimated distance range is: 0≤Cloud z ≤1100.
[0079] After determining the estimated distance range, the point cloud is filtered by the estimated distance range to remove points in the point cloud that are not within the range, and obtain the target point cloud, wherein the target point cloud is the point cloud whose distance from the camera is within the estimated distance range.
[0080] The corresponding estimated distance range is: 0≤Cloud z In the example of ≤1100, the target point cloud is the point cloud whose height is within the estimated height range from the camera.
[0081] In some embodiments, for the convenience of statistics, the area constructed by the target point cloud is recorded as the ROI area. The number of points in the ROI area is also determined.
[0082] Step S2400: determining a minimum cut set according to the target point cloud.
[0083] In this embodiment, the source points and sink points are first determined according to the target point cloud, and then the minimum cut set is determined according to the target point cloud, the source points and the sink points. The minimum cut set may refer to dividing the vertex set into two non-intersecting subsets so that the total capacity (or weight) of the edges connecting the two subsets is minimized. The vertex set includes the vertices, sink points and source points in the target point cloud, the source points correspond to the target objects, and the sink points correspond to the non-target objects.
[0084] Those skilled in the art should understand that the source points and the sink points may be determined according to the distribution of the target point cloud, including a variety of settings, and the specific setting method of the source points and the sink points is not limited here.
[0085] In some embodiments, determining the minimum cut set according to the target point cloud in step S2400 includes: steps S2400.1 to S2400.4.
[0086] Step S2400.1, determining source points and sink points according to the target point cloud.
[0087] In the example of conveyor belts transporting goods, the goods on the conveyor belt are generally located in the center of the conveyor belt, and even if they deviate occasionally, they will not deviate too much. Therefore, the center point of the target point cloud can be set as the center of the target object, that is, the centroid of the target point cloud is used as the source point, and the point in the target point cloud farthest from the centroid is used as the sink point.
[0088] Step S2400.2, constructing a point cloud map according to the target point cloud, the source points and the sink points.
[0089] In this embodiment, the target point cloud has naturally separated points, and each point in the target point cloud can be regarded as a vertex. Then, the source point S and sink point T determined in step S2400.1 are added, and any two adjacent points in the target point cloud are connected. The source point S is connected to the point in the target point cloud, and the sink point T is connected to the point in the target point cloud, thus forming a point cloud map.
[0090] For example, the constructed point cloud is as follows: Figure 4 As shown, it includes the line connecting any two adjacent points in the target point cloud, the line connecting the source point S and the point in the target point cloud, and the line connecting the sink point T and the point in the target point cloud.
[0091] Step S2400.3, assigning a corresponding weight to each edge in the point cloud graph.
[0092] In this embodiment, the line between two points in the point cloud graph is an edge. In order to find the minimum cut set, it is necessary to first assign a corresponding weight to each edge to represent the connection strength or similarity between the points connected by the edge.
[0093] The higher the weight corresponding to an edge, the more similar the two points connected by the edge are, and the less likely they are to be cut off.
[0094] When finding the minimum cut set of the point cloud, the weight of the edge is used to determine which edges in the point cloud should be prioritized in the segmentation process. The goal of the minimum cut set is to find a set of edges in the point cloud whose corresponding total weight is the smallest and can split the point cloud into two subsets.
[0095] Since point cloud data usually contains a large number of points and edges, the minimum cut set can be found more efficiently by assigning a weight to each edge.
[0096] In some embodiments, Figure 4 As shown, the point cloud graph includes the first type of edge, the second type of edge and the third type of edge. The first type of edge is the line between any two adjacent points in the target point cloud, that is, Figure 4 The solid line in , denoted as n-link. The second type of edge is the line between the source point S and the point in the target point cloud, denoted as t-link. The third type of edge is the line between the sink point T and the point in the target point cloud, also denoted as t-link. The second and third types of edges are Figure 4 The dotted line in .
[0097] The weights corresponding to the first-type edges, the second-type edges, and the third-type edges may be completely different or partially the same, which is not limited here.
[0098] In some embodiments, step S2400.3 assigns a corresponding weight to each edge in the point cloud graph, including: steps SA1 to SA3.
[0099] Step SA1: assigning a first weight to the first type of edge in the point cloud graph.
[0100] In some embodiments, the first weight is calculated based on the Euclidean distance and the scale parameter between any two adjacent points in the target point cloud.
[0101] Exemplarily, the calculation formula of the first weight is as follows (3):
[0102]
[0103] Among them, dist is the Euclidean distance between the two points connected by the first type of edge. The larger the Euclidean distance, the more likely the edge is to be cut. σ is a scale parameter used to control the sensitivity of the distance factor in weight calculation. It is a constant and its specific value can be set based on experience.
[0104] Those skilled in the art should understand that the calculation of the Euclidean distance between two points is well known in the art and will not be elaborated here.
[0105] Step SA2: assigning a second weight to the second type of edge in the point cloud graph.
[0106] In some embodiments, the second weight is a preset weight.
[0107] In this embodiment, the weight cost of the second type of edge connected to the source point S is forward It is a fixed value, which can be set according to experience.
[0108] Those skilled in the art should understand that the specific setting method of the second weight here is well known in the art and will not be elaborated here.
[0109] Step SA3: assigning a third weight to the third type of edge in the point cloud graph.
[0110] In some embodiments, the third weight is calculated based on the Euclidean distance between a point in the target point cloud and the source point and a preset radius of the target object.
[0111] Exemplarily, the calculation formula of the third weight is shown in formula (4):
[0112]
[0113] Among them, r is the preset radius of the target object, c x , c y is the coordinate of the source point S, x, y are the x-axis and y-axis coordinates of the point in the target point cloud.
[0114] Formula (4) means that points within the radius r tend to be retained, and points outside the radius r tend to be cut.
[0115] Step S2400.4, determining a minimum cut set in the point cloud image using a minimum cut algorithm according to the weight.
[0116] In this embodiment, the minimum cut algorithm can be calculated based on the maximum flow minimum cut theorem. The maximum flow minimum cut theorem means that in a network flow, the maximum flow that can be transmitted from the source point to the sink point is equal to the total capacity of the edges in the minimum cut set. In other words, if the maximum flow in the point cloud graph is found, then the minimum cut set corresponding to this maximum flow is the set of edges that, if removed, will reduce the flow to the maximum flow value. Therefore, the minimum cut set can be the path corresponding to the maximum flow from the source point to the sink point in the point cloud graph.
[0117] The maximum flow in the point cloud graph is solved by the maximum flow algorithm to obtain the minimum cut set. The weight in the above step S2400.3 is the flow in the maximum flow algorithm.
[0118] The maximum flow algorithm can be implemented by Ford-Fulkerson algorithm, Edmons-Karp algorithm, Dinic algorithm, Boykov-Kolmogorov algorithm, etc.
[0119] Those skilled in the art should understand that these maximum flow algorithms are well known in the art and will not be elaborated here.
[0120] In some embodiments, before determining the minimum cut set according to the target point cloud in step S2400, the method further includes:
[0121] The target point cloud is downsampled to obtain a downsampled target point cloud as a new target point cloud.
[0122] In this embodiment, before determining the minimum cut set, the target point cloud is downsampled to obtain the downsampled target point cloud, so as to determine the minimum cut set through the downsampled target point cloud.
[0123] The downsampling process may be: random sampling, voxel grid filtering, nearest neighbor sampling, uniform sampling, etc.
[0124] For example, uniform sampling is used to downsample the target point cloud (ie, the ROI region), which has fast calculation speed and small memory usage. The sampling interval is set to k, and only the points with the subscript m*k are retained from the n target point clouds.
[0125] By downsampling the target point cloud, the number of points in the target point cloud can be reduced, thereby simplifying the data volume of the target point cloud, reducing the complexity of subsequent processing (for example, calculation of determining the minimum cut set), and reducing computing resource consumption, while retaining the important features and structural information of the target point cloud data as much as possible.
[0126] Correspondingly, after determining the minimum cut set according to the target point cloud in step S2400, the method further includes:
[0127] The target point cloud is upsampled to obtain an upsampled target point cloud.
[0128] In this embodiment, since the minimum cut set is determined based on the down-sampled target point cloud, it lacks a part of the points in the target point cloud. In order to facilitate the subsequent correspondence of the minimum cut set to the color image during the image segmentation processing, the down-sampled target point cloud can be up-sampled after determining the minimum cut set to supplement the points simplified by the downsampling, that is, the number of points in the target point cloud after upsampling is the same as that of the target point cloud before downsampling.
[0129] Step S2500: performing image segmentation processing on the color image according to the minimum cut set to obtain and output a second image.
[0130] In this embodiment, after the minimum cut set is determined based on the depth image, the color image aligned with the depth image pixels is segmented by the minimum cut set to obtain a second image, wherein the second image is an image corresponding to the target object and does not include non-target objects.
[0131] The image segmentation process may be to set the pixel values corresponding to non-target objects in the color image to zero according to the minimum cut set, or to separate the pixel points corresponding to the target object and the pixel points corresponding to the non-target object in the color image according to the minimum cut set, which is not limited here.
[0132] In the example where the image segmentation process is to set the pixel values corresponding to the non-target object to zero, the output second image includes a black area, which is the non-target object, and the pixel value of the target object is the same as the pixel value of the target object in the first image. In the example where the image segmentation process is to separate the pixel points corresponding to the target object from the pixel points corresponding to the non-target object, the output second image is only the image of the target object, that is, the second image is smaller in size than the first image.
[0133] In order to enhance the accuracy of target segmentation, after the color image is segmented according to the minimum cut set, the target object can be confirmed by the pixel points of the segmented target object. When it is confirmed that the pixel points of the target object meet the requirements, the second image is output.
[0134] In some embodiments, obtaining and outputting the second image in step S2500 includes:
[0135] When the ratio of the pixel points corresponding to the target object in the obtained second image to the target point cloud is greater than or equal to the proportion threshold, the second image is output.
[0136] In this embodiment, the conveyor belt is used to transport a specific target object. For the specific target object, a proportion threshold can be set according to its area proportion on the conveyor belt. In addition, for a target object with a larger volume, a larger proportion threshold can be set, and for a target object with a smaller volume, a smaller proportion threshold can be set.
[0137] For example, for a particular item transported on a conveyor belt, the percentage threshold may be 2 / 3.
[0138] After obtaining the second image through step S2500, the ratio of the number of pixels corresponding to the target object and the number of target point clouds can be calculated, where the number of target point clouds is the number of pixels in the ROI area, and then the ratio is compared with the proportion threshold to confirm the target object. When the ratio of the number of pixels corresponding to the target object and the number of target point clouds is greater than or equal to the proportion threshold, the target object is segmented correctly and the second image is output. If the proportion is less than the proportion threshold, it means that the target object is segmented incorrectly, and step S2400 is re-executed to determine the minimum cut set based on the target point cloud.
[0139] According to the embodiment of the present application, the minimum cut set is determined by processing the depth image, and then the color image aligned with the depth image pixels is segmented by the minimum cut set to obtain the second image. Compared with the traditional target segmentation method, it can cope with target segmentation processing in complex scenes. Compared with the existing deep learning target segmentation method, it saves a lot of data collection and model training, improves the efficiency of target segmentation, and achieves more efficient and accurate target segmentation to meet the needs of industrial automation production. In addition, by screening the point cloud converted from the depth image according to the estimated distance range of the target object to obtain the target point cloud, abnormal points in the point cloud can be removed, simplifying the complexity of subsequent processing (such as determining the minimum cut set), and further improving the efficiency of target segmentation.
[0140] <Device Example>
[0141] Figure 5 FIG. 5 is a principle block diagram of an electronic device 500 according to an embodiment of the present invention.
[0142] In this embodiment, if Figure 5 As shown, the electronic device 500 includes a memory 510 and a processor 520, wherein the memory 510 is used to store executable instructions; the processor 520 is used to operate according to the control of the instructions to execute the method described in any of the above method embodiments.
[0143] In other embodiments, the electronic device may also be Figure 1 Electronic equipment shown.
[0144] An embodiment of the present application further provides a computer-readable storage medium having computer-executable instructions stored thereon, and when the executable instructions are executed by a processor, the target segmentation method as described in any of the above method embodiments is implemented.
[0145] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0146] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0147] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0148] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0149] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0152] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.
[0153] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.
Claims
1. A target segmentation method, characterized in that: include: Acquire a first image captured by a camera; wherein the first image includes a pixel-aligned color image and a depth image, and the color image includes target objects and non-target objects; According to the camera intrinsic parameters of the camera, the depth image is converted into a point cloud in a camera coordinate system; The point cloud is screened and processed according to an estimated distance range for the target object to obtain a target point cloud; wherein the estimated distance range is an estimated distance range between the target object and the camera; Determine a minimum cut set according to the target point cloud; Perform image segmentation processing on the color image according to the minimum cut set to obtain and output a second image.
2. The method according to claim 1, characterized in that In the case where the first image is a monitoring image of a set monitoring area for a conveyor belt, the estimated distance range is determined according to the installation height of the camera, the laying height of the conveyor belt, and the noise of the shooting scene.
3. The method according to claim 1, characterized in that Determining a minimum cut set according to the target point cloud includes: Determine a source point and a sink point according to the target point cloud; wherein the source point corresponds to the target object, and the sink point corresponds to the non-target object; Constructing a point cloud map according to the target point cloud, the source point and the sink point; Assigning a corresponding weight to each edge in the point cloud graph; According to the weights, a minimum cut set is determined in the point cloud image by using a minimum cut algorithm.
4. The method according to claim 3, characterized in that The point cloud graph includes a first type of edge, a second type of edge, and a third type of edge, wherein the first type of edge is a line between any two adjacent points in the target point cloud, the second type of edge is a line between the source point and a point in the target point cloud, and the third type of edge is a line between the sink point and a point in the target point cloud, and the corresponding weight is assigned to each edge in the point cloud graph, including: Assigning a first weight to the first type of edge in the point cloud graph; assigning a second weight to the second type of edge in the point cloud graph; A third weight is assigned to the third type of edge in the point cloud graph.
5. The method according to claim 4, characterized in that The first weight is calculated based on the Euclidean distance and scale parameter of any two adjacent points in the target point cloud, the second weight is a preset weight, and the third weight is calculated based on the Euclidean distance between the point in the target point cloud and the source point and the preset radius of the target object.
6. The method according to claim 3, characterized in that The source point is the centroid of the target point cloud, and the sink point is the point farthest from the centroid.
7. The method according to claim 1, characterized in that Before determining the minimum cut set according to the target point cloud, the method further includes: The target point cloud is downsampled to obtain a downsampled target point cloud as a new target point cloud.
8. The method according to claim 1, characterized in that The obtaining and outputting the second image comprises: When the ratio of the pixel points corresponding to the target object in the obtained second image to the target point cloud is greater than or equal to the proportion threshold, the second image is output.
9. An electronic device comprising a memory and a processor, wherein the memory is used to store executable instructions; and the processor is used to operate according to the control of the instructions to execute the method according to any one of claims 1 to 8.
10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by the processor, the target segmentation method according to any one of claims 1 to 8 is implemented.