Spherical or quasi-spherical object image segmentation method, device, equipment and storage medium

Abstract

The present invention relates to artificial intelligence technology and provides a method, device, equipment and storage medium for image segmentation of spherical or quasi-spherical objects. The method comprises: obtaining an image to be processed and a depth map to be processed, wherein the image to be processed includes multiple pixels and the depth map to be processed is used to indicate the depth information of the pixels; performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes multiple spherical or quasi-spherical objects; performing semantic segmentation on the target detection area to obtain a target pixel area; mapping the target pixel area to the depth map to be processed to obtain a mapped image; and performing segmentation processing on the mapped image to obtain a segmented image of each of the spherical or quasi-spherical objects. According to the solution provided in an embodiment of the present invention, spherical or quasi-spherical objects that occlude or overlap each other in an image can be effectively distinguished.

Description

Technical Field

[0001] The present invention relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for segmenting images of spherical or quasi-spherical objects. Background Art

[0002] Industrial automation generally refers to the use of digital technology to detect, control, optimize, schedule, manage and make decisions on industrial production processes in order to achieve comprehensive goals such as increasing output, improving quality, reducing consumption and ensuring safety.

[0003] In the industrial production process, when spherical or quasi-spherical objects need to be processed, the spherical or quasi-spherical objects are usually identified and located first. At present, the object image is obtained through a visual system combining a camera and a depth sensor, and then the edge detection of the object image is performed to obtain the target boundary. When spherical or quasi-spherical objects that occlude or overlap each other appear in the object image, the spherical or quasi-spherical objects are distinguished by performing concave point detection on the target boundary, or the spherical or quasi-spherical objects are distinguished by the color information of the object image, thereby realizing the identification and positioning of the spherical or quasi-spherical objects; however, for complex working environments, the target boundary usually cannot accurately reflect the edge of the spherical or quasi-spherical object, and the concave point detection cannot effectively distinguish the spherical or quasi-spherical objects that occlude or overlap each other in the image. In addition, the background color in the working environment is complex, and the color information of the object image cannot effectively distinguish the spherical or quasi-spherical objects that occlude or overlap each other in the image. Summary of the Invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] Embodiments of the present invention provide a method, apparatus, device, and storage medium for segmenting images of spherical or quasi-spherical objects, which can effectively distinguish spherical or quasi-spherical objects that occlude or overlap each other in an image.

[0006] In the first aspect, the present invention also provides a method for image segmentation of spherical or quasi-spherical objects, comprising: obtaining an image to be processed and a depth map to be processed, wherein the image to be processed includes multiple pixel points, and the depth map to be processed is used to indicate the depth information of the pixel points; performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes multiple spherical or quasi-spherical objects; performing semantic segmentation on the target detection area to obtain a target pixel area; mapping the target pixel area to the depth map to be processed to obtain a mapping image; performing segmentation processing on the mapping image to obtain a segmented image of each of the spherical or quasi-spherical objects.

[0007] In some embodiments, the segmentation processing of the mapping image to obtain the segmented images of each of the spherical or quasi-spherical objects includes: determining a histogram based on the pixels of the mapping image and the depth information of the pixels; smoothing the histogram to determine a fitting curve; determining segmentation information based on the fitting curve; and segmenting the mapping image according to the segmentation information to obtain the segmented images of each of the spherical or quasi-spherical objects.

[0008] In some embodiments, the segmentation information is the number of all peaks of the fitting curve and the depth information corresponding to each of the peaks; the segmentation processing of the mapping image according to the segmentation information to obtain the segmented image of each of the spherical or quasi-spherical objects includes: clustering the mapping image based on the number of the peaks, the depth information corresponding to each of the peaks and a preset clustering algorithm to determine multiple cluster sets; segmenting the mapping image based on the cluster sets to obtain the segmented image of each of the spherical or quasi-spherical objects.

[0009] In some embodiments, the segmentation information is the number of all troughs of the fitting curve and the depth information corresponding to each of the troughs; the segmentation processing of the mapping image according to the segmentation information to obtain the segmented image of each of the spherical or quasi-spherical objects includes: based on the number of the troughs and the depth information corresponding to each of the troughs, the segmentation processing of the mapping image to obtain the segmented image of each of the spherical or quasi-spherical objects.

[0010] In some embodiments, the segmentation processing of the mapped image to obtain segmented images of each of the spherical or quasi-spherical objects includes: determining a minimum bounding rectangle based on the geometric features of the target pixel area; determining the lengths of two adjacent sides of the minimum bounding rectangle, and when the lengths of the two adjacent sides of the minimum bounding rectangle are different, connecting the midpoints of the two short sides of the minimum bounding rectangle to obtain a baseline; determining the depth information of each pixel point of the target pixel area located on the baseline based on the depth map to be processed; determining depth mutation pixels from the pixels of the target pixel area based on the depth information of each pixel point of the target pixel area located on the baseline and a preset depth mutation threshold, and determining the depth information of the depth mutation pixels; segmenting the mapped image based on the depth information of the depth mutation pixels to obtain segmented images of each of the spherical or quasi-spherical objects.

[0011] In some embodiments, the segmentation processing of the mapping image to obtain the segmented image of each of the spherical or quasi-spherical objects includes: segmenting the mapping image to obtain image subintervals corresponding to each of the spherical or quasi-spherical objects; assigning values ​​to each of the image subintervals based on a preset color palette so that the color values ​​of pixels in any two of the image subintervals are different; and determining the segmented image of each of the spherical or quasi-spherical objects based on all the image subintervals after the assignment processing.

[0012] In some embodiments, the semantic segmentation of the target detection area to obtain the target pixel area includes: determining the red channel value, green channel value, and blue channel value of each pixel in the target detection area; determining the color difference of each pixel in the target detection area based on the red channel value, green channel value, and blue channel data; screening each pixel in the target detection area according to the preset difference threshold and the color difference, and determining the target pixel from the pixels in the target detection area; and obtaining the target pixel area based on the target pixel.

[0013] In some embodiments, the semantic segmentation of the target detection area to obtain the target pixel area includes: inputting the target detection area into a trained spherical or quasi-spherical object semantic segmentation model to obtain the target pixel area; wherein the spherical or quasi-spherical object semantic segmentation model is trained by the following steps: obtaining a spherical or quasi-spherical object training image and a spherical or quasi-spherical object training label image; inputting the spherical or quasi-spherical object training image into a preset semantic segmentation model to obtain a predicted area; determining a cross-entropy loss function based on the spherical or quasi-spherical object training label image and the predicted area; and updating the model parameters of the semantic segmentation model based on the cross-entropy loss function to obtain a spherical or quasi-spherical object semantic segmentation model.

[0014] In a second aspect, the present invention also provides a device for segmenting images of spherical or quasi-spherical objects, comprising: an acquisition unit for acquiring an image to be processed and a depth map to be processed, wherein the image to be processed includes multiple pixel points, and the depth map to be processed is used to indicate the depth information of the pixel points; a detection unit for performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes multiple spherical or quasi-spherical objects; a first segmentation unit for performing semantic segmentation on the target detection area to obtain a target pixel area; a depth image determination unit for mapping the target pixel area to the depth map to be processed to obtain a mapping image; and a second segmentation unit for performing segmentation processing on the mapping image to obtain segmented images of each of the spherical or quasi-spherical objects.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for image segmentation of spherical or quasi-spherical objects as described in the first aspect above is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for segmenting spherical or quasi-spherical object images as described in the first aspect above.

[0017] The embodiment of the present invention includes: obtaining an image to be processed and a depth map to be processed, wherein the image to be processed includes a plurality of pixels, and the depth map to be processed is used to indicate the depth information of the pixels; performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects; performing semantic segmentation on the target detection area to obtain a target pixel area; mapping the target pixel area to the depth map to be processed to obtain a mapped image; and performing segmentation processing on the mapped image to obtain a segmented image of each of the spherical or quasi-spherical objects. According to the solution provided by the embodiment of the present invention, by performing target detection on the image to be processed to obtain a target detection area including a plurality of spherical or quasi-spherical objects, performing semantic segmentation on the target detection area, mapping the obtained target pixel area to the depth map to be processed to obtain a mapped image, and then performing segmentation processing on the mapped image to obtain a segmented image of each spherical or quasi-spherical object, it is achieved that in a complex working environment, spherical or quasi-spherical objects that are mutually occluded or overlapped in the image can be effectively distinguished.

[0018] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0020] Figure 1 A schematic diagram of an implementation environment provided for an embodiment of the present invention;

[0021] Figure 2 is a flow chart of a method for segmenting a spherical or quasi-spherical object image provided by one embodiment of the present invention;

[0022] Figure 3is a flow chart of segmentation according to segmentation information provided by another embodiment of the present invention;

[0023] Figure 4 This is a flow chart of segmentation according to peaks provided by another embodiment of the present invention;

[0024] Figure 5 This is a flow chart of segmentation according to troughs provided by another embodiment of the present invention;

[0025] Figure 6 is a flowchart of segmentation according to a baseline provided by another embodiment of the present invention;

[0026] Figure 7 is a flowchart of determining a segmented image provided by another embodiment of the present invention;

[0027] Figure 8 is another flow chart for obtaining a target pixel area provided by another embodiment of the present invention;

[0028] Figure 9 is another flow chart for obtaining a target pixel area provided by another embodiment of the present invention;

[0029] Figure 10 is a schematic diagram of an image to be processed provided by another embodiment of the present invention;

[0030] Figure 11 is a schematic diagram of a mapping image provided by another embodiment of the present invention;

[0031] Figure 12 is a schematic diagram of a histogram provided by another embodiment of the present invention;

[0032] Figure 13 is a schematic diagram of a fitting curve provided by another embodiment of the present invention;

[0033] Figure 14 is a schematic diagram of a target pixel area including a reference line provided by another embodiment of the present invention;

[0034] Figure 15 is a structural diagram of a spherical or quasi-spherical object image segmentation device provided by another embodiment of the present invention;

[0035] Figure 16 It is a structural diagram of an electronic device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] In the description of the present invention, “several” means one or more, “more” means more than two, “greater than”, “less than”, “exceed”, etc. are understood to exclude the number itself, and “above”, “below”, “within”, etc. are understood to include the number itself.

[0038] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0039] Before further explaining the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations:

[0040] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of perception, reasoning, and decision-making. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily include computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning. With the research and advancement of AI technology, AI has been studied and applied in many fields, such as smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, driverless and autonomous driving, robotics, smart healthcare, and smart cities. It is believed that with the development of technology, AI will be applied in more and more fields and play an increasingly important role.

[0041] Machine Learning (ML) is a multidisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0042] Spherical and quasi-spherical objects refer to standard spherical objects, ellipsoids and irregular spheres, such as apples, pears, peaches, oranges, tangerines, coconuts and other fruits and vegetables.

[0043] Object detection, also known as object extraction, is an image segmentation based on the geometric and statistical features of the object.

[0044] Structured light is a system consisting of a projector and a camera. The projector projects specific light information onto the surface of an object and the background. The camera then collects the information and calculates the object's position, depth, and other information based on the changes in the light signal caused by the object, thereby restoring the entire three-dimensional space.

[0045] Time of flight (TOF) can be broadly understood as a technology that further understands certain properties of ions or media by measuring the time it takes for an object, particle, or wave to fly a certain distance in a fixed medium (the medium / distance / time are all known or measurable).

[0046] Kernel density estimation is used to estimate unknown density functions in probability theory and is one of the non-parametric test methods.

[0047] The K-means clustering algorithm is an iterative clustering analysis algorithm. Its steps are as follows: divide the data into K groups in advance, randomly select K objects as the initial cluster centers, then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it; the cluster centers and the objects assigned to them represent a cluster; each time a sample is assigned, the cluster center will be recalculated based on the existing objects in the cluster; this process will be repeated until a termination condition is met; the termination condition can be that no (or a minimum number of) objects are reassigned to different clusters, no (or a minimum number of) cluster centers change again, and the sum of squared errors is locally minimized.

[0048] Cross Entropy is mainly used to measure the difference information between two probability distributions.

[0049] In the industrial production process, when spherical or quasi-spherical objects need to be processed, the spherical or quasi-spherical objects are usually identified and located first. At present, the object image is obtained through a visual system combining a camera and a depth sensor, and then the edge detection of the object image is performed to obtain the target boundary. When spherical or quasi-spherical objects that occlude or overlap each other appear in the object image, the spherical or quasi-spherical objects are distinguished by performing concave point detection on the target boundary, or the spherical or quasi-spherical objects are distinguished by the color information of the object image, thereby realizing the identification and positioning of the spherical or quasi-spherical objects; however, for complex working environments, the target boundary usually cannot accurately reflect the edge of the spherical or quasi-spherical object, and the concave point detection cannot effectively distinguish the spherical or quasi-spherical objects that occlude or overlap each other in the image. In addition, the background color in the working environment is complex, and the color information of the object image cannot effectively distinguish the spherical or quasi-spherical objects that occlude or overlap each other in the image.

[0050] In response to the problem of being unable to effectively distinguish spherical or quasi-spherical objects that occlude or overlap each other in an image, the present invention provides a method, device, equipment and storage medium for image segmentation of spherical or quasi-spherical objects. The method includes: obtaining an image to be processed and a depth map to be processed, wherein the image to be processed includes multiple pixels and the depth map to be processed is used to indicate the depth information of the pixels; performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes multiple spherical or quasi-spherical objects; performing semantic segmentation on the target detection area to obtain a target pixel area; mapping the target pixel area to the depth map to be processed to obtain a mapped image; and performing segmentation processing on the mapped image to obtain a segmented image of each spherical or quasi-spherical object. According to the solution provided by an embodiment of the present invention, target detection is performed on the image to be processed to obtain a target detection area including multiple spherical or quasi-spherical objects, and semantic segmentation is performed on the target detection area. The obtained target pixel area is mapped to the depth map to be processed to obtain a mapping image, and then the mapping image is segmented to obtain segmented images of each spherical or quasi-spherical object. This achieves the ability to effectively distinguish spherical or quasi-spherical objects that occlude or overlap each other in the image in a complex working environment.

[0051] The embodiment of the present invention provides a method for segmenting an image of a spherical or quasi-spherical object, which can be applied in a terminal or a server, and can also be software running in a terminal or a server.

[0052] Reference Figure 1 , Figure 1This is a schematic diagram of an implementation environment provided by an embodiment of the present invention. The implementation environment includes a terminal 101 and a server 102 , wherein the terminal 101 and the server 102 are connected via a communication network 103 .

[0053] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0054] In addition, server 102 may also be a node server in the blockchain network.

[0055] The terminal 101 may be a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a Pocket PC (PPC), a tablet computer, a camera with a camera function, a data acquisition device, etc., but is not limited thereto. The terminal 101 and the server 102 may be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0056] based on Figure 1 The implementation environment, such as Figure 2 As shown, Figure 2 This is a flowchart of a method for segmenting a spherical or quasi-spherical object image provided by one embodiment of the present invention, including but not limited to the following steps:

[0057] Step S210: obtaining an image to be processed and a depth map to be processed, wherein the image to be processed includes a plurality of pixels, and the depth map to be processed is used to indicate depth information of the pixels;

[0058] Step S220, performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects;

[0059] Step S230, performing semantic segmentation on the target detection area to obtain a target pixel area;

[0060] Step S240, mapping the target pixel area to the depth map to be processed to obtain a mapped image;

[0061] Step S250 : performing segmentation processing on the mapped image to obtain segmented images of each spherical or quasi-spherical object.

[0062] It can be understood that when a part of a spherical or quasi-spherical object is blocked by an occluder, relative to edge detection of the object image, by obtaining a depth image of the spherical or quasi-spherical object, and then performing semantic segmentation on the spherical or quasi-spherical object image, the target pixel area of ​​the area where the spherical or quasi-spherical object is located is obtained, and then the target pixel area is mapped to the spherical or quasi-spherical object depth map, and the target point cloud containing point cloud information is determined. The point cloud information refers to the position information and depth information of each point in the point cloud. Through multiple experiments, the target depth value of the spherical or quasi-spherical object is determined, and the point in the target point cloud whose depth value meets the target depth value is taken as The points to be fitted are collected to form a set of points to be fitted, and then circle detection is performed on the set of points to be fitted, so as to determine the target fitting circle and the target center coordinates corresponding to the target fitting circle. Based on this, target detection is performed on the image to be processed to obtain a target detection area including multiple spherical or quasi-spherical objects, and semantic segmentation is performed on the target detection area. The obtained target pixel area is mapped to the depth map to be processed to obtain a mapping image, and then the mapping image is segmented to obtain segmented images of each spherical or quasi-spherical object, so that spherical or quasi-spherical objects that are occluded or overlapped with each other in the image can be effectively distinguished in a complex working environment.

[0063] It should be noted that the cameras for obtaining the images to be processed and the depth maps to be processed include but are not limited to: structured light depth cameras, binocular depth cameras, and optical time-of-flight depth cameras; the images to be processed include but are not limited to: color images, pseudo-color images, grayscale images, and black and white images.

[0064] In addition, refer to Figure 3 In one embodiment, Figure 2 Step S250 in the illustrated embodiment includes but is not limited to the following steps:

[0065] Step S310, determining a histogram based on the pixels of the mapped image and the depth information of the pixels;

[0066] Step S320, smoothing the histogram to determine a fitting curve;

[0067] Step S330, determining segmentation information according to the fitting curve;

[0068] Step S340 : performing segmentation processing on the mapped image according to the segmentation information to obtain segmented images of each spherical or quasi-spherical object.

[0069] It can be understood that by performing smoothing, a fitting curve can be obtained, and smoothing can remove noise to ensure the accuracy of the segmentation information; the fitting curve is determined by the number of pixels and the depth information of the pixels. By analyzing the fitting curve, the segmentation information can be determined, and the mapped image can be effectively segmented according to the segmentation information, thereby obtaining the segmented images of each spherical or quasi-spherical object.

[0070] In specific practice, the smoothing methods include but are not limited to: kernel density estimation method and Gaussian filtering method; the horizontal axis of the histogram is the depth value of the pixel point, and the vertical axis of the histogram is the number of pixels or the frequency of the pixel points, where the frequency of the pixel point is the ratio of the current number of pixels to the total number of pixels at the current depth value.

[0071] In addition, refer to Figure 4 In one embodiment, the segmentation information is the number of all peaks of the fitting curve and the depth information corresponding to each peak; Figure 3 Step S340 in the illustrated embodiment includes but is not limited to the following steps:

[0072] Step S410, clustering the mapped image based on the number of peaks, depth information corresponding to each peak, and a preset clustering algorithm to determine a plurality of cluster sets;

[0073] Step S420 : performing segmentation processing on the mapped image based on the cluster set to obtain segmented images of each spherical or quasi-spherical object.

[0074] It can be understood that by analyzing the fitting curve, the number of all peaks and the position of each peak are determined, and the depth information corresponding to each peak in the histogram is determined through the position of the peak. The number of spherical or quasi-spherical objects that occlude or overlap each other in the image is the number of all peaks in the fitting curve. After determining the number of spherical or quasi-spherical objects and the depth information corresponding to each peak in the histogram, since the position and depth value of the pixel points corresponding to the same spherical or quasi-spherical object in the mapped image are correlated, the clustering algorithm can be used to effectively determine the position of each spherical or quasi-spherical object in the mapped image. The clustering set corresponds one-to-one to the spherical or quasi-spherical object, and the mapped image can be effectively segmented to obtain a segmented image of each spherical or quasi-spherical object.

[0075] In specific practice, clustering algorithms include but are not limited to: K-means clustering algorithm.

[0076] It is worth noting that the method for determining the number and position of peaks in the fitting curve is a technique well known to those skilled in the art and will not be elaborated herein.

[0077] In addition, refer to Figure 5In one embodiment, the segmentation information is the number of all troughs of the fitting curve and the depth information corresponding to each trough; Figure 3 Step S340 in the illustrated embodiment includes but is not limited to the following steps:

[0078] Step S510 : Segment the mapped image based on the number of troughs and the depth information corresponding to each trough to obtain segmented images of each spherical or quasi-spherical object.

[0079] It can be understood that by analyzing the fitting curve, the number of all troughs and the position of each trough can be determined, and the depth information corresponding to each trough in the histogram can be determined through the position of the trough. The number of intersection points of spherical or quasi-spherical objects that occlude or overlap each other in the image is the number of all troughs in the fitting curve, and the position of the intersection point of the spherical or quasi-spherical object is equivalent to the position corresponding to the depth information of the trough in the mapped image. Therefore, through the number of troughs and the depth information corresponding to each trough, the mapped image can be effectively segmented to obtain a segmented image of each spherical or quasi-spherical object.

[0080] It is worth noting that the method for determining the number and position of troughs in the fitting curve is a technique well known to those skilled in the art and will not be elaborated herein.

[0081] In addition, refer to Figure 6 In one embodiment, Figure 2 Step S250 in the illustrated embodiment includes but is not limited to the following steps:

[0082] Step S610, determining a minimum bounding rectangle based on the geometric features of the target pixel area;

[0083] Step S620, determining the lengths of two adjacent sides of the minimum circumscribed rectangle. If the lengths of the two adjacent sides of the minimum circumscribed rectangle are different, connecting the midpoints of the two short sides of the minimum circumscribed rectangle to obtain a reference line.

[0084] Step S630, determining the depth information of each pixel point on the baseline in the target pixel area based on the depth map to be processed;

[0085] Step S640, determining a depth mutation pixel from the pixels in the target pixel area according to the depth information of each pixel on the baseline in the target pixel area and a preset depth mutation threshold, and determining the depth information of the depth mutation pixel;

[0086] Step S650 : Segment the mapped image according to the depth information of the depth mutation pixel points to obtain segmented images of the spherical or quasi-spherical objects.

[0087] It can be understood that the geometric features of the target pixel area refer to the shape of the area where the spherical or quasi-spherical object is located within the target pixel area. The geometric features of the target pixel area are processed into a circumscribed rectangle to determine the minimum circumscribed rectangle. For two mutually occluding or overlapping spherical or quasi-spherical objects in the image, when the lengths of the two adjacent sides of the minimum circumscribed rectangle are different, that is, when the shape of the minimum circumscribed rectangle is a rectangle, the two mutually occluding or overlapping spherical or quasi-spherical objects in the image are in a state of being arranged in sequence along one direction. The line segment connecting the midpoints of the two short sides of the target detection area is used as the baseline. The baseline will pass through each spherical or quasi-spherical object, and the two spherical or quasi-spherical objects will pass through the spherical or quasi-spherical objects. The depth value at the intersection of two spherical or quasi-spherical objects will change significantly. The user sets the depth mutation threshold based on the experimental results and determines the depth information of each pixel on the baseline of the target pixel area based on the depth map to be processed. When the difference between the depth value of a pixel and the depth value of the adjacent pixel exceeds the depth mutation threshold, it means that the pixel is located at the intersection of two spherical or quasi-spherical objects that occlude or overlap each other in the image. The pixel is then used as the depth mutation pixel. The depth information of the depth mutation pixel is then determined. Based on the depth information of all depth mutation pixels, the mapped image can be segmented to obtain segmented images of each spherical or quasi-spherical object.

[0088] It should be noted that the direction of the right-angled side of the minimum circumscribed rectangle may be different from the direction of the right-angled side of the target detection area, the direction of the right-angled side of the minimum circumscribed rectangle may form an angle with the direction of the right-angled side of the target detection area, and the direction of the coordinate axis of the minimum circumscribed rectangle is determined by the area where the spherical or quasi-spherical object is located in the target pixel area.

[0089] In actual practice, when multiple spherical or quasi-spherical objects that occlude or overlap each other are arranged in sequence along one direction, the position of the intersection of each spherical or quasi-spherical object in the image can be accurately determined by the baseline, and there is no limit to the number of spherical or quasi-spherical objects.

[0090] like Figure 7 As shown, in one embodiment, Figure 2 Step S250 in the illustrated embodiment includes but is not limited to the following steps:

[0091] Step S710, segmenting the mapped image to obtain image sub-intervals corresponding to various spherical or quasi-spherical objects;

[0092] Step S720: Based on a preset color palette, assign values ​​to each image sub-interval so that the color values ​​of pixels in any two image sub-intervals are different.

[0093] Step S730 : determining the segmented images of each spherical or quasi-spherical object based on all the image sub-intervals after the value assignment process.

[0094] It can be understood that, according to the color sorting of the color palette, each image sub-interval is assigned a value in turn. For example, the color values ​​of all pixels in the first image sub-interval are assigned to the color value corresponding to the first color in the color palette, and the color values ​​of all pixels in the second image sub-interval are assigned to the color value corresponding to the second color in the color palette. This ensures that all pixels in the same image sub-interval are assigned the same color value, and pixels in different image sub-intervals are assigned different color values. The assigned sub-intervals are then mapped to a color image or a pseudo-color image to obtain segmented images of each spherical or quasi-spherical object. The colors of different spherical or quasi-spherical objects are different and easy to distinguish, so that users can intuitively and effectively determine the positions of each spherical or quasi-spherical object that occludes or overlaps each other in the image.

[0095] It is worth noting that assigning color values ​​to all pixels within an image subinterval is a technique well known to those skilled in the art and will not be elaborated on here.

[0096] In specific practice, a color palette refers to a color palette that is systematically arranged by combining red, green, blue, yellow, magenta, cyan and other colors, plus a neutral gray palette from white to black. Preferably, in order to better distinguish between various spherical or spherical objects, a color palette with significantly different colors is used.

[0097] like Figure 8 As shown, in one embodiment, Figure 2 Step S230 in the illustrated embodiment includes but is not limited to the following steps:

[0098] Step S810, determining the red channel value, green channel value, and blue channel value of each pixel in the target detection area;

[0099] Step S820, determining the color difference of each pixel in the target detection area according to the red channel value, the green channel value and the blue channel data;

[0100] Step S830, screening each pixel in the target detection area according to a preset difference threshold and color difference, and determining the target pixel from the pixels in the target detection area;

[0101] Step S840: Obtain a target pixel area according to the target pixel point.

[0102] It can be understood that each pixel has three color channels, namely red channel, green channel and blue channel, and the color difference of the pixel refers to the difference between the red channel value and the green channel value, or the difference between the red channel value and the blue channel value, or the difference between the green channel value and the blue channel value; in an image of a spherical or quasi-spherical object, when the difference between the color of the spherical or quasi-spherical object and the color of the image background is large, by calculating the color difference of each pixel and combining it with a preset difference threshold, the target pixel area where the spherical or quasi-spherical object is located can be accurately obtained.

[0103] It is worth noting that determining the values ​​of each color channel of a pixel is a technique well known to those skilled in the art and will not be elaborated on here.

[0104] In specific practice, when the color of the spherical or quasi-spherical object is red and the color of the image background is green, the difference between the red channel value and the green channel value is taken as the target color difference. By calculating the target color difference of each pixel point, the difference threshold is set to a larger value, and it is judged whether the target color difference of the pixel point is greater than the difference threshold. If so, it means that the pixel point is in the area where the spherical or quasi-spherical object is located, and the pixel point is taken as the target pixel point; otherwise, the next pixel point is judged; after judging all pixel points, all target pixel points are screened to obtain, thereby obtaining the target pixel area of ​​the area where the spherical or quasi-spherical object is located.

[0105] like Figure 9 As shown, in one embodiment, Figure 2 Step S230 in the illustrated embodiment includes but is not limited to the following steps:

[0106] Step S910, inputting the target detection area into the trained spherical or quasi-spherical object semantic segmentation model to obtain the target pixel area;

[0107] The semantic segmentation model for spherical or quasi-spherical objects is trained through the following steps:

[0108] Step 910: Acquire a spherical or quasi-spherical object training image and a spherical or quasi-spherical object training label image;

[0109] Step 920: Input the spherical or quasi-spherical object training image into a preset semantic segmentation model to obtain a predicted region;

[0110] Step 930 , determining a cross entropy loss function based on the training label image and the prediction region of the spherical or quasi-spherical object;

[0111] Step 940: Based on the cross entropy loss function, update the model parameters of the semantic segmentation model to obtain a semantic segmentation model of spherical or quasi-spherical objects.

[0112] It can be understood that the cross entropy loss function is determined by training the label image and the prediction area of ​​the spherical or quasi-spherical object, thereby determining the cross entropy loss, and then the cross entropy loss is used to update the model parameters of the semantic segmentation model, so that the spherical or quasi-spherical object semantic segmentation model can improve the segmentation accuracy, and then the spherical or quasi-spherical object semantic segmentation model can accurately obtain the target pixel area in the area where the spherical or quasi-spherical object is located.

[0113] In specific practice, the semantic segmentation model uses the Unet segmentation network.

[0114] It is worth noting that the calculation method of the cross entropy loss function and the method of obtaining the Unet segmentation network are technologies well known to those skilled in the art and will not be elaborated here.

[0115] In addition, reference Figures 10 to 14 , Figure 10 is a schematic diagram of an image to be processed provided by another embodiment of the present invention; Figure 11 is a schematic diagram of a mapping image provided by another embodiment of the present invention; Figure 12 is a schematic diagram of a histogram provided by another embodiment of the present invention; Figure 13 is a schematic diagram of a fitting curve provided by another embodiment of the present invention; Figure 14 FIG. 4 is a schematic diagram of a target pixel area including a baseline provided by another embodiment of the present invention.

[0116] It should be noted that with the rapid development of global informatization, digital agriculture has become a new trend in modern agriculture. Using machine vision technology to accurately determine the number of fruits on fruit trees is crucial for fruit yield measurement and precision orchard management. Traditional fruit yield estimation relies on historical data, weather conditions, and manual counting. This method is not only subject to significant errors but also time-consuming and labor-intensive, and no longer meets the needs of modern agricultural development. With the advancement of modern information technology, machine vision-based yield measurement methods have shown potential to replace manual methods. Currently, the automated harvesting of agricultural products requires fruit identification and location. For spherical or quasi-spherical fruits, existing methods using pit detection and color information to distinguish between occluded or overlapping objects in images are unable to effectively distinguish between occluded or overlapping fruits in complex growing environments.

[0117] It can be understood that the image segmentation method of spherical or quasi-spherical objects in the embodiment of the present invention can be applied to digital agriculture, such as an automated picking system for spherical or quasi-spherical fruits, wherein spherical or quasi-spherical fruits refer to fruits such as apples, pears, peaches, oranges, tangerines, and coconuts; by performing target detection on the image to be processed, the image to be processed refers to an image containing fruits that are mutually occluded or overlapped in the image, obtaining a target detection area, the target detection area includes multiple spherical or quasi-spherical fruits, and then performing semantic segmentation on the target detection area to obtain the following: Figure 10 The target pixel area shown is then mapped to the depth map to be processed to obtain the following Figure 11 The mapping image is then segmented; Figure 12 As shown, the segmentation process is performed through the histogram. First, the histogram is smoothed to obtain Figure 13 The diagram shown in the figure is then used to determine the number and location of the peaks or troughs, or as Figure 14 As shown, segmentation processing is performed through the baseline to determine the depth mutation pixel points; then the mapped image is segmented to obtain the segmented images of each spherical or quasi-spherical fruit. This makes it possible to effectively distinguish spherical or quasi-spherical fruits that are occluded or overlapped with each other in the image in a complex working environment, thereby ensuring the effective operation of the automated picking system for spherical or quasi-spherical fruits.

[0118] In addition, reference Figure 15 The present invention further provides a spherical or quasi-spherical object image segmentation device 1500, comprising:

[0119] An acquiring unit 1510 is configured to acquire an image to be processed and a depth map to be processed, wherein the image to be processed includes a plurality of pixels, and the depth map to be processed is used to indicate depth information of the pixels;

[0120] A detection unit 1520 is configured to perform target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects;

[0121] The first segmentation unit 1530 is used to perform semantic segmentation on the target detection area to obtain a target pixel area;

[0122] The depth image determining unit 1540 is configured to map the target pixel area to the depth map to be processed to obtain a mapped image;

[0123] The second segmentation unit 1550 is configured to perform segmentation processing on the mapped image to obtain segmented images of various spherical or quasi-spherical objects.

[0124] It can be understood that the specific implementation of the spherical or quasi-spherical object image segmentation device 1500 is basically the same as the specific embodiment of the above-mentioned spherical or quasi-spherical object image segmentation method, and will not be repeated here; based on this, by performing target detection on the image to be processed, a target detection area including multiple spherical or quasi-spherical objects is obtained, and the target detection area is semantically segmented, the obtained target pixel area is mapped to the depth map to be processed to obtain a mapping image, and then the mapping image is segmented to obtain segmented images of each spherical or quasi-spherical object, thereby achieving effective distinction between spherical or quasi-spherical objects that occlude or overlap each other in the image in a complex working environment.

[0125] In addition, refer to Figure 16 An embodiment of the present invention further provides an electronic device, which includes a memory 1610, a processor 1620, and a computer program stored in the memory 1610 and executable on the processor 1620.

[0126] The processor 1620 and the memory 1610 may be connected via a bus or other means.

[0127] The non-transient software program and instructions required to implement the spherical or quasi-spherical object image segmentation method of the above embodiment are stored in the memory 1610. When executed by the processor 1620, the spherical or quasi-spherical object image segmentation method of the above embodiment is executed, for example, the above-described Figure 2 Steps S210 to S250 of the method, Figure 3 Steps S310 to S340 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5 Step S510 of the method, Figure 6 Steps S610 to S650 of the method, Figure 7 Steps S710 to S730 of the method, Figure 8 Steps S810 to S840 of the method, Figure 9In step S910 of the method, an image to be processed and a depth map to be processed are obtained, wherein the image to be processed includes a plurality of pixels and the depth map to be processed is used to indicate the depth information of the pixels; target detection is performed on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects; semantic segmentation is performed on the target detection area to obtain a target pixel area; the target pixel area is mapped to the depth map to be processed to obtain a mapping image; and segmentation processing is performed on the mapping image to obtain a segmented image of each spherical or quasi-spherical object. Based on this, by performing target detection on the image to be processed to obtain a target detection area including a plurality of spherical or quasi-spherical objects, semantic segmentation is performed on the target detection area, the obtained target pixel area is mapped to the depth map to be processed to obtain a mapping image, and then segmentation processing is performed on the mapping image to obtain a segmented image of each spherical or quasi-spherical object, it is achieved that in a complex working environment, spherical or quasi-spherical objects that are mutually occluded or overlapped in the image can be effectively distinguished.

[0128] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0129] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or controller, for example, by a processor in the above-mentioned electronic device embodiment, so that the above-mentioned processor can execute the spherical or quasi-spherical object image segmentation method in the above-mentioned embodiment, for example, executing the above-mentioned Figure 2 Steps S210 to S250 of the method, Figure 3 Steps S310 to S340 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5 Method step S510, Figure 6 Steps S610 to S650 of the method, Figure 7 Steps S710 to S730 of the method, Figure 8 Steps S810 to S840 of the method, Figure 9In step S910 of the method, an image to be processed and a depth map to be processed are obtained, wherein the image to be processed includes a plurality of pixels and the depth map to be processed is used to indicate the depth information of the pixels; target detection is performed on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects; semantic segmentation is performed on the target detection area to obtain a target pixel area; the target pixel area is mapped to the depth map to be processed to obtain a mapping image; and segmentation processing is performed on the mapping image to obtain a segmented image of each spherical or quasi-spherical object. Based on this, by performing target detection on the image to be processed to obtain a target detection area including a plurality of spherical or quasi-spherical objects, semantic segmentation is performed on the target detection area, the obtained target pixel area is mapped to the depth map to be processed to obtain a mapping image, and then segmentation processing is performed on the mapping image to obtain a segmented image of each spherical or quasi-spherical object, it is achieved that in a complex working environment, spherical or quasi-spherical objects that are mutually occluded or overlapped in the image can be effectively distinguished.

[0130] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0131] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for segmenting an image of a spherical or quasi-spherical object, characterized in that: include: Acquire an image to be processed and a depth map to be processed, wherein the image to be processed includes a plurality of pixels, and the depth map to be processed is used to indicate depth information of the pixels; Performing target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects; Performing semantic segmentation on the target detection area to obtain a target pixel area; Mapping the target pixel area to the depth map to be processed to obtain a mapped image; Determining a histogram based on the pixels of the mapped image and the depth information of the pixels; Performing smoothing on the histogram to determine a fitting curve; Determining segmentation information according to the fitting curve, wherein the segmentation information is the number of all peaks of the fitting curve and depth information corresponding to each peak, or the segmentation information is the number of all troughs of the fitting curve and depth information corresponding to each trough; The mapping image is segmented according to the segmentation information to obtain segmented images of the spherical or quasi-spherical objects.

2. The method according to claim 1, characterized in that The step of performing segmentation processing on the mapped image according to the segmentation information to obtain segmented images of the spherical or quasi-spherical objects includes: Based on the number of the peaks, the depth information corresponding to each of the peaks and a preset clustering algorithm, clustering the mapped image to determine a plurality of cluster sets; Based on the cluster set, the mapped image is segmented to obtain segmented images of the spherical or quasi-spherical objects.

3. The method according to claim 1, characterized in that The step of performing segmentation processing on the mapped image according to the segmentation information to obtain segmented images of the spherical or quasi-spherical objects includes: Based on the number of the troughs and the depth information corresponding to each of the troughs, the mapped image is segmented to obtain a segmented image of each of the spherical or quasi-spherical objects.

4. The method according to claim 1, wherein The segmenting process of the mapped image to obtain segmented images of the spherical or quasi-spherical objects includes: Determining a minimum circumscribed rectangle based on the geometric features of the target pixel area; Determine the lengths of two adjacent sides of the minimum circumscribed rectangle, and if the lengths of the two adjacent sides of the minimum circumscribed rectangle are different, connect the midpoints of the two short sides of the minimum circumscribed rectangle to obtain a reference line; Determining depth information of each pixel point on the baseline in the target pixel area based on the depth map to be processed; Determining a depth mutation pixel point from the pixels of the target pixel area according to the depth information of each pixel point on the baseline in the target pixel area and a preset depth mutation threshold, and determining the depth information of the depth mutation pixel point; The mapped image is segmented according to the depth information of the depth mutation pixel point to obtain segmented images of each of the spherical or quasi-spherical objects.

5. The method according to claim 1, characterized in that The segmenting process of the mapped image to obtain segmented images of the spherical or quasi-spherical objects includes: Segmenting the mapped image to obtain image subintervals corresponding to the spherical or quasi-spherical objects; Based on a preset color palette, assigning values ​​to each of the image subintervals so that the color values ​​of pixels in any two of the image subintervals are different; The segmented images of the spherical or quasi-spherical objects are determined according to all the image subintervals after the value assignment process.

6. The method according to claim 1, characterized in that The performing semantic segmentation on the target detection area to obtain a target pixel area includes: Determine the red channel value, green channel value, and blue channel value of each pixel in the target detection area; Determine the color difference of each pixel in the target detection area according to the red channel value, the green channel value and the blue channel data; Screening each pixel in the target detection area according to a preset difference threshold and the color difference, and determining a target pixel from the pixels in the target detection area; A target pixel area is obtained according to the target pixel point.

7. The method according to claim 1, characterized in that The performing semantic segmentation on the target detection area to obtain a target pixel area includes: Inputting the target detection area into a trained semantic segmentation model for spherical or quasi-spherical objects to obtain a target pixel area; The semantic segmentation model for spherical or quasi-spherical objects is trained by the following steps: Obtaining a spherical or quasi-spherical object training image and a spherical or quasi-spherical object training label image; Inputting the spherical or quasi-spherical object training image into a preset semantic segmentation model to obtain a predicted area; Determining a cross entropy loss function based on the spherical or quasi-spherical object training label image and the prediction area; Based on the cross entropy loss function, the model parameters of the semantic segmentation model are updated to obtain a semantic segmentation model of spherical or quasi-spherical objects.

8. A device for segmenting images of spherical or quasi-spherical objects, characterized in that: include: an acquisition unit, configured to acquire an image to be processed and a depth map to be processed, wherein the image to be processed includes a plurality of pixels, and the depth map to be processed is used to indicate depth information of the pixels; a detection unit, configured to perform target detection on the image to be processed to obtain a target detection area, wherein the target detection area includes a plurality of spherical or quasi-spherical objects; A first segmentation unit is used to perform semantic segmentation on the target detection area to obtain a target pixel area; a depth image determining unit, configured to map the target pixel area to the depth map to be processed to obtain a mapped image; A second segmentation unit is configured to determine a histogram based on the pixels of the mapped image and the depth information of the pixels; smooth the histogram to determine a fitting curve; determine segmentation information based on the fitting curve, wherein the segmentation information is the number of all peaks of the fitting curve and the depth information corresponding to each peak, or the segmentation information is the number of all troughs of the fitting curve and the depth information corresponding to each trough; and segment the mapped image according to the segmentation information to obtain segmented images of each of the spherical or quasi-spherical objects.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for image segmentation of a spherical or quasi-spherical object as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer-executable instructions are used to execute the method for segmenting a spherical or quasi-spherical object image according to any one of claims 1 to 7.

Images