Image segmentation method and device and electronic equipment
By combining the two-step segmentation strategy of CNN sub-model and region growth sub-model, the problem of low accuracy of complex image segmentation is solved, and accurate identification and segmentation of objects of different shapes and textures is achieved, and the efficiency and accuracy of image segmentation are improved. .
Patent Information
- Application Number
- CN202411554034.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-11-04
AI Technical Summary
The existing image segmentation technology has low accuracy when processing complex images, especially when light and shadow disturbances, making it difficult to accurately identify objects of different shapes, sizes and textures, resulting in increased segmentation difficulty.
Using an image segmentation method combining CNN sub-model and regional growth sub-model, image segmentation is performed through a two-step segmentation strategy (first segmentation and second segmentation). The first segmentation uses the CNN sub-model for preliminary segmentation, and the second segmentation uses regional growth. The sub-model refines the segmentation results and optimizes the seed point selection and regional growth direction by calculating the similarity and grayscale values of the seed point and adjacent pixel points.
It improves the accuracy and efficiency of image segmentation, can more accurately identify different areas in complex images, reduce missegment, and improves the calculation efficiency and the fineness of segmentation results.
Smart Images

Figure CN119067996A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image segmentation, and in particular to image segmentation methods, devices, and electronic devices. Background Art
[0002] Image segmentation is a crucial technology in today's digital image processing. It aims to divide an image into multiple meaningful regions or objects to facilitate subsequent image analysis, understanding, and recognition. With the rapid development of computer vision and deep learning technologies, image segmentation technology has been widely used in various fields such as medical diagnosis, remote sensing image processing, autonomous driving, and security monitoring.
[0003] The Chinese invention patent, with a publication date of May 12, 2023 and application publication number CN116109652A, is titled "Image segmentation method, system, and device based on an improved region growing algorithm." The patent uses a target detection algorithm to identify the tower bolt region, then determines the threshold range for region growth. The seed point is then input into the region growing algorithm to obtain a region growth map. Edge detection is then performed on the tower image and the region growth map to obtain the image segmentation result. However, the content of real-world images is often complex and varied, containing objects of different shapes, sizes, and textures. Furthermore, objects in images acquired at different times are affected by interference factors such as lighting and shadows, which increases the difficulty of segmentation and reduces the accuracy of image segmentation. Summary of the Invention
[0004] In order to improve the accuracy of image segmentation, the present application provides an image segmentation method, apparatus, and electronic device.
[0005] In the first aspect, the present application provides an image segmentation method, which adopts the following technical solutions: The image segmentation method comprises the following steps: First acquisition: acquire the image to be segmented; Modeling: establishing an image segmentation model, wherein the image segmentation model includes a CNN sub-model and a region growing sub-model; Image segmentation: including first segmentation, setting seed points and second segmentation; First segmentation: Use the CNN sub-model to segment the image to be segmented and obtain n segments; Set the seed point: take an edge pixel of the i-th segmentation block as the seed point; Second segmentation: The region growing sub-model uses the preset growth criterion to process the j-th pixel point adjacent to the seed point to obtain the processing result; First judgment: Based on the processing results, determine whether the j-th pixel point can be merged into the i-th segmentation block. If so, perform the incremental step; if not, perform the iterative step; Add: add the j-th pixel to the i-th segmentation block; Iteration: includes the first traversal and the second traversal; First traversal: take the j+1th pixel as the new jth pixel and perform the second segmentation step until the first stop condition is met; Second traversal: take the (i+1)th segmentation block as the new (i)th segmentation block and perform the second segmentation step until the second stop condition is met.
[0006] By adopting the above technical solution, the image segmentation model of the present application includes: a CNN sub-model and a region growing sub-model. The CNN sub-model is based on a deep learning algorithm and can learn the feature representation in the image. It is suitable for different image segmentation tasks. Even when segmenting complex images, it can quickly identify different regions in the image, thereby improving the generalization ability of the image segmentation model. The present application adopts a two-step segmentation strategy (first segmentation and second segmentation). The first segmentation uses the CNN sub-model for preliminary segmentation, which can quickly obtain a rough segmentation area and provide a basis for subsequent region growth. The second segmentation uses the region growing sub-model to refine the segmentation result, which helps to deal with the ambiguity and irregularity of the segmentation boundary. Before the second segmentation step, an edge pixel point of the i-th segmentation block is selected as the seed point, which makes it easier to distinguish different regions and increases the accuracy of the segmentation of the current segmentation block. After that, the segmentation of the entire image is completed by setting iterative steps.
[0007] Optionally, after performing the first acquisition step and before performing the modeling step, the method further includes: The third segmentation: along the length or width direction of the image to be segmented, the image to be segmented is divided into m sub-images; Set pixel edge: locate the edge between the xth sub-image and the x+1th sub-image, record the edge of the xth sub-image as the first pixel edge, and record the edge of the x+1th sub-image as the second pixel edge; First calculation: calculate the difference between the grayscale value of the y-th pixel and the grayscale value of the y+1-th pixel in the first pixel edge, and record it as the first data; Second judgment: judging whether the first data is greater than a preset grayscale value threshold, if so, executing the marking step; if not, executing the second calculation step; Second calculation: Calculate the difference between the grayscale value of the y-th pixel and the grayscale value of the y+1-th pixel in the second pixel edge, and record it as the second data; Third judgment: judging whether the second data is greater than a preset gray value threshold, if so, executing the marking step; if not, executing the third traversal step; Mark: mark the yth pixel or the y+1th pixel as the first pixel; Third traversal: Use the x+1th sub-image as the new xth sub-image and execute the step of setting pixel edges until the third stopping condition is met.
[0008] By adopting the above technical solution, the present application divides the image to be segmented into m sub-images, which helps to simplify subsequent processing. In particular, when processing large images, the image is divided into smaller parts (sub-images). Then, by calculating the grayscale value difference between the edge pixels of adjacent sub-images, the edge or feature change area in the sub-image can be identified. The change area is often the key point of image segmentation and helps to distinguish different image parts. The present application converts the search for seed points in the entire image into searching for seed points in the pixel edge of each sub-image, which reduces computing resources and improves computing efficiency. By setting a grayscale value threshold, the present application can screen out pixel points with significant grayscale value changes. The pixel points with significant changes are likely to be edge points or feature points of a certain area in the image. By marking the pixel points with significant grayscale value changes, a prompt is obtained in the image to be segmented that the pixel point is most likely the edge point of the segmentation block, which helps to quickly locate the edge of the segmentation block to be segmented and improve the speed of image segmentation.
[0009] Optionally, in the step of setting the seed point, the first pixel point is used to replace the edge pixel point.
[0010] By adopting the above technical solution, the present application uses the first pixel point (i.e., the previously marked edge or feature point) instead of the original edge pixel point in the step of setting the seed point, which helps to more accurately guide the region growing process, reduce the possibility of mis-segmentation, and improve segmentation efficiency. Since the first pixel point more accurately locates the edge or feature point, the region growing will converge to the correct segmentation result more quickly, and using the first pixel point as the seed point can optimize the region growing process. The region growing algorithm usually relies on the selection of seed points and the growth criterion to gradually expand the segmented area. Using more accurate seed points can guide the algorithm to more quickly identify different image parts and reduce unnecessary calculations. And because the selection of seed points is more accurate, the final image segmentation result will be more refined and accurate.
[0011] Optionally, the image segmentation step further includes: First acquisition: obtain the pixel points adjacent to the seed point in the i-th segmentation block, recorded as the first pixel point set, and obtain the pixel points adjacent to the seed point in the i+1-th segmentation block, recorded as the second pixel point set; The third calculation: calculate the similarity between the seed point and all the pixels in the first pixel set, and calculate the sum of all similarities as the first similarity; Fourth calculation: calculating the similarity between the seed point and all the pixels in the second pixel set, and calculating the sum of all similarities as the second similarity; Fourth judgment: judging whether the first similarity is greater than the second similarity, if so, executing the first processing step; if not, executing the second processing step; First processing: the region growing sub-model processes the pixels in the second pixel set using a preset growth criterion; Second processing: the region growing sub-model processes the pixels in the first pixel set using a preset growth criterion.
[0012] By adopting the above technical solution, the present application can quantify the degree of association between the seed point and the surrounding pixels by calculating the total similarity (first similarity and second similarity) between the seed point and the pixels in the adjacent pixel set. The present application determines the direction of region growth based on the comparison results of the two similarities: if the first similarity is greater than the second similarity, the second pixel set is processed first; otherwise, the first pixel set is processed, which helps to ensure that the region growth process proceeds in the opposite direction of the most similar pixel points. In other words, if the seed point is located at the edge of the segmentation block, if the seed point is more similar to the characteristics of the current pixel set, it proves that the seed point belongs to the segmentation block. If the segmentation block is to be segmented in the image to be segmented, region growth should be performed in another pixel set to obtain the edge pixels of the segmentation block. This step reduces the calculation content of the region growing algorithm and improves calculation efficiency.
[0013] Optionally, after performing the fourth calculation step and before performing the fourth judgment step, the method further includes: Sorting: sorting all pixels in the first pixel set and / or the second pixel set according to the order of similarity from large to small to obtain a first pixel sequence and / or a second pixel sequence; Deletion: Delete pixels whose similarity is less than a preset similarity threshold to obtain a new first pixel set and / or second pixel set.
[0014] By adopting the above technical solution, by sorting the pixels in the first pixel set and the second pixel set according to the similarity, the pixels that are most similar to the seed point are given priority during the processing, which helps to reduce unnecessary calculations, because once the region growing process starts, the pixels that are highly similar to the seed point are more likely to belong to the same region. By deleting pixels whose similarity is lower than the preset threshold, the amount of calculation for subsequent processing can be further reduced. Since pixels with low similarity in the calculated similarity are likely not to belong to the same region as the seed point, removing them from the processing list can improve overall efficiency. By giving priority to pixels with high similarity and deleting pixels with low similarity, the region growing process is carried out more accurately in the correct direction, which helps to reduce the situation where pixels that do not belong to the same region are mistakenly classified into the same segmented region.
[0015] Optionally, in the first processing and / or second processing steps, the region growing sub-model is based on the first pixel point sequence and / or the second pixel point sequence, and uses a preset growth criterion to process the pixels in the first pixel point set and / or the second pixel point set.
[0016] By adopting the above technical solution, the pixel points in the deleted first pixel point set and the second pixel point set are processed according to the order of the pixel points in the first pixel point sequence and / or the second pixel point sequence through the region growing sub-model. During the processing, once a pixel point in the first pixel point sequence and / or the second pixel point sequence no longer meets the growth criterion, it is no longer necessary to process the pixel points after the pixel point, thereby reducing the amount of calculation and improving the calculation speed and the speed of image segmentation.
[0017] Optionally, the image segmentation model further includes a BI-LSTM sub-model, and after performing the modeling step and before performing the image segmentation step, further includes: Second acquisition: collecting historical images and segmentation data of historical images to obtain k segments of the historical images; First determination: determine the center pixel point of the p-th segmentation block in the historical image, recorded as the third pixel point; Second determination: obtaining q segments adjacent to the p-th segment in the historical image, and determining the center pixel of the q segments, which is recorded as the fourth pixel; Third acquisition: obtaining the pixel point between the third pixel point and the fourth pixel point, which is recorded as the fifth pixel point; Fourth acquisition: obtaining the grayscale values of all fifth pixels; Model training: The grayscale value of the fifth pixel is used to train the BI-LSTM sub-model to obtain the trained BI-LSTM sub-model.
[0018] By adopting the above technical solution, by collecting the grayscale values of the pixel points (fifth pixel point) between the third pixel point (the center pixel point of the p-th segmentation block in the historical image) and the fourth pixel point (the center pixel point of the q segments adjacent to the p-th segmentation block), and using these grayscale values to train the BI-LSTM sub-model, the BI-LSTM sub-model can learn the spatial relationship and grayscale change pattern between the center pixel points of the pixel blocks in the image.
[0019] Optionally, after performing the first segmentation step and before performing the seed point setting step, the method further includes: Fifth acquisition: Acquire the center pixel points of the i-th segmentation block and the i+1-th segmentation block; Prediction: Input the center pixels of the i-th segmentation block and the i+1-th segmentation block into the trained BI-LSTM sub-model, and predict the grayscale value of the pixel between the two center pixels, which is recorded as the first grayscale value; Fifth calculation: calculate the difference between adjacent first grayscale values, recorded as the first difference; Fifth judgment: judging whether the first difference is greater than a preset difference threshold, if so, executing the replacing step; if not, executing the setting seed point step; Replace: Get the two pixels corresponding to the first difference and use one of the pixels as the new seed point.
[0020] By adopting the above technical solution, this step takes advantage of the powerful modeling ability of BI-LSTM on sequence data. The trained BI-LSTM sub-model predicts the grayscale value of the pixel points between the center pixels of two adjacent segmentation blocks, which can capture the potential relationship between the pixels and thus more accurately predict the grayscale value. For grayscale images, the pixels inside the segmentation blocks generally have grayscale similarity, while there is generally grayscale discontinuity at the boundaries of the segmentation blocks. Therefore, this application calculates the difference in the predicted grayscale values of adjacent pixels and compares it with the preset difference threshold. If the first difference is greater than the preset difference threshold, it indicates that the grayscale is discontinuous here and there is an edge point. One of the pixel points is used as a seed point and the second segmentation step is continued; otherwise, a seed point is set and the second segmentation step is performed. This step helps to quickly locate the seed point, improve computational efficiency, and optimize the segmentation boundary.
[0021] In a second aspect, the present invention provides an image segmentation device, which adopts the following technical solution: An image segmentation device, comprising: A first acquisition module, used for acquiring images to be segmented; A modeling module is used to establish an image segmentation model, wherein the image segmentation model includes: a CNN sub-model and a region growing sub-model; An image segmentation module, which is in communication with the first acquisition module and the modeling module, comprises a first segmentation unit, a seed point setting unit, and a second segmentation unit; The first segmentation unit is used to segment the image to be segmented using the CNN sub-model to obtain n segmentation blocks; Set a seed point unit to use an edge pixel point of the i-th segment block as a seed point; The second segmentation unit is used for the region growth sub-model to process the j-th pixel point adjacent to the seed point using a preset growth criterion to obtain a processing result; A first judgment module is connected to the image segmentation module for judging whether the j-th pixel point can be merged into the i-th segmentation block based on the processing result; An adding module is used to add the j-th pixel point to the i-th segmentation block; An iteration module, connected to the image segmentation module for communication, comprising a first traversal unit and a second traversal unit; The first traversal unit is used to take the j+1th pixel point as the new jth pixel point; The second traversal unit is used to take the (i+1)th segmentation block as the new (i)th segmentation block.
[0022] By adopting the above-mentioned technical solution, the present device achieves preliminary segmentation and further refinement of the image by combining the CNN sub-model and the region growing sub-model. The first segmentation unit uses the CNN sub-model to quickly identify different regions (segment blocks) in the image, while the second segmentation unit refines the segmentation results using the region growing sub-model. The first judgment module and the addition module ensure the accuracy and continuity of the segmentation results, while the iteration module implements an iterative segmentation process by traversing all pixels and segment blocks. This application combines deep learning methods with region growing strategies to improve the accuracy and efficiency of image segmentation, while reducing reliance on manual annotation and improving the generalization of the image segmentation model.
[0023] In a third aspect, the present invention provides an electronic device, which adopts the following technical solution: An electronic device includes: a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and the program or instruction implements the method when executed by the processor.
[0024] In summary, this application includes at least one of the following beneficial technical effects: 1. The image segmentation model of the present application includes: a CNN sub-model and a region growing sub-model. The CNN sub-model is based on a deep learning algorithm and can learn feature representations in an image. It is applicable to different image segmentation tasks. Even when segmenting complex images, it can quickly identify different regions in the image, thereby improving the generalization ability of the image segmentation model. The present application adopts a two-step segmentation strategy (first segmentation and second segmentation). The first segmentation uses the CNN sub-model for preliminary segmentation, which can quickly obtain a rough segmentation area and provide a basis for subsequent region growth. The second segmentation uses the region growing sub-model to refine the segmentation results, which helps to deal with the ambiguity and irregularity of the segmentation boundary. Before the second segmentation step, an edge pixel point of the i-th segmentation block is selected as the seed point, which makes it easier to distinguish different regions and increases the accuracy of the segmentation of the current segmentation block. After that, the segmentation of the entire image is completed by setting iterative steps.
[0025] 2. This application quantifies the degree of association between a seed point and surrounding pixels by calculating the total similarity (first similarity and second similarity) between the seed point and the pixels in the adjacent pixel set. This application determines the direction of region growth based on the comparison of the two similarities: if the first similarity is greater than the second similarity, the second pixel set is prioritized; otherwise, the first pixel set is processed. This helps ensure that the region growing process proceeds in the opposite direction of the most similar pixel points. In other words, if the seed point is located at the edge of the segmented block, if the seed point has more similar features than the current pixel set, it proves that the seed point belongs to the segmented block. If the segmented block is to be segmented in the image to be segmented, region growing should be performed in another pixel set to obtain the edge pixels of the segmented block. This step reduces the computational complexity of the region growing algorithm and improves computational efficiency.
[0026] 3. This step utilizes the powerful modeling ability of BI-LSTM on sequence data. The trained BI-LSTM sub-model predicts the grayscale value of the pixel points between the center pixels of two adjacent segmentation blocks, which can capture the potential relationship between the pixels and thus more accurately predict the grayscale value. For grayscale images, the pixels inside the segmentation blocks generally have grayscale similarity, while there is generally grayscale discontinuity at the boundaries of the segmentation blocks. Therefore, this application calculates the difference in the predicted grayscale values of adjacent pixels and compares it with the preset difference threshold. If the first difference is greater than the preset difference threshold, it indicates that the grayscale is discontinuous here and there is an edge point. One of the pixels is used as a seed point and the second segmentation step is continued; otherwise, a seed point is set and the second segmentation step is performed. This step helps to quickly locate the seed point, improve computational efficiency, and optimize the segmentation boundary. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a flow chart of Example 1 of the present application; Figure 2 This is a flow chart from S1 first acquisition to S2 modeling in Example 2 of the present application; Figure 3 is a schematic diagram of equally dividing the image to be segmented in Example 2 of the present application; Figure 4 is a schematic diagram of setting pixel edges in S72 of Example 2 of the present application; Figure 5 is a flow chart of Example 3 of the present application; Figure 6 This is a flowchart from S2 modeling to S3 image segmentation in Example 4 of the present application; Figure 7 This is a flowchart from S31 first segmentation to S32 setting seed points in Example 4 of the present application.
[0028] Reference numerals: 1 - first pixel side; 2 - second pixel side. DETAILED DESCRIPTION
[0029] The following combination Figures 1 to 7 This application is described in further detail.
[0030] Example 1: This example discloses an image segmentation method, referring to Figure 1 The method includes: S1 first acquisition, S2 modeling, S3 image segmentation, S4 first judgment, S5 addition and S6 iteration. First, the image to be segmented is acquired, then an image segmentation model is established, and then the image to be segmented is segmented and further processed using the image segmentation model. Then, based on the processing result, it is judged whether the current pixel point can be added to the segmentation block. Then, segmentation and judgment are iteratively performed to achieve segmentation of the entire image. This embodiment includes the following steps: S1 is the first acquisition, which involves acquiring the image to be segmented.
[0031] S2 modeling, establishing an image segmentation model, wherein the image segmentation model includes: a CNN sub-model and a region growing sub-model.
[0032] The CNN sub-model uses deep learning technology. After being trained with a large number of historical images and corresponding historical image segmentation data, it can learn the feature representation of pixels in the image and acquire the ability to segment the image. The trained CNN sub-model is then used to perform preliminary segmentation on the image to be segmented.
[0033] Region growing sub-model: Based on the preliminary segmentation results (provided by the CNN sub-model), the segmentation results are refined using a preset growing criterion. The region growing sub-model relies on local information in the image, such as color and texture, to decide which pixels should be merged into the same segmentation block.
[0034] The preset growth criterion may be one of an adjacent region grayscale difference criterion, a regional grayscale distribution statistics criterion, and a color criterion.
[0035] Adjacent region grayscale difference criterion: Region growing is performed based on the similarity of pixel grayscale values. By setting a grayscale difference threshold, if the grayscale difference between adjacent pixels is less than the threshold, they are considered similar and can be merged into the same region.
[0036] Regional grayscale distribution statistical criteria: Calculate the grayscale distribution statistics (such as mean, variance, etc.) of the pixel points that have been determined as the target area, and use these statistics to evaluate whether the adjacent pixels should be merged into the target area.
[0037] Color criterion: If the color values of adjacent pixels are similar within a certain range, they can be merged into the same area.
[0038] S3 image segmentation includes S31 first segmentation, S32 setting seed points and S33 second segmentation.
[0039] S31 is the first segmentation, which uses the CNN sub-model to segment the image to be segmented to obtain n segmentation blocks.
[0040] S32 sets a seed point, randomly selecting an edge pixel point of the i-th segmentation block and using it as the seed point.
[0041] S33 Second segmentation: The region growth sub-model processes the j-th pixel point adjacent to the seed point according to a preset growth criterion (such as a color criterion, an adjacent region grayscale difference criterion, etc.) and outputs the processing result.
[0042] S4 is the first judgment. Based on the processing results, it is determined whether the j-th pixel point can be merged into the i-th segmentation block. If so, it means that the j-th pixel point and the seed point meet the growth criteria, and S5 is executed to increase; if not, it means that the j-th pixel point and the seed point do not meet the growth criteria, and S6 is executed to iterate.
[0043] S5 increases, adding the j-th pixel point to the i-th segmentation block.
[0044] S6 iteration includes S61 first traversal and S62 second traversal.
[0045] S61 is the first traversal, taking the j+1th pixel point as the new jth pixel point, and executing the second segmentation of S33 until the first stopping condition is met. The first stopping condition is: the jth pixel point added in S5 is the last pixel point adjacent to the seed point, that is, the jth pixel point is the last pixel point to be processed.
[0046] This step is to traverse and process the pixel points in each segmentation block (each image block after image segmentation is called a segmentation block. For example, in an image, including the background and the target object, after the image is segmented, two segmentation blocks of the background and the target object are obtained). By continuously taking the next adjacent pixel point as the new processing object, and repeating the second segmentation of S33 and the first judgment of S4, until the first stopping condition is met (that is, all pixels adjacent to the seed point are processed).
[0047] S62 is a second traversal, taking the (i+1)th segment block as a new (i)th segment block, and executing S33 the second segmentation until a second stopping condition is met, wherein the second stopping condition is: i+1>n.
[0048] This step is to traverse and process all the segmented blocks, by continuously taking the next segmented block as a new processing object and repeatedly executing the second segmentation in S33 until the second stop condition is met (ie, all the segmented blocks are processed).
[0049] This embodiment combines the CNN sub-model and the region growing sub-model to achieve preliminary segmentation and further refinement of the image. By iteratively processing the pixels within each segmentation block and traversing all segmentation blocks, the segmentation results can be gradually optimized to improve the accuracy and efficiency of segmentation.
[0050] Example 2: Reference Figure 2 The difference between this embodiment and embodiment 1 is that after executing S1 first acquisition and before executing S2 modeling, the following is further included: S71: a third segmentation, dividing the image to be segmented into m sub-images along the length direction or the width direction of the image to be segmented.
[0051] S72 sets the pixel edge. For each pair of adjacent sub-images (the x-th sub-image and the x+1-th sub-image), find the boundary between them, and record all adjacent boundary pixel points between the x-th sub-image and the x+1-th sub-image as the first pixel edge, and record all adjacent boundary pixel points between the x+1-th sub-image and the x-th sub-image as the second pixel edge.
[0052] S73: First calculation: for the y-th pixel point in the first pixel side, calculate the difference between its grayscale value and the grayscale value of the y+1-th pixel point to obtain first data.
[0053] S74 is the second judgment, which determines whether the first data is greater than the preset grayscale value threshold. If so, it means that the y-th pixel point and the y+1-th pixel point are significantly different, corresponding to the edge feature in the image, and S77 marking is executed; if not, S75 is executed for the second calculation.
[0054] S75: second calculation, calculating the difference between the grayscale value of the yth pixel point and the grayscale value of the y+1th pixel point in the second pixel edge, and recording it as the second data.
[0055] S76 is the third judgment, judging whether the second data is greater than the preset gray value threshold. If so, execute S77 marking; if not, execute S78 third traversal.
[0056] S77 Marking: In the image to be segmented, mark the yth pixel or the y+1th pixel as the first pixel.
[0057] S78 is a third traversal, where the x+1th sub-image is used as a new xth sub-image, and S72 is executed to set pixel edges until a third stopping condition is satisfied, and then S2 is executed to perform modeling. The third stopping condition is: x+1>m.
[0058] In setting the seed point in S32 , the first pixel point is used to replace the edge pixel point.
[0059] For example: Refer to Figure 3 and Figure 4 ,Suppose there is a grayscale image containing multiple objects.,The goal is to identify and mark certain boundary points between objects through the,above steps so that these boundary points can be used in subsequent image,segmentation tasks.
[0060] S1 first acquires an image to be segmented using a camera or image scanning device, and the image contains multiple different objects.
[0061] S71: third segmentation: converting the acquired image to be segmented into a grayscale image, and evenly segmenting it into four sub-images (m=4) along its length direction.
[0062] S72 sets the pixel edge. For the first pair of adjacent sub-images (the first sub-image and the second sub-image), locate the boundary between them, and record the boundary pixels of the first sub-image as the first pixel edge and the boundary pixels of the second sub-image as the second pixel edge.
[0063] S73 is a first calculation, traversing each pixel point in the first pixel edge and calculating the difference between its grayscale value and the grayscale value of the next pixel point.
[0064] In this embodiment, the grayscale value of the first pixel is 120, and the grayscale value of the next pixel is 150, so the first data is 30.
[0065] S74 is the second judgment. A grayscale value threshold of 20 is preset. Since the first data is greater than the threshold, it is considered that the grayscale value difference of the pixel point is significant and may be the boundary between the segmented blocks.
[0066] S77 marking: marking the pixel point with a significant grayscale value difference in the second judgment of S74 as the first pixel point in the image to be segmented, and recording its position.
[0067] S75 is a second calculation, and S76 is a third judgment. If the grayscale value difference of the first pixel side is not significant (ie, less than the threshold), the grayscale value difference of adjacent pixels in the second pixel side is calculated and similar judgments are performed.
[0068] In step S78, the third traversal is performed. The currently processed sub-image pair is updated to the next pair of adjacent sub-images (the second and third sub-images), and steps S72 to S77 are repeated. This process continues until all sub-image pairs have been processed (i.e., the third stopping condition x+1>m is satisfied).
[0069] After the above steps, a series of marked first pixel points are obtained, where the first pixel points correspond to boundary pixel points between segmented blocks on the edge of the image pixel.
[0070] This embodiment divides the image to be processed, obtains the pixel edges of the two sub-images at the division point, obtains the pixel points whose grayscale value changes exceed a preset threshold within the pixel edge and marks them, and replaces the edge pixel points with the marked pixel points for subsequent steps, which helps to quickly locate the edge points between the divided blocks and improve the accuracy and efficiency of image segmentation.
[0071] Example 3: Reference Figure 5 The difference between this embodiment and embodiment 1 is that the S3 image segmentation further includes the following steps: S81 is the first acquisition, which is to acquire the pixel points adjacent to the seed point in the i-th segmentation block, recorded as the first pixel point set, and to acquire the pixel points adjacent to the seed point in the i+1-th segmentation block, recorded as the second pixel point set.
[0072] S82 is a third calculation, which is to calculate the similarity between the seed point and all the pixel points in the first pixel point set, and calculate the sum of all the similarities as the first similarity.
[0073] S83 is a fourth calculation, which is to calculate the similarity between the seed point and all the pixel points in the second pixel point set, and calculate the sum of all the similarities as the second similarity.
[0074] In the third calculation in S82 and the fourth calculation in S83, the similarity can be calculated based on multiple dimensions such as color, brightness, texture, etc.
[0075] Take color as an example: set the color of the seed point to , the color of the i-th pixel in the first pixel set is , and there are M pixels in the first pixel set, and the color of the j-th pixel in the second pixel set is , and there are N pixels in the second pixel set, then the model for calculating similarity based on Euclidean distance is as follows: ; .
[0076] is the value of the first similarity in this example; is the value of the second similarity in this example.
[0077] Taking texture as an example: First, the local binary pattern (LBP) algorithm is used to extract texture features for each pixel. Each pixel in the first pixel set is traversed, and a 3×3 neighborhood is defined around it. The pixel values of the eight pixels at the edge of the neighborhood (referred to as the neighborhood pixel values) are compared with the pixel value of the center pixel (in this example, the center pixel is the pixel at the center of the 3×3 neighborhood) (referred to as the center pixel value). If the neighborhood pixel value is greater than or equal to the center pixel value, the position is marked as 1; otherwise, it is marked as 0. The eight binary values in the neighborhood are arranged clockwise or counterclockwise, with the upper-right binary value as the first binary digit, to form an 8-bit binary number. This number is then converted to decimal to obtain the LBP feature value of the pixel. Similarly, the LBP feature value of each pixel in the second pixel set and the LBP feature value of the seed point are calculated.
[0078] Create a matrix with the same total size as the two segmented blocks to store the LBP eigenvalues of the first pixel set, the seed point, and the second pixel set. Then, traverse the LBP eigenvalues of each pixel in the first and second pixel sets and store them in the corresponding matrix positions. Simultaneously, store the LBP eigenvalues of the seed point in the corresponding matrix positions. Calculate the LBP eigenvector for each pixel in the first pixel set. Similarly, calculate the LBP eigenvector for each pixel in the second pixel set, as well as the LBP eigenvector corresponding to the seed point.
[0079] Based on the above LBP feature vector, the similarity between the seed point and the pixels in the first pixel set or the second pixel set is calculated: ; ; is the value of the first similarity in this example; is the value of the second similarity in this example; is the i-th element in the feature vector of the seed point; is the i-th element in the feature vector of the first pixel set; is the jth element in the feature vector of the seed point; is the jth element in the feature vector of the second pixel set; is the dimension of the feature vector of the seed point or the first pixel point set; is the dimension of the feature vector of the seed point or the second pixel point set; because in this example the feature vectors of the seed point, the first pixel point set, and the second pixel point set are all extracted from the same matrix, .
[0080] In other embodiments, the first similarity and the second similarity are calculated from multiple perspectives such as color, brightness, and texture. For example, the similarity is calculated from two perspectives, color and texture, and different weights are assigned to color and texture to obtain the first similarity and the second similarity. ; ; is the value of the first similarity in this example; is the value of the second similarity in this example; 、 、 and The calculation method is as above; is the weight of the color; is the weight of the texture.
[0081] S84 sorting: in the first pixel set, all the pixels are sorted in descending order according to the similarities calculated in the third calculation of S82 to obtain a first pixel sequence.
[0082] In the second pixel point set, all the pixel points are sorted in descending order according to the similarities calculated in the fourth calculation in S83 to obtain a second pixel point sequence.
[0083] S85 is deletion, in which pixels whose similarity is less than a preset similarity threshold are deleted from the first pixel point set to obtain a new first pixel point set.
[0084] Pixels whose second similarity is less than a preset similarity threshold are deleted from the second pixel point set to obtain a new second pixel point set.
[0085] S86 is a fourth judgment, which is to judge whether the first similarity is greater than the second similarity. If so, the first processing of S87 is executed; if not, the second processing of S88 is executed.
[0086] In step S87, the first processing is to process the pixel points in the new second pixel point set deleted in step S85 by the region growing sub-model based on the second pixel point sequence and using a preset growth criterion.
[0087] In the second processing step S88, the region growing sub-model processes the pixels in the new first pixel point set in the deletion step S85 based on the first pixel point sequence and using a preset growth criterion.
[0088] The first processing in S87 and the second processing in S88 described in this embodiment are sub-steps of the second segmentation in S33.
[0089] This embodiment first obtains the pixel point sets (first and second pixel point sets) adjacent to the seed point in the i-th and i+1-th segmented blocks, calculates the sum of the similarities between the seed point and the pixels in these two sets (first and second similarities), and sorts and deletes the pixels whose similarities are lower than a threshold. Subsequently, based on the comparison of the first similarity and the second similarity, it is determined whether the region growing sub-model will process the second pixel point set (if the first similarity is greater) or the first pixel point set (if the second similarity is greater) according to a preset growth criterion. The sorted pixel point sequence will be referenced during the processing. Since the color, texture and other features in the same segmented block should be the same or similar, through the above steps, it is only necessary to use the region growing sub-module to process the pixels of the segmented block with the least similarity to the pixels where the seed point is located, thereby reducing the amount of calculation and improving the calculation efficiency.
[0090] Example 4: Reference Figure 6 The difference between this embodiment and embodiment 1 is that the image segmentation model further includes a BI-LSTM sub-model, and after executing S2 modeling and before executing S3 image segmentation, it also includes: S21 is the second acquisition step, which involves collecting historical images and their segmentation data to obtain k segments of the historical images. The historical images originate from past experiments, observations, or records, while the segmentation data is obtained by preprocessing or labeling these images, indicating that the images are segmented into different segments. This process yields k segments, which are the result of segmenting the historical images. Each segmentation segment represents a specific region or object in the image.
[0091] S22 first determines the center pixel of the pth segmentation block in the historical image, which is recorded as the third pixel. The center pixel represents the core position of the segmentation block and can be a geometric center, a centroid, or other representative points.
[0092] S23 is a second determination, obtaining q segmented blocks adjacent to the p-th segmented block in the historical image, and determining the central pixel points of the q segmented blocks, which are recorded as the fourth pixel points.
[0093] S24 is a third acquisition step, in which pixels between the third pixel and the fourth pixel are obtained and recorded as fifth pixel points. All fifth pixel points constitute a path or region connecting central pixel points of adjacent segmented blocks.
[0094] S25 fourth acquisition, obtaining the grayscale values of all fifth pixels.
[0095] S26 model training uses the grayscale value of the fifth pixel to train the BI-LSTM sub-model to obtain the trained BI-LSTM sub-model.
[0096] By training the BI-LSTM sub-model, the temporal or spatial dependencies between the grayscale values of the fifth pixel and the potential relationship between these grayscale values and image segmentation or recognition tasks can be learned.
[0097] Reference Figure 7 In other embodiments, after executing S31 for the first segmentation and before executing S32 for setting the seed point, the method further includes: S91 fifth acquisition, acquiring the central pixel points of the i-th segmentation block and the (i+1)-th segmentation block.
[0098] S92 predicts that the central pixel points of the i-th segmentation block and the i+1-th segmentation block are input into the trained BI-LSTM sub-model, and the grayscale value of the pixel point between the two center pixels is predicted, which is recorded as the first grayscale value.
[0099] S93 is a fifth calculation: calculating the difference between adjacent first grayscale values, recorded as the first difference. The purpose of calculating the difference between adjacent grayscale values is to detect changes in grayscale values, which can reflect changes in features such as edges, textures, or object boundaries in the image. By comparing these differences, it is possible to determine whether the grayscale values of the pixels between the two center pixels have changed significantly, thereby locating the boundary between the two segmented blocks.
[0100] S94 is the fifth judgment, which determines whether the first difference is greater than the preset difference threshold. If so, it means that the grayscale value of the pixel point between the two center pixels has changed significantly, which may be due to the existence of edges, textures or object boundaries, and S95 replacement needs to be executed; if not, it means that the grayscale value change is not significant, and S32 needs to be executed to set the seed point.
[0101] S95 is replaced by obtaining two pixel points corresponding to the first difference value, and using one of the pixel points as a new seed point.
[0102] This embodiment identifies features or object boundaries in an image by predicting and analyzing the grayscale values of pixels between the center pixels of adjacent segments, and replaces seed points as needed for further analysis or processing. This embodiment locates seed points based on the changes in the grayscale values of the pixels passed by the line connecting the center pixels of two segments, thereby improving the accuracy and efficiency of subsequent processing.
[0103] Embodiment 5: This embodiment discloses an image segmentation device, comprising: The first acquisition module is used to acquire the image to be segmented.
[0104] The modeling module is used to build an image segmentation model, which includes a CNN sub-model and a region-growing sub-model. The CNN sub-model automatically learns image features and performs preliminary image segmentation based on these features. The region-growing sub-model uses the initial segmentation results of the CNN sub-model to iteratively merge adjacent pixels to form a more accurate segmentation result.
[0105] The image segmentation module is in communication with the first acquisition module and the modeling module, and includes a first segmentation unit, a seed point setting unit and a second segmentation unit.
[0106] The first segmentation unit is used to segment the image to be segmented by using the CNN sub-model to obtain n segmentation blocks.
[0107] A seed point unit is set to use an edge pixel point of the i-th segmentation block as a seed point. This seed point will serve as the starting point for the region growing sub-model processing.
[0108] The second segmentation unit is used for the region growth sub-model to process the j-th pixel point adjacent to the seed point using a preset growth criterion.
[0109] The first judgment module is in communication with the image segmentation module and is configured to judge whether the j-th pixel point can be merged into the i-th segmentation block based on the processing result.
[0110] The adding module is used to add the j-th pixel point to the i-th segmentation block.
[0111] The iteration module is connected to the image segmentation module in communication, and includes a first traversal unit and a second traversal unit.
[0112] The first traversal unit is used to take the j+1th pixel point as the new jth pixel point.
[0113] The second traversal unit is used to take the (i+1)th segmentation block as the new (i)th segmentation block.
[0114] This embodiment acquires the image to be segmented through the first acquisition module, performs preliminary segmentation using the CNN sub-model in the modeling module, and then performs fine segmentation using the region growing sub-model combined with the set seed point and growth criterion. The first judgment module and the addition module decide whether to merge adjacent pixels. The iteration module is responsible for traversing all pixels and segmentation blocks to complete the task of accurate segmentation of the entire image.
[0115] Embodiment 6: This embodiment provides an electronic device, which includes: a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the described method are implemented.
[0116] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.
Claims
1. An image segmentation method, characterized in that: include: First acquisition: acquiring the image to be segmented; Modeling: establishing an image segmentation model, wherein the image segmentation model includes: a CNN sub-model and a region growing sub-model; Image segmentation: including first segmentation, setting seed points and second segmentation; First segmentation: Use the CNN sub-model to segment the image to be segmented and obtain n segmentation blocks; Set the seed point: take an edge pixel of the i-th segment block as the seed point; Second segmentation: The region growth sub-model processes the j-th pixel point adjacent to the seed point using a preset growth criterion to obtain a processing result; First judgment: based on the processing result, judge whether the j-th pixel point can be merged into the i-th segmentation block. If so, perform the increasing step; if not, perform the iterative step; Add: add the jth pixel to the i-th segmentation block; Iteration: includes the first traversal and the second traversal; First traversal: take the j+1th pixel point as the new jth pixel point and perform the second segmentation step until the first stop condition is met; Second traversal: take the i+1th segmentation block as the new ith segmentation block and perform the second segmentation step until the second stop condition is met.
2. The image segmentation method according to claim 1, characterized in that: After executing the first acquisition step and before executing the modeling step, the method further includes: Third segmentation: along the length direction or width direction of the image to be segmented, the image to be segmented is equally divided into m sub-images; Set pixel edge: locate the edge between the xth sub-image and the x+1th sub-image, record the edge of the xth sub-image as the first pixel edge, and record the edge of the x+1th sub-image as the second pixel edge; First calculation: calculate the difference between the grayscale value of the yth pixel point and the grayscale value of the y+1th pixel point in the first pixel edge, and record it as the first data; Second judgment: judging whether the first data is greater than a preset gray value threshold, if so, executing the marking step; if not, executing the second calculation step; Second calculation: calculate the difference between the grayscale value of the yth pixel point and the grayscale value of the y+1th pixel point in the second pixel edge, and record it as the second data; Third judgment: judging whether the second data is greater than a preset gray value threshold, if so, executing the marking step; if not, executing the third traversal step; Mark: mark the yth pixel or the y+1th pixel as the first pixel; Third traversal: take the x+1th sub-image as the new xth sub-image, and execute the step of setting the pixel edge until the third stop condition is met.
3. The image segmentation method according to claim 2, characterized in that: In the step of setting the seed point, the first pixel point is used to replace the edge pixel point.
4. The image segmentation method according to any one of claims 1 to 3, characterized in that: The step of image segmentation further comprises: First acquisition: obtain the pixel points adjacent to the seed point in the i-th segmentation block, recorded as the first pixel point set, and obtain the pixel points adjacent to the seed point in the i+1-th segmentation block, recorded as the second pixel point set; Third calculation: calculating the similarity between the seed point and all the pixels in the first pixel set, and calculating the sum of all similarities as the first similarity; Fourth calculation: calculating the similarity between the seed point and all the pixels in the second pixel point set, and calculating the sum of all the similarities as the second similarity; Fourth judgment: judging whether the first similarity is greater than the second similarity, if so, executing the first processing step; if not, executing the second processing step; First processing: the region growth sub-model processes the pixels in the second pixel point set using a preset growth criterion; Second processing: the region growing sub-model processes the pixels in the first pixel set using a preset growth criterion.
5. The image segmentation method according to claim 4, characterized in that: After executing the fourth calculation step and before executing the fourth judgment step, the method further includes: Sorting: in the first pixel point set and / or the second pixel point set, all the pixels are sorted in descending order according to the similarity to obtain a first pixel point sequence and / or a second pixel point sequence; Deletion: Delete pixels whose similarity is less than a preset similarity threshold to obtain a new first pixel set and / or second pixel set.
6. The image segmentation method according to claim 5, characterized in that: In the first processing and / or second processing steps, the region growing sub-model is based on the first pixel point sequence and / or the second pixel point sequence, and processes the pixels in the first pixel point set and / or the second pixel point set using a preset growth criterion.
7. The image segmentation method according to any one of claims 1 to 3, characterized in that: The image segmentation model also includes a BI-LSTM sub-model, and after executing the modeling step and before executing the image segmentation step, it also includes: Second collection: collecting historical images and segmentation data of historical images to obtain k segmentation blocks after the historical images are segmented; First determination: determine the central pixel point of the pth segmentation block in the historical image, recorded as the third pixel point; Second determination: obtaining q segmentation blocks adjacent to the pth segmentation block in the historical image, and determining the central pixel point of the q segmentation blocks, which is recorded as the fourth pixel point; Third acquisition: acquiring a pixel point between the third pixel point and the fourth pixel point, recorded as the fifth pixel point; Fourth acquisition: obtaining the grayscale values of all fifth pixels; Model training: The grayscale value of the fifth pixel is used to train the BI-LSTM sub-model to obtain the trained BI-LSTM sub-model.
8. The image segmentation method according to claim 7, characterized in that: After performing the first segmentation step and before performing the seed point setting step, the method further includes: Fifth acquisition: acquiring the central pixel points of the i-th segmentation block and the i+1-th segmentation block; Prediction: Input the central pixels of the i-th segmentation block and the i+1-th segmentation block into the trained BI-LSTM sub-model, and predict the grayscale value of the pixel between the two central pixels, which is recorded as the first grayscale value; Fifth calculation: calculating the difference between adjacent first grayscale values, recorded as the first difference; Fifth judgment: judging whether the first difference is greater than a preset difference threshold, if so, executing the replacing step; if not, executing the setting seed point step; Replacement: Get the two pixels corresponding to the first difference and use one of the pixels as the new seed point.
9. An image segmentation device, characterized in that: include: A first acquisition module, used for acquiring an image to be segmented; A modeling module, used to establish an image segmentation model, wherein the image segmentation model includes: a CNN sub-model and a region growing sub-model; An image segmentation module, which is in communication with the first acquisition module and the modeling module, comprises a first segmentation unit, a seed point setting unit and a second segmentation unit; The first segmentation unit is used to segment the image to be segmented by using a CNN sub-model to obtain n segmentation blocks; A seed point unit is set to use an edge pixel point of the i-th segmentation block as a seed point; The second segmentation unit is used for the region growth sub-model to process the j-th pixel point adjacent to the seed point using a preset growth criterion to obtain a processing result; A first judgment module is connected to the image segmentation module for judging whether the j-th pixel point can be merged into the i-th segmentation block based on the processing result; An adding module is used to add the j-th pixel point to the i-th segmentation block; An iteration module, which is in communication connection with the image segmentation module, comprises a first traversal unit and a second traversal unit; The first traversal unit is used to take the j+1th pixel point as the new jth pixel point; The second traversal unit is used to take the i+1th segmentation block as the new ith segmentation block.
10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Method, system and related components for image edge detection
CN109102518A
An intelligent medical image segmentation method based on three-dimensional reconstruction
CN109509199A
Segmentation method and device for target object in image, electronic equipment and storage medium
CN111932547A
Object-oriented remote sensing image segmentation method, device, equipment and medium
CN115239746A
Portrait segmentation method and device, equipment and storage medium
CN116843695A
