Automatic segmentation and fitting method for hand-drawn sketch based on greedy strategy

Abstract

The present application relates to hand-drawn sketch stroke segmentation method technical field, especially to hand-drawn sketch automatic segmentation fitting method based on greedy strategy, the present application includes the following steps: resampling and smoothing filtering to input sketch stroke sequence, then according to the greedy strategy automatic segmentation hand-drawn sketch stroke, in the segmentation process, each iteration starts to read in a data in the segmentation set, the least square method fitting is carried out to the substroke between two adjacent segmentation points and the fitting error is calculated, the segmentation point and the parameter of the substroke that the fitting accuracy meets the requirement are stored, the substroke that the fitting accuracy does not meet the requirement needs to be re-segmented is recorded, and the region to be segmented is reduced.The method of the present application realizes the segmentation fitting of stroke through multiple rounds of iteration segmentation fitting by using the greedy strategy, the algorithm is simple and efficient, and the exhaustive operation needed for finding the global optimal segmentation point is saved;and the segmentation point is selected by using the Euclidean distance, so that the accuracy and efficiency of stroke segmentation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hand-drawn sketch stroke segmentation methods, in particular to a hand-drawn sketch automatic segmentation and fitting method based on a greedy strategy. BACKGROUND

[0002] Hand-drawn sketches are a natural and direct way of thinking for human beings to communicate with each other. With the development of human-computer interaction technology, hand-drawn sketches play a huge role in industrial design, animation design and mechanical engineering design.

[0003] A computer can obtain a hand-drawn sketch through a mouse or a handwriting pen. However, due to different levels of proficiency and different drawing speeds of different users, and different sampling rates of different input devices, a hand-drawn sketch image has randomness and semantics has ambiguity, which brings great difficulty to the computer to realize automatic segmentation of the hand-drawn sketch.

[0004] Currently, common stroke segmentation methods include a curvature-based stroke segmentation method, a stroke speed-based stroke segmentation method and a deep learning-based stroke segmentation method. The curvature-based stroke segmentation method finds sampling points with relatively large curvature in a stroke sequence of a hand-drawn sketch as segmentation points. Sampling errors or user's drawing jitter will greatly affect the calculation of the curvature, and further affect the final segmentation result. The stroke speed-based stroke segmentation method is based on the fact that when a user encounters a turning point during drawing, the stroke speed will obviously decrease. This information is used to find the segmentation points. The user's drawing fluency greatly affects the stroke speed, and further affects the segmentation result. The deep learning-based stroke segmentation method usually establishes a feature model based on sketch semantic information. The disadvantage is that the semantics of sketches drawn by different users is ambiguous, which leads to segmentation failure, and the deep learning model training needs to establish a large amount of data set and takes a long time to calculate.

[0005] Therefore, a hand-drawn sketch automatic segmentation and fitting method based on a greedy strategy is proposed to provide a solution to the above problems. SUMMARY

[0006] The present application aims to provide a hand-drawn sketch automatic segmentation and fitting method based on a greedy strategy to solve the problems raised in the background.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical scheme: comprising the following steps:

[0008] S1: obtaining all points on the sketch strokes, storing the x and y coordinate values, and generating an original stroke point set MP;

[0009] S2: equally interval resampling the original stroke point set to generate a resampling set REMP;

[0010] S3: Smooth filtering the resampled stroke point set to filter out the stroke noise introduced by jitter, and generating a smooth filtered point set FMP;

[0011] S4: Setting an allowed fitting error δ, the start point and end point of the smooth filtered point set form an initial fitting set FIT, an initial re-fitting set CFIT is an empty set, j = 1, k = 1;

[0012] S5: Reading the jth element fit j of the fitting set, denoted as fit j = (s, e); taking the point set between the s-th point and the e-th point in the smooth filtered point set to form a fitting sub-stroke MMP = {mmp i} = {(fmp i .x, fmp i .y)}, s≤i≤e;

[0013] S6: Fitting the sub-stroke to calculate the fitting parameters and the fitting error;

[0014] S7: Judging whether the fitting error meets the requirement, if yes, storing the end point sequence number and the fitting parameters of the current sub-stroke, j = j + 1, and turning to S8; otherwise, storing the start point sequence number and the end point sequence number of the current sub-stroke into the re-fitting set;

[0015] S8: Judging whether all the sub-strokes in this round of iteration have been fitted, if not, turning to S5 to fit the next sub-stroke, otherwise, turning to S9;

[0016] S9: Judging whether the re-fitting set is an empty set, if yes, turning to S14; otherwise, turning to S10;

[0017] S10: Connecting the start point and the end point of the kth sub-stroke in the re-fitting set into a straight line, calculating the Euclidean distance of each point on the sub-stroke to the straight line, and finding the point corresponding to the maximum distance as a new segmentation point according to the greedy strategy;

[0018] S11: The new segmentation point further divides the sub-stroke that needs to be re-fitted into two sub-strokes, and the start point and end point sequence numbers of the two new sub-strokes are added to the fitting set in the next round of iteration, k = k + 1;

[0019] S12: Judging whether k is greater than the number of elements in the re-fitting set, if yes, it means that the sub-stroke whose fitting error does not meet the requirement in the current iteration round has been further segmented according to the greedy strategy, and turning to S13; otherwise, turning to S10 for loop processing;

[0020] S13: Setting the re-fitting set to be an empty set, k = 1, and turning to S5 for a new round of iteration fitting;

[0021] S14: deleting redundant segmentation points;

[0022] S15: repeating the above steps to traverse all adjacent sub-strokes, deleting redundant segmentation points, and obtaining a corrected segmentation point sequence.

[0023] Further, the least square method is used to fit the sub-strokes in S6.

[0024] Further, in S14, when deleting the redundant segmentation points, starting from the first segmented sub-stroke, if two adjacent continuous sub-strokes can be combined into one sub-stroke, it is considered that the two adjacent sub-strokes belong to the same stroke, and the redundant segmentation point connecting the two sub-strokes is deleted.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] (1) The method of the present application realizes the segmentation and fitting of strokes through multiple rounds of iterative segmentation and fitting by using the greedy strategy, and the algorithm is simple and efficient, and the exhaustive operation required for finding the global optimal segmentation point is saved.

[0027] (2) The method of the present application uses the Euclidean distance to select the segmentation point, effectively overcomes the influence of the randomness and semantic ambiguity of the hand-drawn sketch image, and improves the accuracy and efficiency of stroke segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0028] Fig. 1 is the overall flowchart of the present application;

[0029] Fig. 2 is a first example of a hand-drawn sketch of the present application;

[0030] Fig. 3 is a second example of a hand-drawn sketch of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] Please refer to Figs. 1-3 , the automatic segmentation and fitting method of hand-drawn sketches based on the greedy strategy specifically includes the following steps:

[0033] S1: reading a stroke in a hand-drawn sketch, denoted as a set MP = {mp i} = {(mp i .x, mpi .y), i = 1, 2, …, M}, mp i is the i-th element in the set MP, (mp i .x, mp i .y) is the x, y coordinate of the i-th element in the set MP, M≥2 is the number of stroke points.

[0034] S2: Calculate the closed contour length of the stroke;

[0035]

[0036] Resampling interval Δ = loop / M. Linear interpolation resampling with equal interval is performed on MP in sequence to obtain the resampling set REMP = {remp i} = {(remp i .x, remp i .y)}, i = 1, 2, …, M}, remp i is the i-th element in the set REMP, (remp i .x, remp i .y) is the x, y coordinate of the i-th element in the set REMP, M≥2. In the resampling process, the first sample point is unchanged remp1=mp1, and starting from the second sample point i = 2, the distance between sample point i and sample point i-1 is calculated,

[0037]

[0038] When Dis=Δ, remp i =mp i ;

[0039] When Dis≠Δ, remp i .x=rem i-1 .p i .x+Δ×(mp i-1 .x-remp i .x) / Dis, remp i-1 .y=rem i .p i-1 .y+Δ×(mp .y-remp .y) / Dis.

[0040] S3: Select (1, 3, 1) template for sliding weighted mean filtering to remove noise while retaining the original hand-drawn stroke features. The filtered sampling set FMP = {fmp i} = {(fmp i .x, fmp i .y)}, i = 1, 2, …, M}, fmp i is the i-th element in the set FMP, (fmp i.x, gmp i .y) is the x, y coordinate of the i-th element in the set FMP, M≥2. The first, second and last sampling points are unchanged, fmp1 = remp1, fmp2 = remp2, fmp M = remp M ;

[0041] The rest of the sampling points are calculated according to the formula fmp i = (remp i-1 + 3 x remp i + remp i+1 ) / 5.

[0042] S4: Set the fitting error threshold as δ, such as 0.01; Set the fitting set FIT = {fit j}, fit j is the j-th element in the fitting set FIT, j is a natural number; Re-fit the set CFIT = {cfit k}, cfit k is the k-th element in the re-fitting set CFIT, k is a natural number. In the first iteration, the start point number 1 and the end point number M of the smoothed point set FMP form the initial fitting set FIT = {fit j} = {(1, M)}, CFIT is an empty set j = 1, k = 1.

[0043] S5: Read the j-th element fit j in the fitting set, denoted as fit i = (s, e). Take the point set between the s-th point and the e-th point in the smoothed point set to form the fitting sub-stroke MMP = {mmp i} = {(fmp i .x, fmp 2 .y)}, s≤i≤e.

[0044] S6: Least squares fitting of sub-stroke MMP. Let the quadratic fitting polynomial of sub-stroke MMP be f(x) = a2x 2 +a1x+a0, where a2, a1 and a0 are the coefficients to be solved, i.e. the parameters of the sub-stroke to be fitted. Denoted by the coefficient vector A = [a0, a1, a2] T . The fitting error of the sampling points mmp i on the sub-stroke MMP to the fitting curve is ε i =mmp i .y-f(mmp i .x). In matrix form, Y = XA + ε, where Y = [mmp s .y, mmp s+1 .y, …, mmp ey] T , X = [1, mmp s .x, mmp s .x 2 ; 1, mmp s+1 .x, mmp s+1 .x 2 ;... 1, mmp e .x, mmp e .x 2 ], ε = [ε s , ε s+1 ,..., ε e ] T .

[0045] The fitting error square sum of each sampling point on the sub-stroke MMP is

[0046]

[0047] The least square method is to make the error square sum error minimum, according to the extreme value theorem, it should satisfy:

[0048]

[0049] The coefficient vector A = (X T X) -1 X T Y can be obtained.

[0050] S7: Calculate the relative fitting error derror. Substitute the values of X, Y and A obtained in S6 into

[0051]

[0052] The relative fitting error value is calculated; if derror≤δ, the fitting error meets the requirements, the starting point serial number s, the end point serial number e and the fitting coefficient A of the current sub-stroke are stored, j = j + 1, and turn to S8. If derror≥δ, the element (s, e) is written into the re-fitting set CFIT = {CFIT, (s, e)}.

[0053] S8: Judge whether j is greater than the number of elements in the fitting set. If j is greater than the number of elements in the fitting set, it means that all sub-strokes have been fitted in the current iteration round, and FIT is set to an empty set j = 1, and turn to S9. Otherwise, turn to S5.

[0054] S9: Judge whether the re-fitting set is empty. If the re-fitting set is not empty, turn to S10; if the re-fitting set is empty, it means that all sub-strokes have been fitted and the fitting error is less than the set value, and the program turns to S14.

[0055] S10: Read the k-th element cfit from the refit set. k , denoted as cfit k = (cs, ce). Points in the filtered sample set FMP (fmp) cs .x,fmp cs .y) and point (fmp ce .x,fmp ce Connect .y) to form a straight line L, and calculate the sub-stroke MMP = {mmp} i}={(fmp i .x,fmp i Given the distance from each point on line L to the line y), find the index of the point with the maximum distance and denote it as the index of the newly added dividing point cp.

[0056] S11: Write the elements (cs,cp) and (cp,ce) into the fitted set FIT={FIT,(cs,cp),(cp,ce)}, k=k+1.

[0057] S12: Determine if k is greater than the number of elements in the refit set. If k is greater than the number of elements in the refit set, it means that the sub-strokes whose fitting error does not meet the requirements in the current iteration have been further segmented according to the greedy strategy, and proceed to S13. Otherwise, proceed to S10 for loop processing.

[0058] S13: Set the refit set to an empty set, k=1, and proceed to S5 to perform a new round of iterative fitting.

[0059] S14: Delete redundant split points. Starting from the first sub-stroke after splitting, if two adjacent consecutive sub-strokes can be merged into one sub-stroke, then these two adjacent sub-strokes are considered to belong to the same stroke, and the redundant split points connecting these two sub-strokes are deleted.

[0060] S15: Repeat the above steps to traverse all adjacent sub-strokes, delete redundant segmentation points, and obtain the corrected segmentation point sequence.

[0061] For complex hand-drawn sketches with many strokes, since the strokes are naturally separated, we only need to use the above method to separate each stroke independently.

[0062] To verify the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method. Examples of experiments included... Fig. 2 and Fig. 3The single-stroke sketch and multi-stroke sketch shown, 10 subjects draw each figure 10 times with a mouse or a handwriting pen, and each figure is 100 samples. The computer reads the sketch sequence to form the original stroke point set MP, and after resampling and smoothing filtering, the automatic segmentation fitting based on the greedy strategy is performed. The sketch segmentation accuracy is used as the error standard of sketch segmentation;

[0063]

[0064] The time required for sketch segmentation fitting is used as the efficiency standard of sketch segmentation fitting.

[0065] To verify the effectiveness of the method of the application, the sketch segmentation fitting method based on the pen speed feature and Bezier curve (CS-Bezier), the sketch segmentation fitting method based on curvature and Bezier curve (PC-Bezier) and the method of the application (DS-LSM) are respectively used to segment and fit the above hand-drawn sketches. When the relative fitting error threshold δ is 0.1, the sketch segmentation accuracy, the false positive rate and the false negative rate are as shown in the following table:

[0066] Table 1

[0067]

[0068] Table 1 uses different sketch segmentation fitting methods to obtain the error standard and efficiency standard of sketch segmentation fitting. From Table 1, it can be seen that the sketch segmentation accuracy of the method of the application is higher than that of the two existing segmentation fitting methods, which shows that the accuracy and efficiency of the method of the application for segmenting and fitting the sketch are higher, which is sufficient to show that the method of the application is effective.

[0069] In summary, the automatic segmentation and fitting method for hand-drawn sketches based on the greedy strategy provided by the application, through firstly resampling and smoothing filtering the input sketch stroke sequence, then automatically segmenting the hand-drawn sketch strokes according to the greedy strategy. In the segmentation process, each iteration starts to read a data in the segmentation set, which contains 2 adjacent segmentation points, the least square fitting is performed on the sub-stroke between the 2 adjacent segmentation points and the fitting error is calculated, the segmentation points and the parameters of the sub-stroke meeting the fitting accuracy requirements are stored, the sub-stroke needing to be re-segmented with the fitting accuracy not meeting the requirements is recorded, and the region to be segmented is reduced. Then continue to traverse all the sub-strokes needing to be re-segmented according to the greedy strategy, find the optimal segmentation point, perform the least square fitting on the iteratively segmented sub-stroke, calculate the fitting error, store the next segmentation point and the parameters of the sub-stroke meeting the fitting accuracy requirements obtained by the greedy search, record the sub-stroke needing to be re-segmented with the fitting accuracy not meeting the requirements, further reduce the problem range, and loop the above segmentation and fitting process until all the sub-strokes meet the fitting accuracy, thereby saving the exhaustive operation needed to find the global optimal segmentation point, and effectively overcoming the influence of the randomness and semantic ambiguity of the hand-drawn sketch image, and improving the accuracy and efficiency of the stroke segmentation.

[0070] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0071] Although the embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, alternatives, and variations can be made thereto without departing from the principles and spirit of the application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for automatic segmentation and fitting of hand-drawn sketches based on a greedy strategy, characterized in that, The method comprises the following steps: S1: obtaining all points on the sketch stroke, storing x, y coordinate values thereof, and generating an original stroke point set MP; S2: performing equal-interval resampling on the original stroke point set to generate a resampled set REMP; S3: performing smoothing filtering on the resampled stroke point set to filter out stroke noise introduced due to jitter, and generating a smoothing filtered point set FMP; S4: setting an allowed fitting error δ, the start point and end point of the smoothing filtered point set forming an initial fitting set FIT, an initial re-fitting set CFIT being an empty set, j = 1, and k = 1; S5: read the jth element in the fitting set fit j , denoted as fit j = (s, e); take the point set between the s-th point and the e-th point in the point set after smoothing filtering to constitute the fitting sub-stroke MMP = {mmp i} = {(fmp i .x, fmp i .y)}, s≤i≤e; S6: fitting the sub-stroke, calculating fitting parameters and a fitting error; S7: judging whether the fitting error meets the requirement, if yes, storing the end point sequence number and the fitting parameters of the current sub-stroke, j = j + 1, and turning to S8; otherwise, storing the start point sequence number and the end point sequence number of the current sub-stroke in the re-fitting set; S8: judging whether all sub-strokes in this round of iteration have been fitted, if not, turning to S5 to fit the next sub-stroke, otherwise, turning to S9; S9: judging whether the re-fitting set is an empty set, if yes, turning to S14; otherwise, turning to S10; S10: connecting the start point and the end point of the kth sub-stroke in the re-fitting set into a straight line, calculating the Euclidean distance of each point on the sub-stroke to the straight line, and finding the point corresponding to the maximum distance as a new segmentation point according to the greedy strategy; S11: the new segmentation point further divides the sub-stroke which needs to be re-fitted into two sub-strokes, and the start point and the end point sequence numbers of the two new sub-strokes are added to the fitting set in the next round of iteration, k = k + 1; S12: judging whether k is greater than the number of elements in the re-fitting set, if yes, it is indicated that the sub-stroke whose fitting error does not meet the requirement in the current iteration round has been further segmented according to the greedy strategy, and turning to S13; otherwise, turning to S10 for cyclic processing; S13: setting the re-fitting set to be an empty set, k = 1, and turning to S5 for a new round of iteration fitting; S14: deleting redundant segmentation points; S15: traversing all adjacent sub-strokes according to the above steps, deleting redundant segmentation points, and obtaining a corrected segmentation point sequence.

2. The method of claim 1, wherein, In S6, the least square method is used to fit the sub-stroke.

3. The method of claim 1, wherein, In S14, when the redundant segmentation points are deleted, starting from the first sub-stroke after segmentation, if two adjacent continuous sub-strokes can be combined into one sub-stroke, it is considered that the two adjacent sub-strokes belong to the same stroke, and the redundant segmentation points connecting the two sub-strokes are deleted.

Images