Image segmentation method, device and storage medium

Abstract

The embodiments of this specification provide a method, device and storage medium for image segmentation, which can be applied to the field of artificial intelligence technology. The method includes: marking the coordinates of the segmentation lines of a preset number of images containing numbers to obtain a preset number of sample images; the segmentation lines are the positions for segmenting the images based on the numbers; using a convolutional neural network to extract the eigenvalue vectors of each sample image; based on a preset objective function, fitting the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image to obtain the classification conditions of the segmentation line positions in the sample images; when the objective function converges, segmenting the image to be segmented according to the classification conditions, thereby achieving accurate segmentation of the image and improving the efficiency of date recognition.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of artificial intelligence technology, and more particularly to a method, device and storage medium for image segmentation. Background Art

[0002] The mobile Internet has a rich variety of data sources, among which the amount of image data is increasing, and the amount of information attached to the images is also increasing. In order to extract useful information from these massive images, it is necessary to recognize the text in the images, and then convert it into semi-structured data and store it in the data center to provide more comprehensive data storage and retrieval services.

[0003] With the development of artificial intelligence technology, OCR (Optical Character Recognition) technology is usually used for text recognition in images. As far as the current situation is concerned, OCR recognition tasks are divided into two modes: one is word-by-word recognition, in which the first step is to extract single-word images, the second step is to use image classification algorithms to classify single words in the images, and finally all recognized words are spliced ​​into sentences for output; the other method is whole-line recognition, which takes into account the contextual features in the image at the same time, in which the first step is to locate the entire line sentence, and the second step is to use a sequence text recognition model to directly recognize the entire line of text.

[0004] However, some images also include some handwritten numbers. For example, statistical reports, financial reports, postal codes, various bills, etc. usually include handwritten date information. In these applications, handwritten number recognition algorithms are usually required to have high recognition speed and recognition accuracy as well as high reliability and stability. Existing OCR recognition methods usually segment the characters in the handwritten date according to the blank spaces between the characters, and then recognize each segmented character.

[0005] However, the "year, month, and day" in the middle of a picture containing a date may not necessarily be Chinese characters, but may also be "-", ".", spaces, shadows, etc. Sometimes numbers may overlap with Chinese characters, which results in poor results when simply segmenting based on the blank spaces in the picture, and thus the date in the picture cannot be accurately identified. Summary of the invention

[0006] The purpose of the embodiments of this specification is to provide a method, device and storage medium for image segmentation, so as to achieve accurate segmentation of images and improve the efficiency of date recognition.

[0007] To solve the above problems, an embodiment of the present specification provides a method for image segmentation, the method comprising: marking a preset number of images containing numbers with segmentation line coordinates to obtain a preset number of sample images; the segmentation line is the position where the image is segmented based on the number; using a convolutional neural network to extract the eigenvalue vector of each sample image; based on a preset objective function, fitting the eigenvalue vector of each sample image and the segmentation line coordinates corresponding to each sample image to obtain the classification conditions of the segmentation line position in the sample image; when the objective function converges, segmenting the image to be segmented according to the classification conditions.

[0008] To solve the above problems, an embodiment of the present specification also provides an image segmentation device, which includes: a labeling module, which is used to label the segmentation line coordinates of a preset number of images containing numbers, and obtain a preset number of sample images; the segmentation line is the position where the image is segmented based on the number; an extraction module, which is used to use a convolutional neural network to extract the eigenvalue vector of each sample image; a fitting module, which is used to fit the eigenvalue vector of each sample image and the segmentation line coordinates corresponding to each sample image based on a preset objective function, and obtain the classification condition of the segmentation line position in the sample image; a segmentation module, which is used to segment the image to be segmented according to the classification condition when the objective function converges.

[0009] To solve the above problems, an embodiment of the present specification also provides an electronic device, including: a memory for storing a computer program; a processor for executing the computer program to achieve: marking the dividing line coordinates of a preset number of pictures containing numbers to obtain a preset number of sample pictures; the dividing lines are the positions for dividing the pictures based on the numbers; using a convolutional neural network to extract the eigenvalue vectors of each sample picture; based on a preset objective function, fitting the eigenvalue vectors of each sample picture and the dividing line coordinates corresponding to each sample picture to obtain the classification conditions of the dividing line positions in the sample pictures; when the objective function converges, segmenting the pictures to be segmented according to the classification conditions.

[0010] To solve the above problems, an embodiment of the present specification also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement: marking the dividing line coordinates of a preset number of pictures containing numbers to obtain a preset number of sample pictures; the dividing lines are the positions for dividing the pictures based on the numbers; using a convolutional neural network to extract the eigenvalue vectors of each sample picture; based on a preset objective function, fitting the eigenvalue vectors of each sample picture and the dividing line coordinates corresponding to each sample picture to obtain the classification conditions of the dividing line positions in the sample pictures; when the objective function converges, segmenting the pictures to be segmented according to the classification conditions.

[0011] It can be seen from the technical solutions provided in the above embodiments of this specification that, in the embodiments of this specification, the dividing line coordinates of a preset number of pictures containing numbers can be marked to obtain a preset number of sample pictures; the dividing line is the position where the picture is divided based on the number; a convolutional neural network is used to extract the eigenvalue vector of each sample picture; based on a preset objective function, the eigenvalue vector of each sample picture and the dividing line coordinates corresponding to each sample picture are fitted to obtain the classification conditions of the dividing line position in the sample picture; when the objective function converges, the picture to be divided is segmented according to the classification conditions to achieve accurate segmentation of the picture and improve the efficiency of date recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 This is a schematic diagram of the date picture of the embodiment of this specification;

[0014] Figure 2 A schematic diagram of cutting out a digital image according to an embodiment of this specification;

[0015] Figure 3 This is a sample of the date pictures of the embodiments of this specification;

[0016] Figure 4 This is a schematic diagram of a model training process in a scenario example of an embodiment of this specification;

[0017] Figure 5 A schematic diagram for labeling pictures in the embodiments of this specification;

[0018] Figure 6 This is a schematic diagram of a model structure in a scenario example of an embodiment of this specification;

[0019] Figure 7 This is a flow chart of digital image recognition according to an embodiment of this specification;

[0020] Figure 8 This is a schematic diagram of the segmentation effect of the embodiment of this specification;

[0021] Fig. 9 This is a flow chart of a method for image segmentation according to an embodiment of this specification;

[0022] Fig.10 This is a functional structure diagram of an electronic device according to an embodiment of this specification;

[0023] Fig.11 This is a functional structure diagram of a picture segmentation device according to an embodiment of this specification. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0025] In the embodiments of this specification, OCR (Optical Character Recognition) refers to the process in which an electronic device (such as a scanner or a digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then translates the shapes into computer text using a character recognition method; that is, for printed characters, an optical method is used to convert the text in a paper document into a black and white dot matrix image file, and recognition software is used to convert the text in the image into a text format for further editing and processing by a word processing software.

[0026] As far as the current situation is concerned, OCR recognition tasks are divided into two modes. One is word-by-word recognition. The first step is to extract single-word images, the second step is to use image classification algorithms to classify single words in the images, and finally to splice all recognized words into sentences for output. The other approach is to recognize the entire line. The first step is to locate the entire line sentence, and the second step is to use a sequence text recognition model to directly recognize the entire line of text. The second OCR recognition model takes into account the contextual features in the image at the same time and can output the recognition results of the entire line of text at one time. It has good performance in accuracy and recognition speed, and is a more popular approach. However, it also has a disadvantage that the generalization ability of the recognition model is not enough and the portability is poor. If a model is changed to a scenario that has rarely been learned before, such as reusing a model trained in a medical scenario to a financial scenario, and the professional terms appearing in the two scenarios are completely different, then the accuracy of text recognition may drop significantly, and a large number of samples need to be re-annotated for training. In many cases, it is difficult to obtain a large number of annotated samples in a short time. However, the first solution can be very flexible for transplantation. Because it recognizes word by word, it is not sensitive to the context.

[0027] Due to the lack of more training samples, some OCR recognition models, such as date recognition, directly call the trained digital recognition model. It is necessary to convert dates such as "August 22, 2019" (such as Figure 1As shown in the figure, the images are divided into three pictures: "2019", "8" and "22" and then fed into the downstream digital recognition model for recognition. This requires that the pictures containing dates are divided into the following Figure 2 The existing segmentation method usually segments the characters in the date based on the blank spaces between them, and then recognizes each character after segmentation. However, the "year, month, and day" in the middle of the image containing the date may not necessarily be Chinese characters, but may also be "-", ".", spaces, shadows, etc. Figure 3 As shown. Of course, sometimes numbers may overlap with Chinese characters. This results in poor results when simply segmenting by blank spaces in the image, and thus the date in the image cannot be accurately identified.

[0028] Considering that if a sample image containing the coordinates of the segmentation line is produced based on a deep learning method, and a deep learning algorithm is used to extract features and train the sample image, thereby obtaining the classification conditions of the segmentation line position in the sample image, and the image to be segmented is segmented according to the classification conditions, it is expected to solve the problem of poor segmentation effect based on the blank space between the characters in the date in the prior art, achieve accurate segmentation of the image, and improve the efficiency of date recognition. Based on this, the embodiments of this specification provide a method, device, and storage medium for image segmentation.

[0029] See also Figure 4 and Figure 5 In a scenario example, a machine learning model can be established based on a deep learning algorithm, and the model can be trained using labeled samples. After the trained model meets certain conditions, the trained model is used as a model for digital segmentation extraction. The image to be segmented is input into the model for digital segmentation extraction, and the coordinates of the segmentation line can be output to facilitate segmentation of the image to be segmented.

[0030] In this example scenario, Figure 4 As shown, the model training process can include the following steps.

[0031] S401: Read date image data and adjust it to a fixed size to obtain training data.

[0032] In this scenario example, the date image data includes a preset number of pictures, and the pictures may contain data of the date type.

[0033] In this scenario example, you can use the OpenCV tool to read the date image data into RGB three-channel format, so that the extracted image data can be converted into a three-dimensional matrix, and then the converted three-dimensional matrix is ​​adjusted to a shape of 3×100×400. Among them, 3 represents the RGB three-channel, and 100×400 is the pixel of the image, that is, the image has a fixed height of 100 and a width of 400. The adjusted three-dimensional matrix is ​​the training data x for model training in this scenario example. In this scenario example, 100 sets of training data can be taken to train the model.

[0034] S402: Calculate the normalized segmentation line coordinate vector and obtain a label.

[0035] In this scenario example, the coordinates of the dividing line can be used as annotations to mark the training data. The dividing line is the position where the image is divided based on the number. Figure 5 As shown in the figure, the four vertical lines (y1, y2, y3, y4) are dividing lines. Among them, (y1, y2, y3, y4) correspond to Figure 5 The distance from the four vertical lines in the picture shown to the leftmost border of the picture. Assuming the width of the original picture is W, we can get:

[0036]

[0037] In this scenario example, this 4-dimensional vector (y1, y2, y3, y4) with element values ​​​​being integers between 0 and 100 can be used as the label y, such as (30, 59, 71, 87), and (y1, y2, y3, y4) is called the standardized coordinate values ​​​​of the four dividing lines.

[0038] S403: Build a segmentation model structure based on Resnet18 and FPN.

[0039] Deep residual network (ResNet) is a convolutional neural network (CNN). Resnet18 is a network structure in deep residual network. The number 18 represents the depth of the network, that is, 18 specifies 18 layers with weights, including convolutional layers and fully connected layers, excluding pooling layers and BN layers.

[0040] FPN (Feature Pyramid Networks) is a method that uses conventional CNN models to efficiently extract features of various dimensions in images. The overall goal of FPN is to use the semantic feature pyramid from high to low of the convolutional network to build a pyramid with high-level semantics; top-down and lateral connections are proposed to connect rich semantic features and high resolution, so that the network can adapt to classification and positioning tasks.

[0041] In this scenario example, the model structure is as follows Figure 6 As shown. Resnet18 can be used as a feature extraction model for images to extract feature maps of each sample image; wherein each sample image corresponds to multiple feature maps; and different feature maps are composed of different color channels. Specifically, in each convolution layer, the sample image exists in a three-dimensional form. The sample image can be regarded as many two-dimensional images stacked together, each of which can be called a feature map. In the input layer, if it is a grayscale image, there is only one feature map; if it is a color image, there are generally 3 feature maps (red, green, and blue). There will be several convolution kernels between layers. The previous layer and each feature map are convolved with each convolution kernel to generate a feature map of the next layer. Therefore, multiple corresponding feature maps can be extracted for each sample image.

[0042] In this scenario example, after extracting the feature maps of each sample image, multiple feature maps of each sample image can be aggregated and convolved through FPN, that is, the high-resolution low-level features and high-semantic information of high-level features are simultaneously utilized, and the feature maps of these different layers are fused, and then a 25×1 convolution kernel is used for convolution operation to obtain 4 1×100-dimensional vectors (y′1, y′2, y′3, y′4). Among them, (y′1, y′2, y′3, y′4) represents the coordinates of the fitting segmentation line, corresponding to (y1, y2, y3, y4), 100 dimensions correspond to 100 sets of training data, and each 1×100-dimensional vector corresponds to a segmentation line coordinate of 100 sets of training data.

[0043] S404: Fit the training data and the labeled data according to the designed model structure and objective function.

[0044] In this scenario example, you can define an objective function to train the model. The model training process is also the fitting process, and its goal is to make the objective function converge. Specifically, the defined objective function loss is:

[0045]

[0046] in, is the first loss function, y represents the marked segmentation line coordinates, y′ represents the fitted segmentation line coordinates, and its goal is to continuously reduce the error between y and y′.

[0047] In this scenario example, it is agreed that each vector fits a segmentation line. The position of the maximum value of each vector element corresponds to (y1, y2, y3, y4). It is equivalent to a regression problem, using the index of the maximum value of each of the four vectors (y′1, y′2, y′3, y′4) to fit the label value (y1, y2, y3, y4).

[0048] CrossEntropy represents the cross entropy loss function. When the cross entropy loss function is used as a component of the objective function, the fitting problem of the four segmentation lines can be transformed into four classification problems, each of which is a 100-class classification problem. S403 outputs four 1×100-dimensional vectors. The position of the maximum value of each 1×100 vector is the label of the class to which it is divided. Use this label (y′1, y′2, y′3, y′4) to correspond to the label value (y1, y2, y3, y4) calculated in S402. At this point, you will find that when the class label (y′1, y′2, y′3, y′4) is consistent with (y1, y2, y3, y4), it means that the classification is completely correct. At this time, the error calculated by smoothL1 is exactly 0. The purpose of smoothL1 and CrossEntropy is to make the position of the maximum value element of the four fitted vectors equal to the standardized coordinate value (y1, y2, y3, y4) of the four segmentation lines. One is to fit by regression method and the other is to fit by classification method.

[0049] Represents the second loss function, A is a binary vector obtained based on the fitted segmentation line coordinates, and B is a binary vector obtained based on the marked segmentation line coordinates. The second loss function is equivalent to comparing the intersection and union of two binary vectors. If the two vectors are equal, then the intersection is equal to the union, and the output value of the second loss function is 0. At this time, the four 1×100-dimensional vectors output by S403 will be summed by column to become a 100-dimensional vector, and then the sigmoid function will be used to map the values ​​of all elements of this vector to between 0 and 1, and then take those greater than 0.5 as 1 otherwise 0. A 100-dimensional binary vector A is obtained. According to the coordinate values ​​of S402 (y1, y2, y3, y4), a 100-dimensional binary vector B can also be produced, with the position corresponding to the coordinate value equal to 1 and the others equal to 0. Then these two vectors can be used to calculate DiceLoss, the purpose of which is to make the two vectors as equal as possible. When they are equal, DiceLoss is also equal to 0. Its goal is consistent with smoothL1 and CrossEntropy.

[0050] S405: Output a model that can automatically perform digital segmentation and extraction.

[0051] In this scenario example, the objective function composed of smoothL1, CrossEntropy and DiceLoss is used to train the model, which is equivalent to treating the image segmentation problem as a regression problem, a classification problem and a vector similarity problem for multiple fitting. S405 uses the data processed by S401 and S402 and the model and objective function designed by S403 and S404 to train the model, and finally outputs a model that can automatically perform digital segmentation and extraction when the objective function converges.

[0052] In this scenario example, the trained model can be called a segmentation model. Figure 7 As shown, the segmentation model can be used to segment the image according to the following steps to facilitate the identification of the date in the image.

[0053] S1: Input image data.

[0054] S2: Segmentation model performs image segmentation.

[0055] The segmentation model can segment the image, extract the small digital image, and then provide it to the downstream digital recognition model for recognition and finally output the date recognition result. Even if the numbers and text overlap, they can be segmented accurately, such as Figure 8 As shown, the accuracy of the segmentation model in segmenting the image is above 98%.

[0056] S3: The digital recognition model recognizes the segmented image blocks one by one.

[0057] The digital recognition model may be a model built based on OCR technology, and is used to recognize digital numbers.

[0058] S4: Integrate into date output.

[0059] Specifically, the recognition results of the various image blocks may be combined to obtain a date recognition result of the image data.

[0060] In the scenario example, the date image data can be read and adjusted to a fixed size to obtain training data. For the training data, the standardized segmentation line coordinate vector is calculated to obtain annotations. The segmentation model structure is constructed based on Resnet18 and FPN. The training data and the annotation data are fitted according to the designed model structure and objective function to obtain the segmentation model. The segmentation model is used to segment the image to facilitate the identification of the date in the image, thereby achieving accurate segmentation of the image and improving the efficiency of date recognition. Of course, this scenario example can also be applied to the scene of segmenting long text images. At this time, you only need to mark the coordinates of the position where the long sentence needs to be segmented. You can use the method of this scenario example to train the model and create an automatic segmentation model.

[0061] See also Fig. 9 , this specification embodiment provides a method for image segmentation. In the embodiment of this specification, the subject that executes the image segmentation method may be an electronic device with a logical operation function, and the electronic device may be a server. The server may be an electronic device with a certain operation and processing capability. It may have a network communication unit, a processor, a memory, etc. Of course, the server is not limited to the above-mentioned electronic device with a certain entity, and it may also be software running in the above-mentioned electronic device. The server may also be a distributed server, which may be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server may also be a server cluster formed by several servers. The method may include the following steps.

[0062] S910: Marking the coordinates of the dividing lines for a preset number of pictures containing numbers to obtain a preset number of sample pictures; the dividing lines are positions for dividing the pictures based on the numbers.

[0063] In some embodiments, the image may be an electronic file, such as an image file in the format of pdf, jpg, png, etc. The image may also be obtained by scanning the paper file through a scanner, a digital camera, etc., to obtain a corresponding electronic file. The paper file may be various bills, newspapers, books, manuscripts, and other printed materials.

[0064] The pictures may contain numbers. For example, various pictures may contain date information, and the numbers may be the year, month, and day in the date information.

[0065] In some embodiments, Figure 5As shown, each sample image may have 4 dividing lines, and the 4 dividing lines separate the numbers in the sample image by year, month, and day. Of course, since some images may not have a "day" in the date, the dividing lines may also be 3. Among them, the number of dividing lines for each sample image in the same batch of sample images should be the same, preferably 4. For the case where some images may not have a "day" in the date, the dividing line used to divide the "day" can be overlapped with the previous dividing line.

[0066] In some embodiments, the coordinates of the dividing line can be used as annotations to mark the picture containing the number. The dividing line is the position where the picture is divided based on the number. Figure 5 As shown in the figure, the four vertical lines (y1, y2, y3, y4) are dividing lines. Among them, (y1, y2, y3, y4) correspond to Figure 5 The distance from the four vertical lines in the picture shown to the leftmost border of the picture. Assuming the width of the original picture is W, we can get:

[0067]

[0068] In some embodiments, this 4-dimensional vector (y1, y2, y3, y4) with element values ​​between 0 and 100 can be used as the label y, such as (30, 59, 71, 87), and (y1, y2, y3, y4) is called the normalized coordinate value of the four dividing lines. Of course, for some pictures where the date may not have a "day", y3 = y4.

[0069] In some embodiments, due to the different sources of various pictures, the formats and sizes of the pictures may be quite different. In order to minimize these differences, each picture can be read into an RGB three-channel format before marking the coordinates of the dividing lines of a preset number of pictures containing numbers, so as to obtain a preset number of pictures in the form of a three-dimensional matrix; the pixels of the pictures in the form of a preset number of three-dimensional matrices are set to preset pixels. For example, the OpenCV tool can be used to read the picture into an RGB three-channel format so as to convert the picture into a three-dimensional matrix, and then the converted three-dimensional matrix is ​​adjusted to a shape of 3×100×400. Among them, 3 represents the RGB three-channel, 100×400 is the pixel of the picture, that is, the picture is fixed with a height of 100 and a width of 400. Of course, the pixels of the picture can also be set to other values, for example, the resolution of some pictures may be 300*160, and the resolution of some pictures may be 320*150. Therefore, in order to improve the accuracy of the eigenvalue vector extraction of the picture, the picture can be set to the same resolution. Specifically, the pixels of the subsequent pictures can be set to the same pixels as the first picture based on the pixels of the first picture. For example, if the first picture has a pixel size of 300*160, then the pixels of all pictures can be set to 300*160. Alternatively, a pixel can be preset and all pictures can be set to the preset pixel. For example, if the preset pixel size is 320*150, then the pixels of all pictures can be set to 320*150.

[0070] In some embodiments, the method further includes: preprocessing the preset number of pictures containing numbers; and correspondingly, marking the coordinates of the dividing line on the preprocessed pictures. Specifically, since the sources of various pictures are different, the quality of the pictures cannot be guaranteed. In order to improve the quality of the pictures, the preset number of pictures containing numbers can also be preprocessed.

[0071] In some embodiments, the preprocessing may include standardization, tilt correction, etc. Image standardization is a process of centralizing data by removing the mean, according to convex optimization theory and knowledge related to data probability distribution. Data centralization conforms to the law of data distribution and is more likely to achieve generalization effects after training.

[0072] Since the scanning and shooting process involves manual operation, the image to be recognized input into the computer will have some tilt to a certain extent. Before the printed characters in the image are recognized, it is necessary to detect the image direction and correct the image direction. Specifically, the image to be recognized can be corrected by the Hough transform method. The Hough transform mainly uses the transformation between the space where the image is located and the Hough space to map the curve or straight line with a shape in the rectangular coordinate system where the image is located to a point in the Hough space to form a peak, thereby converting the problem of detecting arbitrary shapes into the problem of calculating peaks. That is, a straight line in the rectangular coordinate system where the image is located becomes a point when converted to the Hough space, and is formed by the intersection of multiple straight lines. The statistical peak is the number of intersecting lines at the intersection point.

[0073] S920: Use a convolutional neural network to extract a feature value vector of each sample image.

[0074] In some embodiments, the eigenvalue vector may include a pixel value vector, which is formed by extracting the RGB value of each pixel in the picture and sorting the extracted RGB values ​​in a certain order, such as sorting by pixel number. Specifically, a picture can generally be composed of multiple pixels and then embodied. For example, the picture displayed on the display is the display function of each light-emitting element on the display screen that can display different colors after being powered on, so that many such light-emitting elements on the screen are combined to restore and display the picture. When the picture displayed on the display screen is the original resolution, each element used to display the picture on the display screen corresponds to each pixel on the picture. Labeling the eigenvalue vector of the picture by the pixel value vector can effectively characterize the characteristics of the picture.

[0075] In some embodiments, the convolutional neural network (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is one of the representative algorithms of deep learning. In the embodiments of this specification, Resnet18 and FPN are used to extract the eigenvalue vectors of each sample image. Specifically, the use of a convolutional neural network to extract the eigenvalue vectors of each sample image may include the following steps.

[0076] S921: Use Resnet18 to extract feature maps of each sample image; wherein each sample image corresponds to multiple feature maps; and different feature maps are composed of different color channels.

[0077] In some embodiments, Resnet18 is a network structure in a deep residual network, and the number 18 represents the depth of the network, that is, 18 specifies 18 layers with weights, including convolutional layers and fully connected layers, but excluding pooling layers and BN layers. Resnet18 can be used as a feature extraction model for images to extract feature maps of each sample image; wherein each sample image corresponds to multiple feature maps; and different feature maps are composed of different color channels. Specifically, in each convolutional layer, the sample image exists in a three-dimensional form, and the sample image can be regarded as many two-dimensional images stacked together, each of which can be called a feature map. In the input layer, if it is a grayscale image, there is only one feature map; if it is a color image, there are generally 3 feature maps (red, green, and blue). There will be several convolution kernels between layers. The previous layer and each feature map are convolved with each convolution kernel, and a feature map of the next layer will be generated. Thus, multiple corresponding feature maps can be extracted from each sample image.

[0078] S922: Use a feature pyramid network to aggregate and convolve multiple feature maps of each sample image to obtain a feature value vector of each sample image.

[0079] In some embodiments, Feature Pyramid Networks (FPN) is a method that uses a conventional CNN model to efficiently extract features of various dimensions in an image. The overall goal of FPN is to use the semantic feature pyramid from high to low of the convolutional network to build a pyramid with high-level semantics; top-down and lateral connections are proposed to connect rich semantic features and high resolution, so that the network can adapt to classification and positioning tasks. After extracting the feature maps of each sample image, multiple feature maps of each sample image can be aggregated and convolved through FPN, that is, the high resolution of low-level features and the high semantic information of high-level features are simultaneously utilized, and by fusing these feature maps of different layers, a 25×1 convolution kernel is used for convolution operations to obtain 4 1×100-dimensional vectors (y′1, y′2, y′3, y′4). Among them, (y′1, y′2, y′3, y′4) represents the coordinates of the fitting segmentation line, corresponding to (y1, y2, y3, y4), 100 dimensions correspond to 100 groups of sample images, and each 1×100-dimensional vector corresponds to a segmentation line coordinate of 100 groups of training data. Of course, the number of sample images is different, and the dimension of the obtained vector is also different. For example, if the number of sample images is 1000, 4 1×1000-dimensional vectors (y′1, y′2, y′3, y′4) are obtained, and if the number of sample images is 2000, 4 1×2000-dimensional vectors (y′1, y′2, y′3, y′4) are obtained.

[0080] In the embodiments of the present specification, Resnet18 and FPN are used to extract the eigenvalue vectors of each sample image, which can reduce the amount of calculation for extracting the eigenvalue vectors of each sample image and improve the efficiency of eigenvalue vector extraction.

[0081] S930: Based on a preset objective function, the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image are fitted to obtain classification conditions for the segmentation line positions in the sample images.

[0082] In some embodiments, the objective function loss is defined as:

[0083]

[0084] in, is the first loss function, y represents the marked segmentation line coordinates, y′ represents the fitted segmentation line coordinates, and its goal is to continuously reduce the error between y and y′.

[0085] In this scenario example, it is agreed that each vector fits a segmentation line. The position of the maximum value of each vector element corresponds to (y1, y2, y3, y4). It is equivalent to a regression problem, using the index of the maximum value of each of the four vectors (y′1, y′2, y′3, y′4) to fit the marked segmentation line coordinates (y1, y2, y3, y4).

[0086] CrossEntropy represents the cross entropy loss function. When the cross entropy loss function is used as a component of the objective function, the fitting problem of the four segmentation lines can be transformed into four classification problems, each of which is a 100-class classification problem. S920 outputs four 1×100-dimensional vectors. The position of the maximum value of each 1×100 vector is the label of the class to which it is divided. Use this label (y′1, y′2, y′3, y′4) to correspond to the segmentation line coordinates (y1, y2, y3, y4) marked in S910. Here you will find that when the class label (y′1, y′2, y′3, y′4) is consistent with (y1, y2, y3, y4), it means that the classification is completely correct. At this time, the error calculated by smoothL1 is exactly 0. The purpose of smoothL1 and CrossEntropy is to make the position of the maximum value element of the four fitted vectors equal to the segmentation line coordinates (y1, y2, y3, y4) marked by the four segmentation lines. One is to fit by regression method and the other is to fit by classification method.

[0087] Represents the second loss function, A is a binary vector obtained based on the fitting segmentation line coordinates, and B is a binary vector obtained based on the marked segmentation line coordinates. The second loss function is equivalent to comparing the intersection and union of two binary vectors. If the two vectors are equal, then the intersection is equal to the union, and the output value of the second loss function is 0. At this time, the four 1×100-dimensional vectors output by S920 are summed by column to become a 100-dimensional vector, and then the sigmoid function is used to map the values ​​of all elements of this vector to between 0 and 1, and then take those greater than 0.5 as 1, otherwise 0. A 100-dimensional binary vector A is obtained. According to the segmentation line coordinates (y1, y2, y3, y4) marked by S910, a 100-dimensional binary vector B can also be produced, with the position corresponding to the coordinate value equal to 1 and the others equal to 0. Then these two vectors can be used to calculate DiceLoss, the purpose of which is to make the two vectors as equal as possible. When they are equal, DiceLoss is also equal to 0. Its goal is consistent with smoothL1 and CrossEntropy.

[0088] Using the objective function composed of smoothL1, CrossEntropy and DiceLoss to train the model is equivalent to treating the image segmentation problem as a regression problem, a classification problem and a vector similarity problem for multiple fitting, which effectively improves the fitting effect.

[0089] In some embodiments, the classification condition may be a mapping relationship between the eigenvalue vector of each sample image and the segmentation line coordinates corresponding to each sample image, for example, f(x)=y′, where x represents the eigenvalue vector of the sample image and y′ represents the fitted segmentation line coordinates.

[0090] S940: When the objective function converges, segment the image to be segmented according to the classification condition.

[0091] In some embodiments, the objective function is equal to 0, which means that the marked segmentation line coordinates and the fitted segmentation line coordinates are equal. However, in practical applications, it is difficult to make the marked segmentation line coordinates and the fitted segmentation line coordinates of all sample images equal. Therefore, the objective function should be as close to zero as possible. When the objective function converges, it means that the objective function region is stable, and even if more data is fitted, the classification condition will not be greatly improved. Therefore, when the objective function converges, the image to be segmented can be segmented according to the classification condition.

[0092] In some embodiments, segmenting the picture to be segmented according to the classification condition may include the following steps.

[0093] S941: extracting a feature value vector of the image to be segmented using a convolutional neural network;

[0094] S942: Substituting the eigenvalue vector of the image to be segmented into the classification condition to obtain the segmentation line coordinates of the image to be segmented;

[0095] S943: Segment the image to be segmented according to the segmentation line coordinates.

[0096] Specifically, first, the image to be segmented can be obtained, and then the image to be segmented can be subjected to similar operations as the sample image, such as preprocessing operations, converting the image to be segmented into a three-dimensional matrix form, and setting it to preset pixels, etc., and then using a convolutional neural network to extract the eigenvalue vector of the image to be segmented, substituting the eigenvalue vector of the image to be segmented into the classification condition, obtaining the segmentation line coordinates of the image to be segmented, and segmenting the image to be segmented according to the segmentation line coordinates. Through the above operations, the image to be segmented is placed in the same environment as the sample image as much as possible, reducing unnecessary variables, thereby improving the accuracy of image segmentation.

[0097] In some embodiments, after segmenting the image to be segmented, multiple image blocks can be obtained. The method may also include digitally identifying the multiple image blocks obtained after segmenting the image to be segmented, respectively, to obtain identification results of each image block; and outputting a date identification result obtained by combining the identification results of each image block. By identifying the image blocks segmented by the method provided in the embodiments of this specification, the accuracy of date identification can be improved.

[0098] It can be seen from the technical solutions provided in the above embodiments of this specification that, in the embodiments of this specification, the dividing line coordinates of a preset number of pictures containing numbers can be marked to obtain a preset number of sample pictures; the dividing line is the position where the picture is divided based on the number; a convolutional neural network is used to extract the eigenvalue vector of each sample picture; based on a preset objective function, the eigenvalue vector of each sample picture and the dividing line coordinates corresponding to each sample picture are fitted to obtain the classification conditions of the dividing line position in the sample picture; when the objective function converges, the picture to be divided is segmented according to the classification conditions to achieve accurate segmentation of the picture and improve the efficiency of date recognition.

[0099] The method embodiments provided in the above embodiments of this specification may be executed in a mobile terminal, a computer terminal or similar electronic devices. Fig.10 This is a functional structure diagram of an electronic device according to an embodiment of the present specification. The electronic device may include a memory and a processor.

[0100] In some embodiments, the memory can be used to store the computer program and / or module, and the processor implements various functions of the image segmentation method by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the user terminal. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0101] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor may execute the computer instructions to implement the following steps: marking the coordinates of the segmentation lines for a preset number of pictures containing numbers to obtain a preset number of sample pictures; the segmentation lines are positions for segmenting the pictures based on the numbers; extracting the eigenvalue vectors of each sample picture using a convolutional neural network; fitting the eigenvalue vectors of each sample picture and the segmentation line coordinates corresponding to each sample picture based on a preset objective function to obtain the classification conditions of the segmentation line positions in the sample pictures; and segmenting the pictures to be segmented according to the classification conditions when the objective function converges.

[0102] In the embodiments of this specification, the functions and effects specifically implemented by the electronic device can be explained in comparison with other embodiments and will not be repeated here.

[0103] Fig.11 This is a functional structural diagram of a picture segmentation device according to an embodiment of the present specification. The device may specifically include the following structural modules.

[0104] The marking module 1110 is used to mark the coordinates of the dividing lines of a preset number of pictures containing numbers to obtain a preset number of sample pictures; the dividing lines are positions where the pictures are divided based on the numbers;

[0105] An extraction module 1120, configured to extract a feature value vector of each sample image using a convolutional neural network;

[0106] A fitting module 1130 is used to fit the feature value vector of each sample image and the segmentation line coordinates corresponding to each sample image based on a preset objective function to obtain classification conditions for the segmentation line positions in the sample images;

[0107] The segmentation module 1140 is configured to segment the image to be segmented according to the classification condition when the objective function converges.

[0108] In some embodiments, the device may further include: a preprocessing module, used to preprocess the preset number of pictures containing numbers; correspondingly, the labeling module 1110 is used to label the preprocessed pictures with the coordinates of the dividing lines.

[0109] In some embodiments, the device may also include: an identification module, used to digitally identify the multiple picture blocks obtained after segmenting the picture to be segmented, and obtain identification results of each picture block; an output module, used to output a date recognition result obtained by combining the identification results of each picture block.

[0110] The embodiments of the present specification also provide a computer-readable storage medium of an image segmentation method, wherein the computer-readable storage medium stores computer program instructions, which, when executed, implement the following steps: annotating segmentation line coordinates for a preset number of images containing numbers to obtain a preset number of sample images; the segmentation lines are positions for segmenting images based on numbers; extracting eigenvalue vectors of each sample image using a convolutional neural network; fitting the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image based on a preset objective function to obtain classification conditions for the segmentation line positions in the sample images; and segmenting the images to be segmented according to the classification conditions when the objective function converges.

[0111] In the embodiments of this specification, the above-mentioned storage medium includes but is not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), cache (Cache), hard disk (Hard Disk Drive, HDD) or memory card (Memory Card). The memory can be used to store the computer program and / or module, and the memory can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system, at least one application required for a function, etc.; the storage data area can store data created according to the use of the user terminal, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory. In the embodiments of this specification, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other implementations, and will not be repeated here.

[0112] It should be noted that the image segmentation method, device and storage medium provided in the embodiments of this specification can be applied to the field of artificial intelligence technology. Of course, it can also be applied to the financial field, or any field other than the financial field. The embodiments of this specification do not limit the application field of the image segmentation method, device and storage medium.

[0113] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment and the equipment embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0114] After reading this specification, those skilled in the art can conceive of any combination of some or all of the embodiments listed in this specification without creative work, and these combinations are also within the scope of disclosure and protection of this specification.

[0115] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog2. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0116] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0117] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the present specification can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0118] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] This specification can be used in many general or special computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0120] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0121] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A method for image segmentation, characterized in that: The method comprises: Marking the coordinates of the dividing lines of a preset number of pictures containing numbers to obtain a preset number of sample pictures; the dividing lines are positions where the pictures are divided based on the numbers; Use convolutional neural network to extract feature value vectors of each sample image; Based on a preset objective function, the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image are fitted to obtain classification conditions for the segmentation line positions in the sample images; When the objective function converges, segmenting the image to be segmented according to the classification condition; Wherein, the objective function is: Where n represents the number of segmentation lines in each sample image; is the first loss function, y represents the marked segmentation line coordinates, y′ represents the fitted segmentation line coordinates; CrossEntropy represents the cross entropy loss function; represents the second loss function, A is a binary vector obtained based on the fitted segmentation line coordinates, and B is a binary vector obtained based on the marked segmentation line coordinates.

2. The method according to claim 1, characterized in that Each sample image has four dividing lines; the four dividing lines separate the numbers in the sample image by year, month, and day.

3. The method according to claim 1, characterized in that Before marking the dividing line coordinates of a preset number of pictures containing numbers, include: Read each image into RGB three-channel format to obtain a preset number of images in three-dimensional matrix form; The pixels of the preset number of pictures in the form of three-dimensional matrices are set as preset pixels.

4. The method according to claim 1, characterized in that: The method further comprises: Preprocessing the preset number of pictures containing numbers; Accordingly, the segmentation line coordinates are marked on the preprocessed image.

5. The method according to claim 4, characterized in that The preprocessing includes at least one of normalization processing and tilt correction.

6. The method according to claim 1, characterized in that The eigenvalue vector includes a pixel value vector.

7. The method according to claim 1, characterized in that The method of extracting the feature value vector of each sample image using a convolutional neural network includes: Resnet18 is used to extract feature maps of each sample image; wherein each sample image corresponds to multiple feature maps; different feature maps are composed of different color channels; Use the feature pyramid network to aggregate and convolve multiple feature maps of each sample image to obtain the feature value vector of each sample image.

8. The method according to claim 1, characterized in that The segmenting of the image to be segmented according to the classification condition comprises: Using a convolutional neural network to extract the feature value vector of the image to be segmented; Substituting the eigenvalue vector of the image to be segmented into the classification condition to obtain the segmentation line coordinates of the image to be segmented; The image to be segmented is segmented according to the segmentation line coordinates.

9. The method according to claim 1, characterized in that: The method further comprises: Digitally identifying the multiple picture blocks obtained after segmenting the picture to be segmented, respectively, to obtain the identification result of each picture block; Output the date recognition result obtained by combining the recognition results of each picture block.

10. A picture segmentation device, characterized in that: The device comprises: A marking module is used to mark the coordinates of the dividing lines of a preset number of pictures containing numbers to obtain a preset number of sample pictures; the dividing lines are positions where the pictures are divided based on the numbers; An extraction module, used to extract the feature value vector of each sample image using a convolutional neural network; A fitting module, used to fit the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image based on a preset objective function, so as to obtain the classification conditions of the segmentation line positions in the sample images; A segmentation module, configured to segment the image to be segmented according to the classification condition when the objective function converges; Wherein, the objective function is: Where n represents the number of segmentation lines in each sample image; is the first loss function, y represents the marked segmentation line coordinates, y′ represents the fitted segmentation line coordinates; CrossEntropy represents the cross entropy loss function; represents the second loss function, A is a binary vector obtained based on the fitted segmentation line coordinates, and B is a binary vector obtained based on the marked segmentation line coordinates.

11. The device according to claim 10, characterized in that The device also includes: A preprocessing module, used for preprocessing the preset number of pictures containing numbers; Correspondingly, the labeling module is used to label the segmentation line coordinates of the preprocessed image.

12. The device according to claim 10, characterized in that The device also includes: An identification module, used to digitally identify the multiple picture blocks obtained after segmenting the picture to be segmented, and obtain the identification result of each picture block; The output module is used to output a date recognition result obtained by combining the recognition results of each picture block.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement: marking a preset number of pictures containing numbers with segmentation line coordinates to obtain a preset number of sample pictures; the segmentation lines are positions where the pictures are segmented based on the numbers; and extracting a feature value vector of each sample picture using a convolutional neural network; Based on a preset objective function, the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image are fitted to obtain the classification conditions of the segmentation line positions in the sample images; when the objective function converges, the image to be segmented is segmented according to the classification conditions; wherein the objective function is: Where n represents the number of segmentation lines in each sample image; is the first loss function, y represents the marked segmentation line coordinates, y′ represents the fitted segmentation line coordinates; CrossEntropy represents the cross entropy loss function; represents the second loss function, A is a binary vector obtained based on the fitted segmentation line coordinates, and B is a binary vector obtained based on the marked segmentation line coordinates.

14. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the following are achieved: marking the coordinates of the segmentation lines of a preset number of pictures containing numbers to obtain a preset number of sample pictures; the segmentation lines are the positions where the pictures are segmented based on the numbers; and extracting the feature value vectors of each sample picture using a convolutional neural network; Based on a preset objective function, the eigenvalue vectors of each sample image and the segmentation line coordinates corresponding to each sample image are fitted to obtain the classification conditions of the segmentation line positions in the sample images; when the objective function converges, the image to be segmented is segmented according to the classification conditions; wherein the objective function is: Where n represents the number of segmentation lines in each sample image; is the first loss function, y represents the marked segmentation line coordinates, y′ represents the fitted segmentation line coordinates; CrossEntropy represents the cross entropy loss function; represents the second loss function, A is a binary vector obtained based on the fitted segmentation line coordinates, and B is a binary vector obtained based on the marked segmentation line coordinates.

Images