A natural scene Tibetan image data enhancement method based on Markov theory

Abstract

The present invention provides a method for enhancing Tibetan image data in natural scenes based on Markov theory, including: identifying input information semantics; defining a Markov process, which gradually adds noise disturbance to training data to form a Markov chain; defining an inverse Markov process, which uses a neural network model to predict and remove disturbances according to the current disturbance image and time step, and gradually restore the original image; training the neural network model to maximize the conditional probability of the inverse Markov process; generating an image, using the result obtained after the Markov process is processed as the initial state matrix, and then traversing the Markov chain in reverse, using the neural network model to gradually restore, and finally obtaining the generated image. The present invention can avoid common adversarial training, mode collapse, gradient disappearance and other problems in the generation model, and can achieve efficient parallelization and scalability, and the generated image has high resolution and diversity.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning and natural language processing, and in particular to a method for enhancing natural scene Tibetan image data based on Markov theory. Background Art

[0002] In the field of machine learning and natural language processing, due to the scarcity of Tibetan language data resources, especially in natural scenes, there is a lack of corresponding Tibetan image data, and natural scene images have different scenes, brightness, contrast, color and other styles. These current situations have hindered the collection and use of Tibetan language data to a certain extent. At the same time, Tibetan language image databases are very scarce, and research on Tibetan language data enhancement is also limited by insufficient data, which brings challenges to Tibetan language-related research and applications.

[0003] At present, the most commonly used image generation method is based on the Generative Adversarial Network (GAN) method, which was proposed by Goodfellow et al. in 2014. GAN consists of a generator and a discriminator. The task of the generator is to generate images from random noise, and the task of the discriminator is to determine whether the image is real or generated. Through adversarial training, the generator can learn the distribution of training data and thus generate images similar to the training data.

[0004] However, GAN also has some problems, such as unstable training, mode collapse, gradient disappearance, etc. In order to solve these problems, researchers have proposed many improved GAN models, such as DCGAN, WGAN, PGGAN, StyleGAN, etc. These models have improved the quality and diversity of generated images to a certain extent, but there are still some limitations, such as limited resolution of generated images, unclear details of generated images, inconsistent semantics of generated images, etc. Summary of the invention

[0005] The purpose of the present invention is to provide a natural scene Tibetan image data enhancement method based on Markov theory, which can avoid the problems existing in GAN and achieve efficient parallelization and scalability, and the generated images have high resolution and diversity.

[0006] The present invention provides a natural scene Tibetan image data enhancement method based on Markov theory, which generates high-quality natural scene Tibetan images similar to training data from random disturbances. In this method, each pixel point in the input image is regarded as a state, and all these states together constitute a state space. At the same time, in the Markov process, the image processing process involves transferring from one state space to another state space. When the state transfer process converges, the method can output the processed image.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] Obtain image information, then preprocess the image, then define a Markov process to add disturbance to the original image, then define an inverse Markov process, and train the neural network to maximize the conditional probability of the inverse Markov process, and finally generate the target image. Specifically, the following steps are included:

[0009] Step S1: Image processing, obtaining image information through field photography, web crawling, and public data set collection, pre-processing the collected Tibetan images, including unifying the size, correcting the text tilt, and then screening images with clear text content and obvious features, and classifying the screened images according to the image features;

[0010] Step S2: text processing, processing the text in the classified image, finding the position of the Tibetan text in the image through edge detection, and then performing text segmentation, including line segmentation, word segmentation and character segmentation, to divide the image into different text units; extracting features of the segmented text units, including structural features, shape features, and stroke features, to describe the form and attributes of the text units; classifying the extracted features, including using machine learning or deep learning models for classification, to identify the category and content of the text units; post-processing the classification results, including using Tibetan grammar rules, Tibetan dictionaries, and Tibetan corpus resources to correct and optimize the recognition results, and obtain the final Tibetan text containing Tibetan text semantic information;

[0011] Step S3: Define a Markov process, the initial state of which is the original training data, i.e., the original Tibetan image. Set a time step sequence for the given training data set. ,in , T represents the total number of steps in the Markov process, and for each time step Set a transfer parameter , and gradually add disturbance information that conforms to a certain distribution law to each training sample, so that the state matrix after the Markov propagation process ends Close to normal distribution;

[0012] Step S4: define an inverse Markov process, traverse the Markov chain in reverse, define a state transition probability matrix to describe the state transition process, predict the next state for a given current state based on the state transition probability, and stop after a specified number of steps, that is, starting from the distribution obtained at the end of the Markov process, the information of the image is gradually restored based on the conditional probability, and finally becomes an image similar to the original training data;

[0013] Step S5: training the neural network model, given the current state space and inputting Tibetan semantic information, generating the next state according to the learned state transition probability, and inputting the semantic information of the characters in the Tibetan image into the neural network model, maximizing the conditional probability of the inverse Markov process through the neural network, and generating a natural scene Tibetan image with relevant features;

[0014] Step S6: Generate an image, use the result obtained after the Markov process as the initial state matrix, then traverse along the Markov chain, use the trained neural network model to decide the Markov state transition probability, and finally obtain the generated image matrix.

[0015] Furthermore, the step S3 specifically includes:

[0016] S301: Given a training data set ,in represents the i-th training sample, Represents the total number of training samples;

[0017] S302: Set a time step sequence ,in , represents the total number of steps of the Markov process;

[0018] S303: For each time step , set a state transition parameter ,satisfy ,and Large enough so that the state after the Markov propagation process ends Close to normal distribution;

[0019] S304: For each training sample , according to the following formula, gradually add disturbance information that conforms to a certain distribution law to obtain the Markov chain :

[0020]

[0021] in is an independent and identically distributed state transition parameter.

[0022] Furthermore, for a single Markov state in the state transition probability matrix and its subsequent status , the state transition probability is defined as:

[0023]

[0024] in, The mean vector is , the covariance matrix is Gaussian distribution, For the initial state The identity matrix of the same dimension, is the state transfer parameter, T is the number of transfer steps, In the state According to the state transition probability matrix, the state is reached probability.

[0025] Furthermore, the step S4 specifically includes:

[0026] S401: Setting up a neural network model ,in Represents the parameters of the neural network model, the input of which is the current state and time step , the output is the state of the previous step The conditional probability distribution of ;

[0027] S402: For each time step , the conditional probability distribution of the inverse Markov process is defined using the following formula, from the state arrive The state transition probability of is distributed according to this probability:

[0028]

[0029] in and Respectively represent the output mean and variance of the neural network model;

[0030] S403: For each training sample , the joint probability distribution of the inverse Markov process is defined using the following formula. This joint probability distribution constitutes the entire inverse Markov process, that is, it describes the process containing all Markov chain of step state transition probability:

[0031]

[0032] in , is a Gaussian distribution, is the identity matrix.

[0033] Furthermore, the neural network model includes a sinusoidal position encoding layer, an encoder, a decoder and a reparameterization layer, wherein the sinusoidal position encoding layer is used to convert the time step Convert to a sinusoidal position code , and the state space matrix splicing, as the input of the encoder; the encoder is used to convert the current state space matrix and time step Encoded as a latent vector ; The decoder is used to transform the latent vector Decode the image into the previous step The mean and variance ; Reparameterization layer, used to get the mean output from the neural network model and variance Sampling a state matrix , as the output of the decoder.

[0034] Furthermore, the step S6 specifically includes:

[0035] S601: The result obtained after the Markov process is processed is used as the initial state matrix ;

[0036] S602: From the time step sequence Iterate through each time step in turn , and do the following:

[0037] S6021: The current state space matrix and time step As the input of the neural network model, get the image of the previous step The conditional probability distribution of ;

[0038] S6022: Sample a state matrix from the conditional probability distribution , which is the state of the next step;

[0039] S603: After T steps, the final image is generated and the final generated image is returned .

[0040] The present invention adopts the Markov model to realize Tibetan image data enhancement. The Markov model is a generative model that can generate high-quality images similar to the training data. The Markov model includes two processes: the forward process and the reverse process. The forward process gradually adds disturbance information that conforms to a certain distribution law to the training samples, so that the state matrix after the Markov propagation process ends is Close to normal distribution, the reverse process defines a state transition probability matrix to describe the state transfer process. Given the current state, the next state can be predicted according to the transition probability matrix to form an inverse Markov chain, and a neural network model is used to learn the Markov state transition probability and gradually restore the original image. The use of this model can avoid common adversarial training, mode collapse, gradient disappearance and other problems in the generation model, and at the same time can achieve efficient parallelization and scalability. When generating images, more detailed information can be captured, thereby generating more realistic and delicate images. At the same time, this method is relatively stable during the training process, and is not prone to problems such as gradient disappearance or explosion, making the model easier to train and optimize. This method also has the characteristics of strong controllability. By adjusting parameters such as step size and perturbation intensity, fine control of the generated image can be achieved to meet the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of a method for enhancing natural scene Tibetan image data based on Markov theory provided by an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of a Markov process provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of an inverse Markov process provided by an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of the structure of a neural network model provided by an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of an image generation step provided by an embodiment of the present invention;

[0047] Figure 6 It is an example diagram of the image generation experiment results provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The present invention relates to a natural scene Tibetan image data enhancement method based on Markov theory, which can identify Tibetan semantic information and generate high-quality multi-style images similar to training data from random perturbations. Markov process is a probabilistic statistical method suitable for a variety of tasks such as image segmentation and texture migration. In this method, each pixel in the input image is regarded as a state, and all these states together constitute a state space. At the same time, in the Markov process, the image processing process involves the system transferring from one state space to another. When the state transfer process converges, the method can output the processed image. Image generation is an important research direction in the field of computer vision and machine learning. Its goal is to generate realistic images based on given conditions or unconditionally. Image generation has many application scenarios, such as image restoration, image enhancement, image style transfer, image animation, image editing, etc.

[0050] Figure 1 The present invention provides a flow chart of a method for enhancing data of a natural scene Tibetan image based on Markov theory. As shown in the figure, the method comprises the following steps:

[0051] Step S1: Image processing, image information is obtained through field photography, web crawler acquisition, public data set collection, etc., and the collected Tibetan images are preprocessed, including unifying the size, correcting the text tilt, etc., and then the images with clear text content and obvious features are screened, and the screened images are classified into scenes according to the image features, such as social media, city streets, rural scenery, plateau snow mountains, grasslands and lakes. Different categories of scenes have different brightness, contrast, color, etc.

[0052] Step S2: text processing, processing the text in the classified image, finding the position of the Tibetan text in the image through edge detection, and then performing text segmentation, including line segmentation, word segmentation and character segmentation, to divide the image into different text units; extracting features of the segmented text units, including structural features, shape features, stroke features, etc., to describe the morphology and attributes of the text units; classifying the extracted features, including using machine learning or deep learning models such as support vector machines, random forests, and convolutional neural networks to identify the category and content of the text units; post-processing the classification results, including using Tibetan grammar rules, Tibetan dictionaries, Tibetan corpora and other resources to correct and optimize the recognition results, and obtaining the final Tibetan text containing Tibetan text semantic information for supervising the neural network training in subsequent steps.

[0053] Step S3: Define a Markov process. The initial state of the process is the original training data, that is, the original image containing Tibetan. In the model, each state is an image and each moment is a time step. Set a time step sequence for a given training data set. ,in , T represents the total number of steps in the Markov process, and for each time step Set a transfer parameter , gradually add disturbance information that conforms to a certain distribution law to each training sample, so that the state matrix after the Markov propagation process ends Close to the normal distribution, we can get the Markov chain Markov chain is a random process, which is characterized by the fact that the current state is only related to the state of the previous moment, but has nothing to do with the previous historical state. The significance of this process is that, starting from the original training data, the information of the image is gradually lost, and the convergence result of the Markov process is close to a state of complete disturbance information.

[0054] Step S4: Define the inverse Markov process, traverse the Markov chain backwards, and prepare for the neural network model to learn the probability distribution and finally obtain the generated image. Define the state transition probability matrix P to describe the state transition process. The state transition probability matrix is ​​a collection of multiple state transition probabilities. For a given current state, predict the next state based on the state transition probability until the specified number of steps are performed and stop. For a single Markov state in the state transition probability matrix and its subsequent status , the state transition probability is defined as:

[0055]

[0056] in, The mean vector is , the covariance matrix is Gaussian distribution, For the initial state The identity matrix of the same dimension, is the state transfer parameter, T is the number of transfer steps, In the state According to the state transition probability matrix, the state is reached The probability of forming an inverse Markov chain. The significance of this process is that, starting from the distribution obtained at the end of the Markov process, the information of the image is gradually restored according to the conditional probability, and finally becomes an image similar to the original training data. The purpose of this is to allow the neural network model to learn the distribution of the training data, so that it can reversely generate images similar to the training data.

[0057] Step S5: training the neural network model, given the current state space and inputting the Tibetan semantic information, generating the next state according to the learned state transition probability, and inputting the semantic information of the characters in the Tibetan image into the neural network model, maximizing the conditional probability of the inverse Markov process through the neural network, generating a natural scene Tibetan image with relevant features, and achieving the purpose of data enhancement.

[0058] Step S6: Generate an image, use the result obtained after the Markov process as the initial state matrix, then traverse along the Markov chain, use the trained neural network model to decide the Markov state transition probability, and finally obtain the generated image matrix.

[0059] Figure 2 is a schematic diagram of the Markov process provided by the present invention. As shown in the figure, the process includes the following steps:

[0060] S301: Given a training data set ,in represents the i-th training sample, Represents the total number of training samples;

[0061] S302: Set a time step sequence ,in , represents the total number of steps of the Markov process;

[0062] S303: For each time step , set a state transition parameter ,satisfy ,and Large enough so that the state after the Markov propagation process ends Follows normal distribution;

[0063] S304: For each training sample , according to the following formula, gradually add disturbance information that conforms to a certain distribution law to obtain the Markov chain :

[0064]

[0065] in is an independent and identically distributed state transition parameter.

[0066] Figure 3 is a schematic diagram of the inverse Markov process provided by the present invention. As shown in the figure, the process includes the following steps:

[0067] S401: Setting up a neural network model ,in Represents the parameters of the neural network model, the input of which is the current state and time step , the output is the state of the previous step The conditional probability distribution of ;

[0068] S402: For each time step , the conditional probability distribution of the inverse Markov process is defined using the following formula, from the state arrive The state transition probability of is distributed according to this probability:

[0069]

[0070] in and Respectively represent the output mean and variance of the neural network model;

[0071] S403: For each training sample , the joint probability distribution of the inverse Markov process is defined using the following formula. This joint probability distribution constitutes the entire inverse Markov process, that is, it describes the process containing all Markov chain of step state transition probability:

[0072]

[0073] in , is a Gaussian distribution, is the identity matrix.

[0074] Figure 4 is a structural diagram of the neural network model provided by the present invention. As shown in the figure, the model includes a sinusoidal position encoding layer, an encoder, a decoder and a re-parameterization layer. The sinusoidal position encoding layer is used to convert the time step Convert to a sinusoidal position code , and the state space matrix splicing, as the input of the encoder; the encoder is used to convert the current state space matrix and time step Encoded as a latent vector ; The decoder is used to transform the latent vector Decode the image into the previous step The mean and variance ; Reparameterization layer, used to get the mean output from the neural network model and variance Sampling a state matrix , as the output of the decoder.

[0075] Figure 5 is a schematic diagram of the image generation step provided by the present invention. As shown in the figure, the step includes the following operations:

[0076] S601: The result obtained after the Markov process is processed is used as the initial state matrix ;

[0077] S602: From the time step sequence Iterate through each time step in turn , and do the following:

[0078] S6021: The current state space matrix and time step As the input of the neural network model, get the image of the previous step The conditional probability distribution of ;

[0079] S6022: Sample a state matrix from the conditional probability distribution , which is the state of the next step;

[0080] S603: After T steps, the final image is generated and the final generated image is returned .

[0081] Figure 6 is an example diagram of the image generation experimental results of the present invention. As shown in the figure, the figure shows images generated from different data sets using the method of the present invention, including MNIST, CIFAR-10, CelebA, LSUN and other data sets. It can be seen from the figure that the method of the present invention can generate high-quality images similar to the training data, with high resolution and diversity.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A natural scene Tibetan image data enhancement method based on Markov theory, characterized in that: Generating high-quality Tibetan images similar to the training data from random perturbations involves the following steps: Step S1: Image processing, obtaining image information through field photography, web crawling, and public data set collection, pre-processing the collected Tibetan images, including unifying the size, correcting the text tilt, and then screening images with clear text content and obvious features, and classifying the screened images according to the image features; Step S2: text processing, processing the text in the classified image, finding the position of the Tibetan text in the image through edge detection, and then performing text segmentation, including line segmentation, word segmentation and character segmentation, to divide the image into different text units; extracting features of the segmented text units, including structural features, shape features, and stroke features, to describe the form and attributes of the text units; classifying the extracted features, including using machine learning or deep learning models for classification, to identify the category and content of the text units; post-processing the classification results, including using Tibetan grammar rules, Tibetan dictionaries, and Tibetan corpus resources to correct and optimize the recognition results, and obtain the final Tibetan text containing Tibetan text semantic information; Step S3: Define a Markov process, the initial state of which is the original training data, i.e., the original Tibetan image. Set a time step sequence for the given training data set. ,in , T represents the total number of steps in the Markov process, and for each time step Set a transfer parameter , and gradually add disturbance information to each training sample; Step S4: define an inverse Markov process, traverse the Markov chain in reverse, define a state transition probability matrix to describe the state transition process, predict the next state for a given current state based on the state transition probability, and stop after a specified number of steps, that is, starting from the distribution obtained at the end of the Markov process, the information of the image is gradually restored based on the conditional probability, and finally becomes an image similar to the original training data; Step S5: training the neural network model, given the current state space and inputting Tibetan semantic information, generating the next state according to the learned state transition probability, and inputting the semantic information of the characters in the Tibetan image into the neural network model, maximizing the conditional probability of the inverse Markov process through the neural network, and generating a natural scene Tibetan image with relevant features; Step S6: Generate an image, use the result obtained after the Markov process as the initial state matrix, then traverse along the Markov chain, use the trained neural network model to decide the Markov state transition probability, and finally obtain the generated image matrix.

2. The method according to claim 1, characterized in that The step S3 further comprises: S301: Given a training data set ,in represents the i-th training sample, Represents the total number of training samples; S302: Set a time step sequence ,in , represents the total number of steps of the Markov process; S303: For each time step , set a state transition parameter ,satisfy ; S304: For each training sample , gradually add disturbance information according to the following formula to obtain the Markov chain : ， in is an independent and identically distributed state transition parameter.

3. The method according to claim 1, characterized in that For a single Markov state in the state transition probability matrix and its subsequent status , the state transition probability is defined as: ， in, The mean vector is , the covariance matrix is Gaussian distribution, For the initial state The identity matrix of the same dimension, is the state transfer parameter, T is the number of transfer steps, In the state According to the state transition probability matrix, the state is reached probability.

4. The method according to claim 1, characterized in that: The step S4 further comprises: S401: Setting up a neural network model ,in Represents the parameters of the neural network model, the input of which is the current state and time step , the output is the state of the previous step The conditional probability distribution of ; S402: For each time step , the conditional probability distribution of the inverse Markov process is defined using the following formula, from the state arrive The state transition probability of is distributed according to this probability: ， in and ) represent the output mean and variance of the neural network model respectively; S403: For each training sample , the joint probability distribution of the inverse Markov process is defined using the following formula, which constitutes the entire inverse Markov process: ， in , is a Gaussian distribution, is the identity matrix.

5. The method according to claim 1, characterized in that: The neural network model includes a sinusoidal position encoding layer, an encoder, a decoder and a re-parameterization layer. The sinusoidal position encoding layer is used to convert the time step Convert to a sinusoidal position code , and the state space matrix splicing, as the input of the encoder; the encoder is used to convert the current state space matrix and time step Encoded as a latent vector ; The decoder is used to transform the latent vector Decode the image into the previous step The mean and variance ); a reparameterized layer for averaging the output of the neural network model and variance ) to sample a state matrix , as the output of the decoder.

6. The method according to claim 1, characterized in that The step S6 further comprises: S601: The result obtained after the Markov process is processed is used as the initial state matrix ; S602: From the time step sequence Iterate through each time step in turn , and do the following: S6021: The current state space matrix and time step As the input of the neural network model, get the image of the previous step The conditional probability distribution of ; S6022: Sample a state matrix from the conditional probability distribution , which is the state of the next step; S603: After T steps, the final image is generated and the final generated image is returned .

Images