Method, device and electronic equipment for generating a stylized description of an image

Abstract

The application provides a kind of generation method, device and electronic equipment of the stylized description of image, belong to educational informatization technical field, method includes: obtaining image sample and the stylized description text corresponding to image sample, and according to different style, stylized description text is divided into different categories;According to image sample and corresponding stylized description text, training set and test set are constructed;Based on the training set and the test set, the method of K-fold cross validation is used to train the pre-constructed stylized description model, to obtain the trained stylized description model;The image to be described is input into the trained stylized description model, and the stylized description of the image to be described is output.The application carries out marking processing to stylized text through pre-constructed stylized description model, produces specific stylized description, and hearing-impaired children can understand the content of the picture through the text description of the picture, which is beneficial to the healthy development of students' mind.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of educational informatization, and in particular to a method and device for generating stylized descriptions of images and electronic equipment. BACKGROUND

[0002] Hearing-impaired children are children whose hearing function is impaired due to congenital or acquired reasons. These children usually need to communicate and learn through sign language, spoken language, hearing aids, and other means.

[0003] In order to cultivate the cognitive ability of hearing-impaired children, some educators and researchers have begun to explore picture description techniques for hearing-impaired children. In picture descriptions for hearing-impaired children, specific and colorful images are often used, which can stimulate the interest of hearing-impaired children in learning and improve their enthusiasm.

[0004] Since hearing-impaired children may not be able to hear or understand spoken language, some auxiliary tools need to be used when communicating with them, and written language is used to convey emotions. Therefore, there is an urgent need for a method for generating stylized descriptions of images to generate stylized descriptions of images. SUMMARY

[0005] The present application provides a method and device for generating stylized descriptions of images and electronic equipment to generate stylized descriptions of images, so that hearing-impaired children can understand the content of the picture and various stylized descriptions of the image through the text description of the image, which is beneficial to the healthy development of the mind of hearing-impaired children.

[0006] In a first aspect, the present application provides a method for generating stylized descriptions of images, comprising: obtaining image samples and stylized description texts corresponding to the image samples, and dividing the stylized description texts into different categories according to different styles; constructing a training set and a test set according to the image samples and the stylized description texts corresponding to the image samples; training a pre-constructed stylized description model based on the training set and the test set using a K-fold cross-validation method to obtain a trained stylized description model; inputting an image to be described into the trained stylized description model to output a stylized description of the image to be described; the stylized description model comprises an encoder network and a decoder network connected in sequence; the encoder network is used to extract image features of an input image; the decoder network is used to output a stylized description of the input image according to the image features; the image samples are images for learning by hearing-impaired children.

[0007] According to the method for generating stylized descriptions of images provided by the present application, the encoder network is established based on a residual neural network, and the decoder network is established based on a bidirectional long short-term memory double-loop network.

[0008] The method for generating a stylized description of an image according to the present application further comprises: performing normalization processing on three channels of each pixel point of the image sample; and performing regularization processing on the three channels of each pixel point of the normalized image sample.

[0009] The method for generating a stylized description of an image according to the present application, wherein the residual neural network is a resnet101 network, and the last two layers of the resnet101 network are replaced by a full connection layer to output image features.

[0010] The method for generating a stylized description of an image according to the present application, wherein the bidirectional long short-term memory double loop network is optimized by using an Adam optimization algorithm, and an activation function of the bidirectional long short-term memory double loop network is a relu function.

[0011] The method for generating a stylized description of an image according to the present application, wherein the method for obtaining the image sample and the stylized description text corresponding to the image sample comprises: crawling the image sample and the stylized description text corresponding to the image sample from a preset website; supplementing missing parts in the crawled image sample and the stylized description text, and deleting redundant data; and associating the processed image sample and the stylized description text by using a json tool.

[0012] The method for generating a stylized description of an image according to the present application, wherein the style categories of the stylized description text at least include: a simple stylized text, a romantic stylized text, and a humorous stylized text.

[0013] In a second aspect, the present application further provides a device for generating a stylized description of an image, comprising:

[0014] An obtaining module is configured to obtain an image sample and a stylized description text corresponding to the image sample, and divide the stylized description text into different categories according to different styles.

[0015] A data set construction module is configured to construct a training set and a test set according to the image sample and the stylized description text corresponding to the image sample.

[0016] A model training module is configured to train a pre-constructed stylized description model by using a K-fold cross-validation method based on the training set and the test set, and obtain a trained stylized description model.

[0017] The style description generation module is configured to input an image to be described into the trained style description model, and output a style description of the image to be described.

[0018] The style description model comprises an encoder network and a decoder network connected in sequence; the encoder network is configured to extract image features of an input image; and the decoder network is configured to output a style description of the input image according to the image features.

[0019] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for generating a style description of an image according to any one of the above aspects when executing the program.

[0020] In a fourth aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the method for generating a style description of an image according to any one of the above aspects.

[0021] The method and device for generating a style description of an image and the electronic device provided by the present application can mark style text by using a pre-constructed style description model, and generate a specific style description, so that hearing-impaired children can understand the content of a picture through the text description of the picture, and various style descriptions of the picture are conducive to the healthy development of students' minds.

[0022] Further, the style description model provided by the present application is trained by K-fold cross-validation, so that the training of the style description model can be realized by using a small amount of data, and the style description model can generate more appropriate and reasonable style descriptions. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0024] Figure 1 is a flowchart of the method for generating a style description of an image provided by the present application;

[0025] Figure 2 is a structural diagram of the device for generating a style description of an image provided by the present application;

[0026] Figure 3 is a structural diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the drawings in the present application. Obviously, the described embodiments are only a part of embodiments of the present application, rather than all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall into the scope of protection of the present application.

[0028] It should be noted that in the description of the embodiments of the present application, the terms “comprising”, “containing” or any other variants thereof are intended to cover the non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitation, the element defined by the sentence “including a…” does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0029] The following will be described with reference to the drawings Figures 1-3 The method for generating the stylized description of the image, the device and the electronic equipment provided by the embodiments of the present application are described.

[0030] Figure 1 The flowchart of the method for generating the stylized description of the image provided by the present application is shown in FIG. 1, which includes but is not limited to the following steps: Figure 1

[0031] Step 101: Obtain image samples and stylized description texts corresponding to the image samples, and divide the stylized description texts into different categories according to different styles.

[0032] The image samples are images for learning of hearing-impaired children. For teaching images (i.e. images for learning of hearing-impaired children), specific and colorful images are often collected, which can stimulate the learning interest of hearing-impaired children and improve their enthusiasm. The stylized description texts should use common and simple font words as much as possible, and the sentences in the texts are preferably short sentences and do not use rhetorical devices as much as possible. In this way, the short and simple stylized description texts are easy for hearing-impaired children to understand, and can improve their learning enthusiasm.

[0033] Optionally, the categories of the stylized description texts include simple stylized texts, romantic stylized texts and humorous stylized texts.

[0034] Step 102: Construct a training set and a test set according to the image samples and the stylized description texts corresponding to the image samples. ​

[0035] Step 103: based on the training set and the test set, a pre-constructed stylized description model is trained by using the method of K-fold cross validation to obtain a trained stylized description model.

[0036] The stylized description model comprises a sequentially connected encoder network and a decoder network; the encoder network is used to extract image features of an input image; and the decoder network is used to output a stylized description of the input image according to the image features.

[0037] The collected image samples and the corresponding stylized description texts of the image samples are divided into several parts, each part is used as a test set, and the other parts are used as training sets; and the training set data is input into the stylized description model (i.e., a stylized picture description model) for model training.

[0038] Specifically, for the picture samples in the training set, an encoder network (usually a convolutional neural network, CNN) is used to extract rich visual semantic information in the picture, i.e., image feature extraction. The decoder network usually uses a long short-term memory neural network (LSTM), which uses the information timing provided by the encoder network to generate accurate natural language descriptions.

[0039] Further, in the case of a small amount of data, the K-fold cross validation method can be used to expand the data. The advantage of K-fold cross validation is that randomly generated sub-samples are simultaneously used for training and verification, and 10-fold cross validation is the most commonly used.

[0040] Step 104: inputting a to-be-described image into the trained stylized description model to output a stylized description of the to-be-described image.

[0041] The present application trains the stylized description model by K-fold cross validation, which can train the stylized description model with a small amount of data, so that the stylized description model can generate more appropriate and reasonable stylized descriptions, so that hearing-impaired children can understand the content of the picture through the text description of the picture, and various stylized descriptions of the picture are beneficial to the healthy development of students' minds.

[0042] Based on the content of the above embodiment, as an optional embodiment, the present application provides a generation method of stylized description of an image, wherein the image samples and the corresponding stylized description texts of the image samples are obtained by crawling the image samples and the corresponding stylized description texts of the image samples from a preset website; the missing parts in the crawled image samples and the stylized description texts are supplemented, and redundant data is deleted; and the image samples and the stylized description texts after processing are associated by using a json tool.

[0043] The preset website can be a website related to special education. In an embodiment, the present application ensures that each image sample has five different simple style descriptions, at least one humorous style description, and at least one romantic style description; then uses a json tool to associate the image and the image-related style text description; and the obtained jaon file format is as follows:

[0044]

[0045] For the part of data missing, the corresponding text description can be supplemented; and for the redundant data, a deletion operation can be performed.

[0046] Based on the content of the above embodiment, as an optional embodiment, the present application provides a generation method of style description of an image, and before constructing a training set and a test set according to the image sample and the style description text corresponding to the image sample, the method further comprises: performing normalization processing on three channels of each pixel point of the image sample respectively; and performing regularization processing on the three channels of each pixel point of the image sample after the normalization processing respectively.

[0047] The embodiment is a pre-processing process of the image sample. It can be understood that the value of the pixel point of the sample image before the pre-processing is between 0 and 255. In order to accelerate the speed of network training, the present application performs normalization processing and regularization processing on the image sample (belonging to an RGB image), so that the three channels of the pixel point of the RGB image are mapped to the interval range of -1 to 1. Taking the r channel as an example, the corresponding value of the r channel after the normalization is r j :

[0048]

[0049] wherein, r max , r min represent the maximum value and the minimum value of the r channel, and r j is the normalized value of the r channel.

[0050] Supposing that the corresponding value of the normalized r channel after the regularization is r k , the regularization operation is:

[0051]

[0052] wherein, r mean , r std respectively represent the mean value and the standard deviation on the normalized r channel, and r k represents the value after the regularization.

[0053] The g, b channels are similar, so that the point value corresponding to each channel is distributed between-1 and 1.

[0054] Further, the stylized description text is divided into different categories according to three different styles, and a stylized vector label is added after the stylized text description according to different categories, which is specifically represented as:

[0055] [a1, a2, a3, a4, …, a L , S]

[0056] Where a i , i∈(1, L) represents the word vector in the stylized description text, L represents the length of the stylized description text, and S represents the stylized vector corresponding to the stylized description text.

[0057] Based on the content of the above embodiment, as an optional embodiment, the encoder network in the application is established based on a residual neural network (ResNet), and the decoder network is established based on a bidirectional long short-term memory double loop network (BiLstm). It can be understood that the application uses a residual bidirectional long short-term memory neural network (ResNet-BiLstm) to establish a stylized description model.

[0058] Specifically, the application uses ResNet as the encoder of the stylized description model, and uses BiLstm with stylized memory as the decoder of the stylized description model. The preprocessed image sample is used as the input of the encoder ResNet, and the output result is used to initialize the BiLstm. At each time step t of the BiLstm, the output y t , s t , according to s t , the stylized loss of time step t is calculated:

[0059] L(y t , S) = -I(s t , S) log (p(s t = S | y t ))

[0060] Where y t , s t respectively represent the hidden state and the style state in the bidirectional long short-term memory neural network. y t , s t will be used as the input of the BiLstm at the next time step t+1.

[0061] For image samples in the training set data and the test set data, a residual neural network (resnet101) is used for feature extraction; the last two layers of the resnet101 are replaced with a fully connected layer to output the features; a BiLstm network is entered, an Adam algorithm optimization is adopted, and finally a fully connected network is used, and a relu activation function is adopted to obtain the trained stylized description model.

[0062] If the loss on each test set and training set has converged, it represents that the stylized description model training is completed.

[0063] The present application uses image samples and stylized description text pairs to train ResNet and BiLstm networks respectively, ResNet network extracts features from the spatial dimension of the image by using a large number of convolution operations, and BiLstm predicts words at each time step according to the features extracted by ResNet, obtains the stylized description of the image, and the hearing-impaired children can understand the content of the picture through the text description of the picture, and various stylized descriptions of the picture are beneficial to the healthy development of the students' mind.

[0064] Figure 2 It is a structural schematic diagram of the image stylized description generation device provided by the present application, as shown in Figure 2 The device comprises an acquisition module 201, a data set construction module 202, a model training module 203 and a stylized description generation module 204.

[0065] The acquisition module 201 is used for acquiring image samples and corresponding stylized description texts of the image samples, and dividing the stylized description texts into different categories according to different styles.

[0066] The data set construction module 202 is used for constructing a training set and a test set according to the image samples and the corresponding stylized description texts of the image samples.

[0067] The model training module 203 is used for training a pre-constructed stylized description model based on the training set and the test set by using a K-fold cross-validation method, and obtaining a trained stylized description model.

[0068] The stylized description generation module 204 is used for inputting a to-be-described image into the trained stylized description model, and outputting a stylized description of the to-be-described image.

[0069] The stylized description model comprises an encoder network and a decoder network connected in sequence; the encoder network is used for extracting image features of an input image; and the decoder network is used for outputting a stylized description of the input image according to the image features.

[0070] It should be noted that the image stylized description generation device provided by the embodiment of the present application can execute the image stylized description generation method of any of the above-mentioned embodiments when actually running, and the embodiment will not be repeated here.

[0071] Figure 3 is a structural schematic diagram of an electronic device provided by the present application, as Figure 3 shown, the electronic device can include a processor 310, a communications interface 320, a memory 330 and a communications bus 340, wherein the processor 310, the communications interface 320 and the memory 330 complete mutual communication through the communications bus 340. The processor 310 can call the logic instructions in the memory 330 to execute the image stylized description generation method, which includes: obtaining image samples and stylized description texts corresponding to the image samples, and dividing the stylized description texts into different categories according to different styles; constructing a training set and a test set according to the image samples and the stylized description texts corresponding to the image samples; training a pre-constructed stylized description model based on the training set and the test set by using a K-fold cross-validation method, obtaining a trained stylized description model; inputting a to-be-described image into the trained stylized description model, and outputting a stylized description of the to-be-described image; the stylized description model includes an encoder network and a decoder network connected in sequence; the encoder network is used to extract image features of an input image; the decoder network is used to output a stylized description of the input image according to the image features; and the image samples are images for learning by hearing-impaired children.

[0072] In addition, the logic instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0073] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method for generating a stylized description of an image provided by any of the above embodiments, the method comprising: obtaining image samples and stylized description texts corresponding to the image samples, and dividing the stylized description texts into different categories according to different styles; constructing a training set and a test set according to the image samples and the stylized description texts corresponding to the image samples; training a pre-constructed stylized description model based on the training set and the test set by using a K-fold cross-validation method, to obtain a trained stylized description model; inputting an image to be described into the trained stylized description model, and outputting a stylized description of the image to be described; the stylized description model comprises an encoder network and a decoder network connected in sequence; the encoder network is configured to extract image features of an input image; the decoder network is configured to output a stylized description of the input image according to the image features; and the image samples are images for learning by hearing-impaired children.

[0074] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method for generating a stylized description of an image provided by any of the above embodiments, the method comprising: obtaining image samples and stylized description texts corresponding to the image samples, and dividing the stylized description texts into different categories according to different styles; constructing a training set and a test set according to the image samples and the stylized description texts corresponding to the image samples; training a pre-constructed stylized description model based on the training set and the test set by using a K-fold cross-validation method, to obtain a trained stylized description model; inputting an image to be described into the trained stylized description model, and outputting a stylized description of the image to be described; the stylized description model comprises an encoder network and a decoder network connected in sequence; the encoder network is configured to extract image features of an input image; the decoder network is configured to output a stylized description of the input image according to the image features; and the image samples are images for learning by hearing-impaired children.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0076] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating stylized descriptions of images, characterized in that, include: Obtain image samples and their corresponding stylized descriptive text, and classify the stylized descriptive text into different categories based on different styles; The image samples are images used for learning by hearing-impaired children; Based on the image samples and the corresponding stylized descriptive text, a training set and a test set are constructed. Based on the training set and the test set, the pre-built stylistic description model is trained using the K-fold cross-validation method to obtain the trained stylistic description model. The image to be described is input into the trained stylization description model, and the stylization description of the image to be described is output. The stylization description model includes an encoder network and a decoder network connected in sequence; the encoder network is used to extract image features of the input image; the decoder network is used to output a stylization description of the input image based on the image features. The encoder network is based on the residual neural network ResNet, and the decoder network is based on the bidirectional long short-term memory dual recurrent network BiLstm. The stylistic description model is established using the residual bidirectional long short-term memory neural network ResNet-BiLstm. The preprocessed image samples are used as input to the ResNet encoder, and the output is used to initialize the BiLstm. At each time step of the BiLstm... Output ,according to Calculate time steps Stylization loss: in, These represent the hidden state and style state in a bidirectional long short-term memory neural network, respectively. It will be at the next time step +1 is used as the input to BiLstm; For image samples in the training and test sets, a residual neural network (ResNet101) is used for feature extraction. The last two layers of ResNet101 are replaced with fully connected layers to output the features. The samples are then fed into a BiLSTM network, where the Adam algorithm is optimized using parameter optimization and adaptation. Finally, a fully connected network is used with ReLU as the activation function to obtain the trained stylistic description model.

2. The method for generating stylized descriptions of images according to claim 1, characterized in that, The acquisition of image samples and corresponding stylized descriptive text includes: The image samples and their corresponding stylized descriptive text are crawled from a preset website. The missing parts of the crawled image samples and the stylized descriptive text are supplemented, and redundant data is deleted. The processed image samples and the stylized descriptive text are associated using a JSON tool.

3. The method for generating stylized descriptions of images according to claim 1, characterized in that, Before constructing the training set and test set based on the image samples and the corresponding stylized descriptive text, the process further includes: Normalize the three channels of each pixel in the image sample respectively; Regularization is performed on the three channels of each pixel in the normalized image sample.

4. The method for generating stylized descriptions of images according to claim 1, characterized in that, The stylized descriptive text styles include at least: minimalist stylized text, romantic stylized text, and humorous stylized text.

5. An apparatus for generating stylized descriptions of images, characterized in that, A method for generating a stylized description of an image as described in any one of claims 1 to 4, comprising: The acquisition module is used to acquire image samples and the stylized description text corresponding to the image samples, and to classify the stylized description text into different categories according to different styles; The dataset construction module is used to construct training and testing sets based on the image samples and the stylized descriptive text corresponding to the image samples. The model training module is used to train the pre-built stylistic description model based on the training set and the test set using the K-fold cross-validation method, and obtain the trained stylistic description model. The stylistic description generation module is used to input the image to be described into the trained stylistic description model and output the stylistic description of the image to be described. The stylization description model includes an encoder network and a decoder network connected in sequence; the encoder network is used to extract image features from the input image; the decoder network is used to output a stylization description of the input image based on the image features.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for generating a stylized description of an image as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for generating a stylized description of an image as described in any one of claims 1 to 4.

Images