Comic bubble recognition training, response method and its device, equipment and medium

Abstract

The present application discloses a comic bubble recognition training and response method and its device, equipment, and medium, wherein the training method comprises: extracting a bubble feature map from a comic original image using an image segmentation model used by a generator of a generative adversarial model; fusing the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map respectively by a fusion network in the generator, and correspondingly generating a synthetic map of a false value type and a true value type; feeding the synthetic map into the discriminator of the generative adversarial model for training, so that it has the ability to identify the true and false categories to which the bubble features contained in the image belong; using the trained discriminator as the basis for distinguishing the true and false categories of the synthetic map output by the generator, training the image segmentation model, so that it improves the accuracy of extracting bubble features from comic original images. The present application can train an image segmentation model suitable for segmenting a bubble highlight map containing its bubble features from a comic image with a small number of samples.

Description

Technical Field

[0001] The present application belongs to image recognition technology, and specifically relates to a comic bubble recognition training and response method and its corresponding comic bubble recognition and response device, as well as related electronic equipment and non-volatile storage media. Background Art

[0002] The technology of using convolutional neural networks for image recognition based on artificial intelligence has matured. As we all know, the realization of artificial intelligence depends on big data and its algorithms. In other words, an effective neural network often depends on the sparsity and total number of its training samples.

[0003] In an application of "bubble recognition", the speech bubbles in comics are highlighted by using computer vision recognition technology. This method is mainly used to assist optical character recognition (OCR) in the comics field to improve the recognition accuracy of OCR in the comics field.

[0004] Traditional bubble recognition algorithms are based on "connected-components" technology. Although the algorithm is simple, it requires defining many stringent conditions (for example, the bubbles must be white and have a specific shape) to identify the bubbles relatively accurately. Therefore, it is difficult to play a role in the engineering field.

[0005] With the widespread application of deep learning in the field of computer vision recognition, the end-to-end "bubble recognition" algorithm based on U-net has also emerged. U-net is an image segmentation / highlighting algorithm, which is mainly used in the field of medical impact. It was originally widely used in tasks such as highlighting cells and tumors. In 2019, David Dubray and Jochen Laubrock published a paper (Dubray, David, and Jochen Laubrock. Deep CNN-based speech balloon detection and segmentation for comic books. 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2019. URL), applying U-net technology to bubble recognition. Aiming at the needs of bubble recognition, an image segmentation model based on U-net was proposed, which achieved good results with the powerful fitting ability of U-net. At the same time, its own end-to-end black box characteristics also reduce the complexity of the bubble extraction process.

[0006] Nevertheless, since training and converging a U-net model requires a large amount of labeled data, the measured data proposed by David Debray and Jochen Laubrock also showed that even with 750 samples, it was difficult to converge the image segmentation model they proposed. Therefore, the training cost of producing this image segmentation model is significantly higher than the traditional method. It can be seen that how to continuously optimize bubble recognition related technologies and further reduce the total number of samples for training related models and thus reduce their production costs has positive significance for this field. Summary of the invention

[0007] One of the purposes of this application is to provide a comic bubble recognition training method and its corresponding devices, electronic devices, and non-volatile storage media.

[0008] Accordingly, one of the objectives of the present application is to further provide a comic bubble recognition response method and its respective corresponding devices, electronic devices, and non-volatile storage media.

[0009] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0010] A comic bubble recognition training method provided for one of the purposes of the present application comprises the following steps:

[0011] The image segmentation model used by the generator of the generative adversarial model is used to extract a bubble feature map containing its bubble features from the comic original image;

[0012] The fusion network in the generator fuses the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map, respectively, to generate synthetic images of false value type and true value type respectively;

[0013] Feeding the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong;

[0014] The trained discriminator is used as the basis for distinguishing the true and false categories of the synthetic images output by the generator, and the image segmentation model of the generator is trained to improve the accuracy of extracting bubble features from the original comic image.

[0015] In some embodiments, the process of extracting a bubble feature map containing bubble features from the comic original image by the image segmentation model includes:

[0016] Feeding the comic original image into the input convolution layer of the image segmentation model constructed based on the U-Net neural network to extract local features and obtain an intermediate feature map;

[0017] The intermediate feature map is encoded step by step through multiple encoders in the encoding path of the image segmentation model to extract multi-scale features;

[0018] The multi-scale features are used as input features, and are gradually decoded by multiple decoders in the decoding path of the image segmentation model to be restored to feature maps with higher resolutions step by step, wherein each decoder uses the feature map obtained by a corresponding encoder as a reference to implement the restoration;

[0019] All the outputs of the decoders are fused to obtain a bubble feature map containing the bubble features of the original comic image.

[0020] In some embodiments, the process of fusing the comic original image with the bubble feature image by the fusion network includes:

[0021] Receive the bubble feature map and its corresponding comic original image, wherein the bubble feature map is derived from the output of the image segmentation model or other external storage address;

[0022] A single neuron composed of a neural network is used to perform linear operations on the weight parameters associated with the original cartoon image and the bubble feature image, respectively, so as to merge the two into a same composite image;

[0023] The composite image is output to the discriminator.

[0024] In some embodiments, the training process of the discriminator includes:

[0025] Perform feature extraction on the synthetic image of each category fed in;

[0026] The synthetic image is classified and judged according to the extracted features to obtain the corresponding true value and false value discrimination results, thereby enhancing its discrimination ability;

[0027] Modify the weight parameters used by itself according to the discrimination results;

[0028] Back propagates to the generative adversarial model and corrects the weight parameters of the fusion network and image segmentation model in the generator accordingly.

[0029] In some embodiments, in the step of using the trained discriminator as a basis for distinguishing true from false categories of the synthetic image output by the generator, the discrimination process of the discriminator includes:

[0030] Perform feature extraction on the synthetic image of each category fed in;

[0031] Classify and discriminate the synthetic image according to the extracted features to obtain the corresponding true value and false value discrimination results;

[0032] Freeze the modifications to the weight parameters used by itself;

[0033] According to the discrimination result, back propagation is performed to the generative adversarial model, and the weight parameters of the fusion network and the image segmentation model in the generator are corrected accordingly.

[0034] A comic bubble recognition training device provided for one of the purposes of the present application comprises:

[0035] A bubble segmentation module is used to extract a bubble feature map containing bubble features from the comic original image using the image segmentation model used by the generator of the generative adversarial model;

[0036] An image fusion module, used for fusing the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map by a fusion network in the generator, and generating synthetic images of false value type and true value type respectively;

[0037] A discrimination training module, used for feeding the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong;

[0038] A training module is generated, which is used to use the trained discriminator as a basis for distinguishing true and false categories of the synthetic image output by the generator, and train the image segmentation model of the generator to improve the accuracy of extracting bubble features from the original comic image.

[0039] A comic bubble recognition and response method provided to meet one of the purposes of this application includes the following steps:

[0040] Inputting the comic picture to be recognized into the pre-trained image segmentation model for image segmentation, and obtaining a bubble feature map contained in the comic picture;

[0041] Acquire a text image in the comic picture according to a correspondence between the bubble feature map and the comic picture;

[0042] Optically recognizing the text image to convert it into computer-recognizable text information;

[0043] The image segmentation model is pre-trained by iteratively performing the following specific steps:

[0044] The image segmentation model used by the generator of the generative adversarial model is used to extract a bubble feature map containing its bubble features from the comic original image;

[0045] The fusion network in the generator fuses the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map, respectively, to generate synthetic images of false value type and true value type respectively;

[0046] Feeding the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong;

[0047] The trained discriminator is used as the basis for distinguishing the true and false categories of the synthetic images output by the generator, and the image segmentation model of the generator is trained to improve the accuracy of extracting bubble features from the original comic image.

[0048] In some embodiments, the method further comprises the following subsequent steps:

[0049] A human voice model is called to synthesize a corresponding human voice file for the text information, and the human voice file is associated with the comic picture and stored for synchronous playback with the comic picture.

[0050] A comic bubble recognition and response device is provided to meet one of the purposes of the present application, comprising:

[0051] The bubble segmentation module is used to input the comic picture to be identified into the pre-trained image segmentation model to perform image segmentation and obtain the bubble feature map contained in the comic picture;

[0052] An image and text extraction module, used for obtaining a text image in the comic picture according to a correspondence between the bubble feature map and the comic picture;

[0053] An image-text conversion module, used for optically recognizing the text image to convert it into computer-recognizable text information;

[0054] The image segmentation model is pre-trained by iteratively performing the following specific steps:

[0055] The image segmentation model used by the generator of the generative adversarial model is used to extract a bubble feature map containing its bubble features from the comic original image;

[0056] The fusion network in the generator fuses the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map, respectively, to generate synthetic images of false value type and true value type respectively;

[0057] Feeding the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong;

[0058] The trained discriminator is used as the basis for distinguishing the true and false categories of the synthetic images output by the generator, and the image segmentation model of the generator is trained to improve the accuracy of extracting bubble features from the original comic image.

[0059] An electronic device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the comic bubble recognition training / response method.

[0060] A non-volatile storage medium is provided to meet one of the purposes of the present application, which stores a computer program implemented according to the comic bubble recognition training / response method in the form of computer-readable instructions. When the computer program is called by a computer, the steps included in the method are executed.

[0061] Compared with the prior art, this application has the following advantages:

[0062] This application adopts an image segmentation model to segment the bubble feature map from the comic original image, inheriting the advantages of this architecture. On this basis, combined with the generative adversarial model, on the one hand, after using the image segmentation model to generate the bubble feature map, it is fused with the comic original image with the help of the fusion network provided by this application to obtain a synthetic image, and this type of synthetic image is used as a sample of the false value category; on the other hand, the artificially pre-labeled bubble feature map and the corresponding comic original image are synthesized through the fusion network to obtain a synthetic image as a sample of the true value category. The discriminator of the production adversarial model is first trained with these two types of samples, so that the discriminator obtains the ability to distinguish whether the bubble features contained in the picture are true or false. Then use the trained discriminator as the basis for judgment to assist in training the generator, mainly to train the image segmentation model therein, so that the bubble feature map generated by the image segmentation model can continue to converge to the ideal effect.

[0063] This application combines the image segmentation model with the generative adversarial model, realizes adversarial learning between the generator and the discriminator, and uses the bubble feature map produced by the image segmentation model during the training process to synthesize the comic original image to produce richer samples for training. Finally, the image segmentation model is optimized, and the required comic original image is significantly reduced, so that it can achieve convergence with only a small amount of sample training. It is suitable for extracting bubble feature maps from comic pictures. By feeding the original comic picture into it, it can be segmented to form a highlighted picture representing its bubble contour features, thereby realizing bubble recognition in the comic picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0065] Figure 1 A schematic diagram of a network architecture suitable for deploying the present application;

[0066] Figure 2 A schematic diagram of the principle of an artificial neural network architecture proposed in this application;

[0067] Figure 3 A flowchart of a typical embodiment of the comic bubble recognition training method of the present application;

[0068] Figure 4 Another embodiment of the comic bubble recognition training method of the present application is based on Figure 3 The disclosed step S11, wherein a flowchart of a process performed by the image segmentation model is presented;

[0069] Figure 5 Another embodiment of the comic bubble recognition training method of the present application is based on Figure 3 The disclosed step S12, wherein a flowchart of the process performed by the fusion model is presented;

[0070] Figure 6 Another embodiment of the comic bubble recognition training method of the present application is based on Figure 3 The disclosed step S13, wherein a flow chart of the discriminator during training is presented;

[0071] Figure 7 Another embodiment of the comic bubble recognition training method of the present application is based on Figure 3 The disclosed step S14, wherein the discriminator is a schematic diagram of a process presented in the process of providing a criterion to the generator;

[0072] Figure 8 A flowchart of a typical embodiment of the comic bubble recognition and response method of the present application;

[0073] Fig. 9 A flowchart of another embodiment of the comic bubble recognition and response method of the present application;

[0074] Fig.10 This is a principle block diagram of the comic bubble recognition training device of the present application;

[0075] Fig.11 This is a functional block diagram of the comic bubble recognition and response device of this application. DETAILED DESCRIPTION

[0076] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0077] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0078] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0079] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0080] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0081] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0082] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud, and is mainly used to deploy the generative adversarial model of this application. It is only a server in a logical concept, representing the entire service cluster used to implement the generative adversarial model or to implement comic bubble recognition. By designing an application open call interface, this logical server is used to provide services for related terminal devices such as the smartphone 81 and personal computer 82 shown in the figure. The smartphone and personal computer can access the Internet through a well-known network access method, and establish a data communication link with the server 80 in the cloud, so as to start the comic bubble recognition operation, implement the training of the generative adversarial model and the image segmentation model adopted by it, and use the image segmentation model to perform comic bubble segmentation and recognition operations.

[0083] See also Figure 2 The technical solution of this application is based on Figure 2 The invention is implemented by the architecture shown in the figure, in which a structure combining a generative adversarial model and an image segmentation model is implemented.

[0084] Generative Adversarial Network (GAN) is a generative model proposed by Goodfellow et al. in 2014. The structure of GAN is inspired by the two-person zero-sum game in game theory (i.e., the sum of the interests of the two people is zero, and what one party gains is what the other party loses). The system consists of a generator 90 and a discriminator 92. The generator captures the potential distribution of real data samples and generates new data samples; the discriminator 92 is a binary classifier that determines whether the input is real data or a generated sample. Both the generator 90 and the discriminator 92 can be constructed using deep neural networks. The optimization process of GAN is a minimax game problem. The optimization goal is to achieve Nash equilibrium so that the generator estimates the distribution of data samples.

[0085] The image segmentation model 901 adopts the model based on the U-net architecture proposed by David Dubray and Jochen Laubrock in 2019 as mentioned above, which is suitable for segmenting a comic picture 903 to generate a bubble feature map 906 containing the dialogue bubble features therein, usually expressed as a bubble highlight map with a solid color mask effect.

[0086] The architecture adopted in the present application uses the image segmentation model 903 as the main component of the generator 90, which is used to produce a bubble feature map 906 corresponding to the fed comic original image 903. In order to organically integrate the two models, a neuron implemented by a neural network is added to the generator 90, called a fusion network 902, which is used to fuse the comic original image 903 with its corresponding bubble feature map 906 extracted and produced by the image segmentation model 901 through linear operations, or independently fuse the comic original image 903 with the bubble feature map fed in manually pre-labeling, and correspondingly obtain two types of synthetic images 909, the former type is automatically labeled as a false value type, and the latter type is labeled as a true value type due to manual pre-labeling, so that the true and false image data can be fed into the discriminator 92 of the production adversarial model to train it.

[0087] When the discriminator 92 is trained and finalized, it can be used in turn to train the generator 90, specifically to train the image segmentation model 901 therein, so that the image segmentation model 901 can continuously improve its accuracy in extracting bubble feature maps from the original comic image, and finally complete the training of the image segmentation model 901.

[0088] It can be seen that the above-mentioned network architecture realized based on the technical solution of the present application combines the advantages of U-net and GAN, and organically integrates the two. U-net is responsible for producing some increasingly realistic training samples, and Gan's discriminator 92 is responsible for improving the identification ability, which is equivalent to putting forward higher requirements on U-net. After many iterations, Nash equilibrium is finally reached. The image segmentation model 901 trained in this way can have the ability to accurately extract dialogue bubbles in comic pictures and accurately output their corresponding bubble feature maps.

[0089] On the basis of understanding the relevant application environment and development environment related to the technical solution of the present application disclosed above, please refer to the following description to continue to understand the various implementation methods of the present application. It should be pointed out that those skilled in the art should be aware of this: although the various methods of the present application are described based on the same concept and show commonality with each other, unless otherwise specified, these methods can be executed independently. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept. Therefore, concepts with the same expression, and concepts that are appropriately transformed only for convenience despite different conceptual expressions, should be understood equally.

[0090] Please combine Figure 3 The present invention discloses a comic bubble recognition training method, which is used to train an image segmentation model suitable for segmenting a bubble feature map of a comic image to separate the contained speech bubbles, and is suitable for deployment and implementation in the cloud. The method provides a calling interface to allow external calls to its services to implement related functions, and includes the following iterative execution steps:

[0091] Step S11: extracting a bubble feature map containing bubble features from the comic original image using the image segmentation model used by the generator of the generative adversarial model.

[0092] The generator of the production adversarial model uses the image segmentation model as its production factory. When a comic original image is fed into it, the image segmentation model will output a bubble feature map for it. As we all know, the bubbles in comic images have different shapes and often have tails. Therefore, their bubble features are often irregular. The bubble features referred to here can be understood as the overall outline of the dialogue bubble. To extract an irregular overall outline feature from a comic image, for an untrained image segmentation model, although its own algorithm has a certain summarization ability, the output results obtained in the early stage are naive. Therefore, the quality of the bubble feature map generated by the image segmentation model in the early stage is uneven, so the accuracy of the bubble feature map is obviously not high. Therefore, in the early stage of training the discriminator of the production adversarial model, the bubble feature map generated by the image segmentation model will be regarded as false value type data, and after being synthesized with the corresponding comic original image, it will be used as a false value type training sample for training the discriminator.

[0093] Step S12: The fusion network in the generator fuses the original comic image with the extracted bubble feature map and the pre-labeled bubble feature map, respectively, to generate synthetic images of false value type and true value type.

[0094] In the stage of training the discriminator, according to the GAN principle, two types of training samples need to be provided to the discriminator, one type is pictures representing real data, and the other type is pictures representing pseudo data, so that the discriminator can learn the ability to distinguish the authenticity of pictures from the two types of pictures.

[0095] For the pictures representing real data, the comic original picture and the bubble feature map that matches the bubble features of the comic original picture pre-marked by manual recognition can be fused two-in-one to form a synthetic picture. The synthetic picture represents the real data. Such synthetic pictures are marked as true value types and provided as training samples to the discriminator for training.

[0096] For the picture representing pseudo data, it is generated by the image segmentation model. Therefore, the generator of the production adversarial model automatically regards it as pseudo data, fuses it with its corresponding cartoon original image into a synthetic image, marks it as a false value type to represent pseudo data, and then provides this type of synthetic image as a training sample to the discriminator for training.

[0097] In one embodiment, during the stage of training the discriminator, the degree of match between the bubble feature map produced by the image segmentation model and the real bubble features of the comic original image can be manually intervened or automatically identified, and the bubble feature map judged to have a higher degree of match, that is, a higher accuracy, is marked as a true value type, and the synthetic image generated by fusing it with the comic original image is classified as a real data training sample and provided to the discriminator for training.

[0098] The fusion network can use a linear operation function to achieve the fusion of the bubble feature map and its corresponding comic original image, such as using the following mathematical expression:

[0099] Z=W1T+W2Y unet (1)

[0100] Among them, T is the original picture, that is, the original cartoon picture, Y unet is the output of the image segmentation model (U-net), Z is the final output of the generator, and W1 and W2 are the parameters that the model needs to optimize.

[0101] Step S13: Feed the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that it has the ability to identify the true and false categories to which the bubble features contained in the image belong.

[0102] As mentioned above, all kinds of synthetic images generated by the generator can be fed into the discriminator as training samples to train the discriminator so as to improve the discriminator's ability to distinguish whether the synthetic image contains an accurate bubble feature map or an inaccurate bubble feature map.

[0103] It can be understood that the composite image is a linearly superimposed effect image of a comic original image and a bubble feature map that theoretically matches it (but actually does not match it for the false value type). This effect image contains both the information of the comic original image and the information of the theoretical bubble feature map. Therefore, the process of the discriminator judging the authenticity of this effect image is essentially the process of judging whether the bubble feature map superimposed thereon matches the real dialogue bubble contour features of the comic original image.

[0104] The discriminator is a relatively simple classification model, whose main function is to distinguish the synthetic images of the fake value type generated by the generator and the synthetic images of the true value type manually labeled from the fed synthetic images. The discriminator is usually composed of two parts in structure. The front-end part is the CNN convolution layer, which is used for feature extraction of synthetic images. Generally, Resnet or VGG network is used as its backbone; the back-end part is a fully connected layer, which is responsible for mapping the features extracted by the CNN convolution layer to the [0,1] space for classification. Here, the value 1 represents the true value type and 0 represents the false value type.

[0105] Since the discriminator is a simple two-class model, the cross entropy loss function can meet the requirements. Its loss function is Loss d The expression is:

[0106] Loss d =-∑TlogY (2)

[0107] Among them, T is the model target of the discriminator and Y is the output of the model.

[0108] It can be understood that in order to train the discriminator, steps S11 to S13 can be iterated multiple times. After multiple iterative trainings, the discriminator improves its recognition ability by continuously correcting its own weight parameters, and finally makes the function converge and acquires the ability to recognize the true or false category of the bubble features contained in the picture.

[0109] It should be pointed out that the images that the discriminator can identify after training are not limited to the various images referred to in this application. However, in this specific application scenario of the present application, in order to solve the specific problems of the present application, the discriminator plays the role of providing criteria for the generator. The images that the discriminator can identify after training generally refer to the synthetic images output by the fusion network.

[0110] Step S14: using the trained discriminator as a basis for distinguishing true from false categories of the composite image output by the generator, training the image segmentation model of the generator to improve the accuracy of extracting bubble features from the original comic image.

[0111] As mentioned above, after the discriminator completes the training, it has the ability to identify whether the bubble features contained in a picture are true value types or false value types. Therefore, the discriminator can be used as a basis for judging the training of the generator in reverse, and reversely guide the generator, especially the image segmentation model it adopts to continuously optimize its own weight parameters to produce a more accurate bubble feature map with a higher degree of matching.

[0112] Therefore, it can be understood that when entering the stage of training the generator, the discriminator is no longer trained, so its weight parameters are frozen and no longer corrected. It only provides its judgment results to the generator as a basis for judgment, and backpropagates to correct the relevant weight parameters of the generator, including correcting the weight parameters used in the fusion network therein, and thereby correcting the weight parameters of the image segmentation model.

[0113] For the generator training phase, the following loss function is used:

[0114] Loss gan =-∑TlogY+norm(W1,W2) (3)

[0115] Where norm(W1,W2)=log(W1)+log(W2) (4)

[0116] Here, W1 and W2 are from formula (1)

[0117] In addition to shortening the difference between false value data and true value data, the generator also needs to ensure that W1 and W2 cannot be zero, so the logarithm is taken in formula (4).

[0118] Note: In another embodiment, it would be more natural to use KL divergence for norm(W1,W2) from a convergence perspective, but this would introduce another parameter 'ρ' to control the distribution of W1 and W2.

[0119] After the above steps, a typical embodiment of the present application first trains the discriminator, and then uses the discriminator to judge the authenticity of the synthetic image output by the generator, provide a criterion for it, and continuously correct the generator so that it produces synthetic images that are closer and closer to the ideal effect.

[0120] Correspondingly, since the generator is implemented using an image segmentation model with a U-net architecture, the process of training the generator is essentially also the process of training the image segmentation model. This process enables the image segmentation model to generate more and more accurate bubble feature maps. As the number of iterative training increases, the bubble feature maps segmented from comic pictures by the image segmentation model will express more and more accurate bubble contours.

[0121] What is more worth mentioning is that since the image segmentation model is trained with the support of the generative adversarial model, the bubble features it segments are Figure 1 On the one hand, it can be efficiently verified by the discriminator, and on the other hand, it generates representative samples for the generative adversarial model. The game mechanism of the generative adversarial model in turn helps the image segmentation model to continuously improve its accuracy in extracting bubble feature maps from the original comics. The two complement each other.

[0122] It can be understood that according to the training process implemented in accordance with the technical solution of the present application, only a small amount of bubble feature maps representing real data need to be used as training sample materials to train an image segmentation model based on U-net.

[0123] In the actual test of the applicant, a U-net image segmentation model can be trained as long as 50 samples with sufficiently sparse distribution are given. If appropriate manual intervention is made in the process of training the discriminator, such as marking a part of the bubble feature maps with relatively good matching effects produced by the image segmentation model as true value type training samples during the iteration process, the convergence speed and robustness of the entire architecture-related model, especially the image segmentation model, will be greatly improved.

[0124] Therefore, it can be seen that the architecture proposed in this application is getting closer and closer to human intelligence. Only a small number of training samples are needed to enable the neural network architecture to summarize relevant rules, greatly improving its learning efficiency and application capabilities.

[0125] Based on the typical first draft example of the comic bubble recognition training method of the present application, a variety of flexible implementations or continuously deepened embodiments can be transformed, and other embodiments are further disclosed as follows:

[0126] In one embodiment, based on the typical embodiment of the present application, step S11 is deepened. In this step, an image segmentation model based on the U-net architecture proposed by David Dubray and Jochen Laubrock in 2019 is adopted. U-net is a neural network based on the Residual U-block structure. It realizes feature extraction and image restoration through an encoder and decoder with a U-shaped structure, and has the characteristic of being able to capture multi-scale features. In principle, the image segmentation model of U-net has an encoding path composed of multiple encoders and a decoding path composed of multiple decoders. The encoding path mainly implements feature compression by encoders at all levels to gradually improve the semantics of feature extraction, and the decoding path mainly implements decoding by decoders at all levels to gradually restore pictures with higher and higher resolutions. Context information is selected in the process of decompression of the decoding path step by step. Therefore, in order to compensate for feature loss, in the U-shaped symmetrical structure, each decoder level uses the decompressed feature map of its previous level and the feature map obtained by the encoder at the same level as the input, that is, the output of the encoder at the same level as the decoder is directly connected to the decoder as input to compensate for context information so as to ensure the quality of the restored image.

[0127] Among them, Figure 4 As shown, the process of extracting a bubble feature map containing bubble features from the comic original image by the image segmentation model includes:

[0128] Step S111: feeding the comic original image into the input convolution layer of the image segmentation model constructed based on the U-Net neural network to extract local features and obtain an intermediate feature map.

[0129] This step uses a fully convolutional approach to predict pixel-based image segmentation.

[0130] Step S112: The intermediate feature map is gradually compressed by multiple encoders in the encoding path of the image segmentation model to extract multi-scale features.

[0131] The encoding path uses the convolution part of the VGG-16 model. Each cartoon image fed in is gradually extracted by each level of encoder to obtain the encoding representation of each level, namely the intermediate feature map. Each intermediate feature map halves the spatial height and width of the intermediate feature map of the previous level. Finally, the multi-scale feature map of the cartoon image is obtained through the encoding path.

[0132] In this step, this can be achieved by using the outputs of the five convolutional blocks of the VGG-16 model.

[0133] Step S113: using the multi-scale features as input features, decoding them step by step through multiple decoders in the decoding path of the image segmentation model, and restoring them step by step to feature maps with higher resolutions, wherein each decoder uses the feature map obtained by its corresponding encoder as a reference to implement the restoration.

[0134] In this step, the decoding path is responsible for decoding to the resolution of the intermediate feature map obtained at the original level. Each of the five decoding steps doubles the width and height of the encoded representation.

[0135] Decoding is achieved through transposed convolution. In each decoding step, a direct connection is used, which means that a copy of the encoded representation corresponding to the width and height of the output result of the decoder at that level is provided directly from the encoding path as a context reference, thereby achieving step-by-step decoding restoration. It can be understood that the image output by the last decoder contains the bubble feature information corresponding to the original comic image, that is, the bubble feature map.

[0136] Step S114: All outputs of the decoders are merged to obtain a bubble feature map containing bubble features of the original comic image.

[0137] Through this embodiment, an implementation architecture of the image segmentation model and its operation process are further revealed. It can be known that the image segmentation model can be used to generate a corresponding bubble feature map for the comic original image fed into it.

[0138] In another embodiment of the present application, in response to the need of combining the image segmentation model with the traditional generative adversarial model, such as Figure 5 As shown, in step S12 of the typical embodiment of the present application, a fusion network is added to the generator of the generative adversarial model, wherein the fusion network fuses the comic original image with the bubble feature map, including:

[0139] Step S121: receiving the bubble feature map and its corresponding comic original image, wherein the bubble feature map is derived from the output of the image segmentation model or other external storage address.

[0140] In this step, the fusion network mainly synthesizes a comic original image with its corresponding bubble feature map. For the bubble feature map output by the image segmentation model, by default, it will be marked as a false value type after being synthesized with the comic original image. Similarly, it is also applicable to synthesizing the bubble feature map that is pre-labeled manually with the corresponding comic original image, and the corresponding synthesized map will be marked as a true value type. For the bubble feature map output by the image segmentation model, it can be synchronously output to certain storage addresses so that manual intervention can be implemented when necessary, such as the manual marking of some bubble feature maps with higher matching degrees in the previous article.

[0141] Step S122: using a single neuron formed by a neural network to perform linear operations on the weight parameters associated with the original cartoon image and the bubble feature image, respectively, to achieve fusion of the two into the same composite image.

[0142] The fusion network is similarly constructed using a neural network. Since the operations involved are mainly linear operations, constructing a neuron using a neural network can meet the application needs of the fusion network. The mathematical expression used is shown in the above formula (1). By associating different weight parameters with the original cartoon image and the corresponding bubble feature image for linear operations, the two images can be combined into one to obtain the corresponding composite image.

[0143] Step S123: output the synthesized image to the discriminator.

[0144] The bubble feature map output by the image segmentation model is regarded as pseudo data by default. Therefore, its corresponding synthetic image will be marked as false value type 0 and passed to the discriminator as a training sample. For the synthetic image synthesized from the artificially pre-labeled bubble feature map, it will be marked as true value type 1 and passed to the discriminator as a training sample.

[0145] It can be seen from this embodiment that the present application cleverly combines GAN and U-net together through a simple fusion network, thus laying the foundation for efficient training of the image segmentation model.

[0146] In another embodiment of the present application, the operation of the discriminator in the generative adversarial model during its training process is mainly deepened, and by optimizing this process, the effective collaboration between the generative adversarial model and the U-net is ensured. In this embodiment, in particular, in step S13, as Figure 6 As shown, the training process embodied by the discriminator includes:

[0147] Step S131: extract features from the input composite image of each category.

[0148] In conjunction with the foregoing, the discriminator is a simple classification model, which first extracts features from the synthetic image through a convolutional layer to obtain a relevant feature map.

[0149] Step S132: classify and distinguish the composite image based on the extracted features to obtain corresponding true value and false value distinction results, thereby enhancing its distinction ability.

[0150] The feature map obtained in the previous step is then processed by a fully connected layer and mapped to the [0, 1] space, thereby obtaining the corresponding discrimination result.

[0151] Step S133: Modify the weight parameters used by itself according to the discrimination result.

[0152] According to the discrimination result, the discriminator modifies its own weight according to the loss function. Since the discriminator is in the training period, its own weight parameters need to be continuously modified so that the discrimination result gradually converges and it continuously learns and enhances its own discrimination ability.

[0153] Step S134: back-propagate to the generative adversarial model and modify the weight parameters of the fusion network and image segmentation model in the generator accordingly.

[0154] During the training process of the discriminator, its discrimination results can also be used as a criterion to correct the generator in the generative adversarial model. Therefore, backpropagation can be performed in this step. First, the two weight parameters in the linear algorithm in the fusion network are corrected. Then, inside the generator, the relevant weight parameters of the image segmentation model are also backpropagated and corrected in the same way, thereby continuously improving the ability of the image segmentation model to extract bubble features.

[0155] In another embodiment of the present application, the main purpose is to further reveal the performance of the discriminator in the process of participating in the training of the generator after the training is completed. In step S14, Figure 7 As shown, the discriminator is no longer trained, but in turn provides a basis for judging the training of the generator. Therefore, in the step of using the trained discriminator as a basis for distinguishing the true and false categories of the synthetic image output by the generator, the discrimination process of the discriminator includes:

[0156] Step S141: Extract features of each type of synthetic image fed in:

[0157] Step S142: classify and distinguish the composite image based on the extracted features to obtain corresponding true value and false value distinction results.

[0158] Step S143: Freeze the modification of the weight parameters used by itself.

[0159] Step S144: back-propagate to the generative adversarial model according to the discrimination result, and modify the weight parameters of the fusion network and the image segmentation model in the generator accordingly.

[0160] According to the above steps, it can be understood that the different performances of the discriminator in providing judgment criteria and in the process of being trained are mainly reflected in the difference between step S133 of the previous embodiment and step S143 of this embodiment. In step S133, since the discriminator is being trained, its own weight parameters need to be continuously revised, while in step S143, since the discriminator has been trained as a mature discriminator in theory, it has the authority to provide judgment basis for the generator, and the stability of this authority needs to be ensured. Therefore, in step S143, the discriminator's own weight parameters will be frozen and no longer participate in the correction of the back propagation process. Therefore, the discriminator skips its own weight parameters through back propagation according to its judgment results, and corrects the parameters of the generator's fusion network in step S144, which ultimately leads to the continuous correction of the weight parameters of the image segmentation model. After so many iterations, the results of the image segmentation model gradually converge.

[0161] It should be noted that the above disclosed embodiments of the present application can be arbitrarily combined or implemented separately, or replaced with other similar technologies or similar technologies based on the creative spirit of the present application. For example, the continuously upgraded network structure of U-net can also be used to replace the image segmentation model described in the present application in theory. Those skilled in the art should be aware of this.

[0162] The present application also provides a comic bubble response method, which mainly utilizes the network architecture proposed in the present application and makes further applications based on the mature training of the comic bubble training method of the present application.

[0163] Specifically, see Figure 8 , which comprises the following steps:

[0164] Step S21: input the comic picture to be identified into a pre-trained image segmentation model for image segmentation to obtain a bubble feature map contained in the comic picture.

[0165] The image segmentation model is an image segmentation model trained by any one of the embodiments of the comic bubble training method of the present application. Since the image segmentation model has the ability to extract the bubble feature map for the comic original image with a high accuracy, the image segmentation model can be used in the present method to serve the utilization of the comic image. In this step, it is only necessary to call the interface provided by the image segmentation model and feed the comic image for which the bubble contour features need to be extracted into the image segmentation model for feature extraction, and then the bubble feature map corresponding to the comic image can be obtained.

[0166] Step S22: acquiring a text image in the comic picture according to the correspondence between the bubble feature map and the comic picture.

[0167] Since the bubble feature map obtained by the image segmentation model has reliably included the bubble contour features of the comic picture, the bubble feature map can be used as a template to extract the text image of the area covered by the bubble contour in the template from the comic picture. It can be understood that the text image only contains text information.

[0168] Step S23: optically recognize the text image to convert it into text information recognizable by a computer.

[0169] After obtaining the text image, the well-known and mature OCR optical recognition technology can be used to optically recognize the text image of each bubble, convert it into computer-recognizable text information, and complete the extraction of text information in the dialogue bubble of the comic picture.

[0170] It can be foreseen that the image segmentation model trained using the technical solution of the present application is more accurate and reliable when used to extract text information from comic images.

[0171] In a further embodiment of the comic bubble response method of the present application, after step S23, as shown in FIG. Fig. 9 As shown, the following subsequent steps can be added:

[0172] Step S24: calling a human voice model to synthesize a corresponding human voice file for the text information, and storing the human voice file in association with the comic picture for synchronous playback with the comic picture.

[0173] After the text information of the comic picture is extracted, it can be used to make audio books for electronic education, and can be called in some live broadcast scenarios. Therefore, for the computer-recognizable text information obtained in the previous embodiment, this embodiment further uses the well-known and mature human voice synthesis technology to call the human voice model to synthesize the corresponding human voice file for the text information, and then associate it with the comic picture and store it in the database, and then accept the call in a live broadcast scenario, and when called, it is played synchronously with the comic picture through the player. Of course, the application of human voice files is not limited to the specific application scenario of live broadcast, but can also be used in scenarios such as point-reading teaching materials, and is stored in association with comic pictures in a point-reading device, and the corresponding human voice file is played by reading the bubbles of the comic picture through the point-reading pen.

[0174] It can be seen that this embodiment further broadens the application scenarios of the technical solution of the present application, expands the ability of the present application to solve similar technical problems in different technical fields, and consolidates the economic contribution capacity of the technical solution of the present application.

[0175] Furthermore, by functionalizing the various embodiments of the above-mentioned comic bubble recognition training method, a comic bubble recognition training device of the present application can be constructed. According to this idea, please refer to Fig.10 In a typical embodiment, the device comprises:

[0176] The bubble segmentation module 11 is used to extract a bubble feature map containing bubble features from the comic original image using the image segmentation model used by the generator of the generative adversarial model;

[0177] An image fusion module 12, configured to fuse the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map respectively by a fusion network in the generator, and generate synthetic images of false value type and true value type respectively;

[0178] A discrimination training module 13 is used to feed the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that it has the ability to identify the true and false categories to which the bubble features contained in the image belong;

[0179] A training module 14 is generated, which is used to use the trained discriminator as a basis for distinguishing true and false categories of the composite image output by the generator, and train the image segmentation model of the generator to improve the accuracy of extracting bubble features from the original comic image.

[0180] Similarly, by functionalizing the various embodiments of the above-mentioned comic bubble recognition and response method, a comic bubble recognition and response device of the present application can be constructed. According to this idea, please refer to Fig.11 In a typical embodiment, the device comprises:

[0181] The bubble segmentation module 21 is used to input the comic picture to be identified into the pre-trained image segmentation model to perform image segmentation and obtain a bubble feature map contained in the comic picture;

[0182] An image and text extraction module 22, used for obtaining a text image in the comic picture according to the correspondence between the bubble feature map and the comic picture;

[0183] The image-text conversion module 23 is used to perform optical recognition on the text image to convert it into computer-recognizable text information;

[0184] The image segmentation model is pre-trained by iteratively performing the following specific steps:

[0185] The image segmentation model used by the generator of the generative adversarial model is used to extract a bubble feature map containing its bubble features from the comic original image;

[0186] The fusion network in the generator fuses the comic original image with the extracted bubble feature map and the pre-labeled bubble feature map, respectively, to generate synthetic images of false value type and true value type respectively;

[0187] Feeding the synthetic image as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong;

[0188] The trained discriminator is used as the basis for distinguishing the true and false categories of the synthetic images output by the generator, and the image segmentation model of the generator is trained to improve the accuracy of extracting bubble features from the original comic image.

[0189] To facilitate the execution of the present application, an electronic device is provided, including a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of a comic bubble recognition training / response method as described above.

[0190] It can be seen that the memory is suitable for using a non-volatile storage medium. By implementing the aforementioned method as a computer program and installing it in an electronic device such as a mobile phone, the relevant program code and data are stored in the non-volatile storage medium of the electronic device. The program is further run by the central processor of the electronic device, and it is transferred from the non-volatile storage medium to the memory for execution, so that the desired purpose of the present application can be achieved. Therefore, it can be understood that in one embodiment of the present application, a non-volatile storage medium can also be provided, which stores a computer program implemented according to various embodiments of the comic bubble recognition training / response method in the form of computer-readable instructions. When the computer program is called and executed by the computer, the steps included in the method are executed.

[0191] In summary, the present application can train an image segmentation model suitable for segmenting a bubble highlight image containing bubble features from a comic image with a small number of samples. Through this image segmentation model, the bubble feature image can be quickly and accurately segmented and extracted from the comic image.

[0192] It will be appreciated by those skilled in the art that the present application package relates to devices for performing one or more of the operations and methods described in the present application. These devices may be specially designed and manufactured for the desired purpose, or may also include known devices in general-purpose computers. These devices have computer programs stored in their memory, which are selectively activated or reconstructed. Such computer programs may be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, respectively, the computer readable medium including but not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable medium includes any medium that stores or transmits information in a readable form by a device (e.g., computer).

[0193] Those skilled in the art will appreciate that each block in these structure diagrams and / or block diagrams and / or flow charts and combinations of blocks in these structure diagrams and / or block diagrams and / or flow charts can be implemented using computer program instructions. Those skilled in the art will appreciate that these computer program instructions can be provided to a general-purpose computer, a professional computer, or a processor of other programmable data processing methods to implement, thereby executing the scheme specified in the block or multiple blocks of the structure diagrams and / or block diagrams and / or flow charts disclosed in this application through the processor of the computer or other programmable data processing method.

[0194] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0195] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A comic bubble recognition training method, characterized in that: The steps include: Extracting a bubble feature map containing bubble features from the comic original image using an image segmentation model used by the generator of the generative adversarial model, wherein the bubble feature is an irregular overall outline of the speech bubble; The fusion network in the generator fuses the comic original image with the extracted bubble feature map and the pre-marked bubble feature map, respectively, to generate synthetic images of false value type and true value type respectively, including: receiving the bubble feature map and its corresponding comic original image, the bubble feature map is derived from the output of the image segmentation model or other external storage address; using a single neuron composed of a neural network to perform linear operations on the weight parameters associated with the comic original image and the bubble feature map, respectively, to achieve fusion of the two into the same synthetic image; outputting the synthetic image to the discriminator; Feeding the synthetic image output by the fusion network as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong; The trained discriminator is used as the basis for distinguishing the true and false categories of the synthetic images output by the generator, and the image segmentation model of the generator is trained to improve the accuracy of extracting bubble features from the original comic image.

2. The method according to claim 1, characterized in that: The process of extracting a bubble feature map containing bubble features from the comic original image by the image segmentation model includes: Feeding the comic original image into the input convolution layer of the image segmentation model constructed based on the U-Net neural network to extract local features and obtain an intermediate feature map; The intermediate feature map is gradually compressed by multiple encoders in the encoding path of the image segmentation model to extract multi-scale features; The multi-scale features are used as input features, and are gradually decoded by multiple decoders in the decoding path of the image segmentation model to be restored to feature maps with higher resolutions step by step, wherein each decoder uses the feature map obtained by a corresponding encoder as a reference to implement the restoration; All the outputs of the decoders are fused to obtain a bubble feature map containing the bubble features of the original comic image.

3. The method according to claim 1, characterized in that: The training process of the discriminator includes: Perform feature extraction on the synthetic image of each category fed in; The synthetic image is classified and judged according to the extracted features to obtain the corresponding true value and false value discrimination results, thereby enhancing its discrimination ability; Modify the weight parameters used by itself according to the discrimination results; Back propagates to the generative adversarial model and corrects the weight parameters of the fusion network and image segmentation model in the generator accordingly.

4. The method according to any one of claims 1 to 3, characterized in that In the step of using the trained discriminator as a basis for distinguishing true from false categories of the synthetic image output by the generator, the discrimination process of the discriminator includes: Perform feature extraction on the synthetic image of each category fed in; Classify and discriminate the synthetic image according to the extracted features to obtain the corresponding true value and false value discrimination results; Freeze the modifications to the weight parameters used by itself; According to the discrimination result, back propagation is performed to the generative adversarial model, and the weight parameters of the fusion network and the image segmentation model in the generator are corrected accordingly.

5. A comic bubble recognition training device, characterized in that: It includes: A bubble segmentation module, used to extract a bubble feature map containing bubble features from the comic original image using the image segmentation model used by the generator of the generative adversarial model, wherein the bubble feature is an irregular overall outline of the dialogue bubble; The image fusion module is used to fuse the comic original image with the extracted bubble feature map and the pre-marked bubble feature map respectively by the fusion network in the generator, and generate synthetic images of false value type and true value type respectively, including: receiving the bubble feature map and its corresponding comic original image, the bubble feature map is derived from the output of the image segmentation model or other external storage address; using a single neuron composed of a neural network to perform linear operations on the associated weight parameters of the comic original image and the bubble feature map respectively, so as to realize the fusion of the two into the same synthetic image; outputting the synthetic image to the discriminator; A discrimination training module, used for feeding the synthetic image output by the fusion network as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong; A training module is generated, which is used to use the trained discriminator as a basis for distinguishing true and false categories of the synthetic image output by the generator, and train the image segmentation model of the generator to improve the accuracy of extracting bubble features from the original comic image.

6. A comic bubble recognition and response method, characterized in that: The steps include: Inputting the comic picture to be recognized into the pre-trained image segmentation model for image segmentation, and obtaining a bubble feature map contained in the comic picture; Acquire a text image in the comic picture according to a correspondence between the bubble feature map and the comic picture; Optically recognizing the text image to convert it into computer-recognizable text information; The image segmentation model is pre-trained by iteratively performing the following specific steps: Extracting a bubble feature map containing bubble features from the comic original image using an image segmentation model used by the generator of the generative adversarial model, wherein the bubble feature is an irregular overall outline of the speech bubble; The fusion network in the generator fuses the comic original image with the extracted bubble feature map and the pre-marked bubble feature map, respectively, to generate synthetic images of false value type and true value type respectively, including: receiving the bubble feature map and its corresponding comic original image, the bubble feature map is derived from the output of the image segmentation model or other external storage address; using a single neuron composed of a neural network to perform linear operations on the weight parameters associated with the comic original image and the bubble feature map, respectively, to achieve fusion of the two into the same synthetic image; outputting the synthetic image to the discriminator; Feeding the synthetic image output by the fusion network as a training sample into the discriminator of the generative adversarial model for training, so that the discriminator has the ability to identify the true and false categories to which the bubble features contained in the image belong; The trained discriminator is used as the basis for distinguishing the true and false categories of the synthetic images output by the generator, and the image segmentation model of the generator is trained to improve the accuracy of extracting bubble features from the original comic image.

7. The method according to claim 6, characterized in that The following follow-up steps are included: A human voice model is called to synthesize a corresponding human voice file for the text information, and the human voice file is associated with the comic picture and stored for synchronous playback with the comic picture.

8. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 4, 6, and 7.

9. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 4, 6, and 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

Images