Image processing method, device, equipment and storage medium

Abstract

The present disclosure provides an image processing method, apparatus, device and storage medium, which relate to the field of artificial intelligence, and in particular to the fields of computer vision, deep learning and map data production technology, and can be specifically used in intelligent prevention and control scenarios. The specific implementation scheme is: performing text detection on the target text image to obtain the target text area in the target text image, and the first text category of the target text area; classifying the text content in the target text area to obtain the second text category of the target text area; fusing the first text category and the second text category of the target text area to obtain the target category of the target text area. Through the above technical scheme, the text category from the visual perspective and the text category from the semantic perspective are fused to determine the target text category, so that the target text category finally obtained has a higher accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision, deep learning and map data production technology, and specifically to an image processing method, device, equipment and storage medium. Background Art

[0002] With the widespread popularity of artificial intelligence technology, neural network models have been applied to various fields. For example, object detection models are used to detect objects in images. At present, for images containing text, the existing object detection models cannot accurately determine the category of the text in the image, and they are in urgent need of improvement. Summary of the invention

[0003] The present disclosure provides an image processing method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a speech processing method, the method comprising:

[0005] Performing text detection on the target text image to obtain a target text area in the target text image and a first text category of the target text area;

[0006] Classifying the text content in the target text area to obtain a second text category of the target text area;

[0007] The first text category and the second text category of the target text area are fused to obtain a target category of the target text area.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, the electronic device comprising:

[0009] at least one processor; and

[0010] a memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method described in any embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the image processing method described in any embodiment of the present disclosure.

[0013] According to the technology disclosed in the present invention, the recognition accuracy of text categories in text images is improved.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0016] Figure 1A is a flowchart of an image processing method provided according to an embodiment of the present disclosure;

[0017] Figure 1B It is a schematic diagram of a synchronous training process of three models provided according to an embodiment of the present disclosure;

[0018] Figure 2 is a flowchart of another image processing method provided according to an embodiment of the present disclosure;

[0019] Figure 3 is a flowchart of another image processing method provided according to an embodiment of the present disclosure;

[0020] Figure 4 is a flowchart of another image processing method provided according to an embodiment of the present disclosure;

[0021] Figure 5 is a structural schematic diagram of an image processing device provided according to an embodiment of the present disclosure;

[0022] Figure 6 is a block diagram of an electronic device for implementing the image processing method of an embodiment of the present disclosure; DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Figure 1A 1 is a flowchart of an image processing method provided according to an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case of how to process an image, and in particular to the case of how to process an image including text. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and can be integrated in an electronic device carrying an image processing function, such as a server. Figure 1A As shown, the image processing method provided in this embodiment may include:

[0025] S101, performing text detection on a target text image to obtain a target text region in the target text image and a first text category of the target text region.

[0026] The so-called target text image is an image including text to be processed, which may be a target stamp image or a target signboard image. The target text may be all text in the target text image, or may be text related to a specified text category in the target text image, wherein the specified text category is a pre-set text category of great significance for data production, etc. For example, when the target image is a target signboard image, the specified text category includes but is not limited to name, telephone number, address, business scope, advertisement and other categories.

[0027] The target text area refers to the area where the target text in the target text image is located. The target text image may contain one or more target text areas, each of which corresponds to a first text category. The first text category refers to the category of the text content in the target text area determined from a visual perspective, such as name, telephone number, address, business scope, and advertisement.

[0028] In one possible implementation, the target text image can be detected based on a machine learning model to obtain a target text region in the target text image and a first text category of the target text region. The machine learning model can be an optical character recognition (OCR) model.

[0029] Furthermore, in order to extract a more effective text region from the target text image, as an implementation method of the present disclosure, the machine learning model is a visual segmentation model, and the target text image can be detected based on the visual segmentation model to obtain the target text region in the target text image and the first text category of the target text region. Furthermore, the position of the target text region can also be obtained, where the position can be the position coordinates of the four corners of the text box corresponding to the target text region.

[0030] Optionally, the visual segmentation model of this embodiment may include a feature extraction network and a candidate region generation network, wherein the feature extraction network is used to extract features of the target text image to obtain image features; and the candidate region generation network is used to process the image features to obtain text regions.

[0031] S102: Classify the text content in the target text area to obtain a second text category of the target text area.

[0032] Among them, the second text category refers to the category of text content in the target text area determined from a semantic perspective. For example, when the target image is a target signboard image, the text category in the second target signboard image includes but is not limited to name, telephone number, address, business scope, advertisement and other categories.

[0033] One possible implementation method can identify the text content in the target text area based on a machine learning model, and then classify the text content to obtain a second text category of the target text area.

[0034] Furthermore, in order to more accurately determine the second text category of the target text area, as an implementation method of the present disclosure, the machine learning model is a text parsing model, and then based on the text parsing model, the text content in the target text area can be classified to obtain the second text category of the target text area.

[0035] Optionally, the text parsing model of this embodiment can be a recursive convolutional neural network (CRNN). Furthermore, the text parsing model can also be composed of a text recognition model and a text classification model, wherein the text recognition model is used to recognize the text content of the target text area; the text classification model is used to classify the text content obtained by the text recognition model.

[0036] S103: performing fusion processing on the first text category and the second text category of the target text area to obtain a target category of the target text area.

[0037] In this embodiment, the first text category and the second text category of the target text area may be analyzed based on statistical analysis to obtain the target category of the target text area.

[0038] For example, the first text category and the second text category of the target text region may be input into a statistical analysis model, and the statistical analysis model may be used to process the target text region to obtain the target category of the target text region.

[0039] After determining the target text area and the target category of the target text area, as an implementable embodiment of the present disclosure, the target text image can also be associated and stored with the position information, text content and target category of the target text area in the target text image. Specifically, the target text image can be associated and stored with the position information, text content and target category of the target text area in the target text image according to the set association relationship.

[0040] It is understandable that by associating and storing the target text image with the location information, text content and target category of the target text area in the target text image, subsequent data production and use can be facilitated, such as producing map Points of Interest (POI) data.

[0041] The technical solution provided by the embodiment of the present disclosure performs text detection on the target text image to obtain the target text area in the target text image and the first text category of the target text area, then classifies the text content in the target text area to obtain the second text category of the target text area, and then fuses the first text category and the second text category of the target text area to obtain the target category of the target text area. The above technical solution fuses the text category from the visual perspective and the text category from the semantic perspective to determine the target text category, so that the target text category finally obtained has a higher accuracy.

[0042] On the basis of the above embodiments, optionally, as a preferred implementation mode of the embodiments of the present disclosure, text detection may be performed on the target text image based on a visual segmentation model to obtain a target text area in the target text image and a first text category of the target text area, and the text content in the target text area is classified based on a text recognition model and a text classification model to obtain a second text category of the target text area, and then the first text category and the second text category of the target text area are fused to obtain a target category of the target text area.

[0043] Exemplarily, in this embodiment, the visual segmentation model, the text recognition model and the text classification model can be obtained by training the convolutional neural network one by one based on the sample images respectively.

[0044] Exemplarily, in this embodiment, the visual segmentation model, the text recognition model and the text classification model can also be obtained by training the convolutional neural network together based on sample images in a cascade manner.

[0045] The sample images in this embodiment are obtained by performing data enhancement on the original images; the data enhancement methods may include but are not limited to data mixing algorithms, random occlusion algorithms, random cropping and / or expansion, and brightness adjustment, etc.; the original images include images of different resolutions. It can be understood that the sample images are obtained by performing data enhancement on the original images, which makes up for the defect of insufficient data and ensures the adequacy of the samples. Furthermore, the sample images are annotated. Specifically, for each sample image, a text area (such as a signboard text) is selected, and the text category and text content in the text area are annotated.

[0046] Specifically, the visual segmentation model, text recognition model and text classification model in this embodiment can be implemented as follows: Figure 1BThe implementation method shown can be: inputting the sample text image into the feature extraction network of the visual segmentation model to obtain the sample image features; inputting the sample image features into the candidate region generation network to obtain the candidate text region; using a multi-category non-maximum algorithm to screen the candidate text region to obtain the screened text region; inputting the screened text region into the result output network of the visual segmentation model (including three branches, namely, the position branch, the region branch and the category branch) to obtain the text box position of the sample text image (i.e., the sample text region position), the segmentation result (i.e., the segmented text box, i.e., the sample text region), and the first text category of the sample text region; and then inputting the sample text region into the text recognition model to obtain the sample text region. The invention relates to a method for training a text recognition model and a text classification model for a text box. The method comprises the following steps: first, a text box, a text box domain; inputting the text content into a text classification model to obtain a second text category of the sample text area; determining a category loss according to the first text category predicted by the visual segmentation model, the second text category predicted by the text classification model, and the annotated category data, that is, fusing the category losses of the visual segmentation model and the text classification model, and determining a segmentation loss according to the annotated text box data and the segmentation result of the visual segmentation model (i.e., the segmented text box); and determining a position loss according to the annotated text box position and the text box position predicted by the visual segmentation model (i.e., the sample text area position); and then training the visual segmentation model, the text recognition model, and the text classification model together according to the classification loss, segmentation loss, and position loss.

[0047] It is understandable that by training the model in a cascade manner, continuously correcting the position of the text area, and optimizing the text category of the text area, the purpose of accurately determining the text category can be achieved. At the same time, compared with training each model one by one, the complexity of model training is reduced.

[0048] Figure 2 is a flowchart of another image processing method provided according to an embodiment of the present disclosure. Based on the above embodiment, this embodiment further explains in detail how to "perform text detection on a target text image, obtain a target text area in the target text image, and a first text category of the target text area". Figure 2 As shown, the image processing method provided in this embodiment may include:

[0049] S201, extracting image features of the target text image based on the feature extraction network in the visual segmentation model.

[0050] Optionally, the feature extraction network in this embodiment may include a convolutional neural network, such as a residual network (ResNet50 / 100).

[0051] Furthermore, the feature extraction network may also include multiple groups of feature extraction layers, each group of feature extraction layers including a deformable convolution layer and a group normalization layer.

[0052] Specifically, the target text image is input into the feature extraction network in the visual segmentation model, and the image features of the target text image are obtained through multiple groups of feature extraction layers in the feature extraction network.

[0053] It can be understood that by using deformable convolution and group normalization to construct a feature extraction network to extract image features of the target image, the feature extraction capability of the model can be improved, laying the foundation for subsequent accurate detection of the area where the text is located.

[0054] S202, based on the candidate region generation network in the visual segmentation model, determine the candidate text region in the target text image according to the image features.

[0055] In this embodiment, the candidate region generation network may include a region proposal network (RPN) and a RoI Align layer.

[0056] Exemplarily, the image features are input into the RPN in the candidate region generation network to obtain at least one region to be detected, and the at least one region to be detected is input into the RoI Align layer for processing to obtain a candidate text region.

[0057] Specifically, for each category of target in the target text image, at least one candidate text region corresponding to the category target will be obtained. That is to say, for the image features corresponding to each category target, the part of the target text image except for the category, i.e., the background, is removed through RPN processing, and at least one region to be detected corresponding to the category target is obtained. Then, at least one region to be detected is input into the RoI Align layer for processing to obtain the candidate text region corresponding to the category target. For example, there are 5 categories of text content in the signboard image. If the target text image contains 3 categories of text content, for each category, at least one candidate text region corresponding to the category will be obtained.

[0058] S203, using a multi-category non-maximum suppression algorithm to screen the candidate text regions.

[0059] For at least one candidate text area corresponding to each category in the target text image, there may be some invalid candidate text areas. Therefore, as an optional method of this embodiment, a multi-category non-maximum suppression algorithm can be used to screen the candidate text areas, filter out invalid candidate text areas, and obtain the screened text areas.

[0060] S204, processing the filtered text area to obtain a target text area in the target text image and a first text category of the target text area.

[0061] In this embodiment, the filtered text area is input into the result output network of the visual segmentation model, wherein the result output network includes three branches, namely, a position branch, a region branch and a category branch, the position branch includes a head layer and multiple fully convolutional layers; the region branch includes multiple fully connected layers and bounding box regression operations; the category branch includes multiple fully connected layers and a Softmax classification operation; specifically, the filtered text area is input into the position branch of the result output network to obtain the target text area position in the target text image, the filtered text area is input into the region branch of the result output network to obtain the target text area in the target text image, and the filtered text area is input into the region branch of the result output network to obtain the first text category of the target text area.

[0062] S205: Classify the text content in the target text area to obtain a second text category of the target text area.

[0063] S206: Perform fusion processing on the first text category and the second text category of the target text area to obtain a target category of the target text area.

[0064] The technical solution provided by the disclosed embodiment extracts the image features of the target text image based on the feature extraction network in the visual segmentation model, determines the candidate text regions in the target text image based on the candidate region generation network in the visual segmentation model according to the image features, and then uses a multi-category non-maximum suppression algorithm to screen the candidate text regions, processes the screened text regions to obtain the target text region in the target text image, and the first text category of the target text region, and then classifies the text content in the target text region to obtain the second text category of the target text region, and finally fuses the first text category and the second text category of the target text region to obtain the target category of the target text region. The above technical solution can accurately distinguish the valid text region through the visual segmentation model, laying a foundation for the subsequent accurate determination of the category of the text in the text image.

[0065] Figure 3 This is a flowchart of another image processing method provided according to an embodiment of the present disclosure. Based on the above embodiment, this embodiment further explains in detail how to "classify the text content in the target text area to obtain the second text category of the target text area". Figure 3 As shown, the image processing method provided in this embodiment may include:

[0066] S301, performing text detection on a target text image to obtain a target text region in the target text image and a first text category of the target text region.

[0067] S302: Based on the text recognition model, the target text area is recognized to obtain the text content of the target text area.

[0068] Among them, the text recognition model can be a CRNN model, CTC (Connectionist Temporal Classification), and a semantic reasoning network (Semantic Reasoning Network, SRN). In this embodiment, SRN is preferred.

[0069] Specifically, the target text region is input into the text recognition model for recognition processing to obtain the text content of the target text region. For example, if there are three target text regions in the target text image, the text content of the three target text regions is obtained.

[0070] S303: Classify the text content based on the text classification model to obtain a second text category of the target text area.

[0071] Among them, the text classification model can be a support vector machine (SVM), a polynomial model, a multiple Bernoulli model, and a bidirectional encoder representation from transformers (BERT) model, etc. In this embodiment, the BERT model is preferred.

[0072] Specifically, the text content is input into the text classification model for classification processing to obtain the second text type of the target text area. For example, if there are three target text areas in the target text image, the second text categories of the three target text areas are obtained.

[0073] S304: Perform fusion processing on the first text category and the second text category of the target text area to obtain a target category of the target text area.

[0074] The technical solution provided by the embodiment of the present disclosure performs text detection on the target text image to obtain the target text area in the target text image and the first text category of the target text area, then identifies the target text area based on the text recognition model to obtain the text content of the target text area, and then classifies the text content based on the text classification model to obtain the second text category of the target text area, and finally fuses the first text category and the second text category of the target text area to obtain the target category of the target text area. The above technical solution identifies the text content through the text recognition model and the text classification model, and adopts the cooperation of the two functional models to identify the text content in the text area, so that the text category finally obtained has a higher accuracy.

[0075] Figure 4 This is a flowchart of another image processing method provided according to an embodiment of the present disclosure. Based on the above embodiment, this embodiment further explains in detail how to "merge the first text category and the second text category of the target text area to obtain the target category of the target text area". Figure 4 As shown, the image processing method provided in this embodiment may include:

[0076] S401, performing text detection on a target text image to obtain a target text region in the target text image and a first text category of the target text region.

[0077] Optionally, text detection may be performed on the target text image based on a visual segmentation model to obtain a target text region in the target text image and a first text category of the target text region.

[0078] S402: Classify the text content in the target text area to obtain a second text category of the target text area.

[0079] S403: Identify whether the first text category of the target text area is the same as the second text category.

[0080] In this embodiment, whether the first text category and the second text category of the target text area are the same can be identified based on a statistical analysis algorithm.

[0081] S404: Select a target category for the target text area from the first text category and the second text category according to the recognition result.

[0082] Optionally, if the recognition result shows that the first text category and the second text category are the same, either the first text category or the second text category is used as the target category of the target text area.

[0083] Optionally, if the recognition result is that the first text category is different from the second text category, the target category of the target text area is selected from the first text category and the second text category according to the probability value of the first text category and the probability value of the second text category. The visual segmentation model outputs the first text category and the probability value of the first text category; the text classification model outputs the second text category and the probability value of the second text category.

[0084] Specifically, if the recognition result is that the first text category is different from the second text category, the probability value of the first text category and the probability value of the second text category are compared, and the one with a larger probability value is used as the target category of the target text area.

[0085] It should be noted that if the probability value of the second text category is greater than the probability value of the first text category, the second text category can also be used to correct the first text category, and then the visual segmentation model can be trained with the corrected first text category to improve the recognition accuracy of the model's text category.

[0086] The technical solution provided by the embodiment of the present disclosure performs text detection on the target text image to obtain the target text area in the target text image and the first text category of the target text area, then classifies the text content in the target text area to obtain the second text category of the target text area, and then identifies whether the first text category of the target text area is the same as the second text category, and finally selects the target category of the target text area from the first text category and the second text category according to the recognition result. The above technical solution determines the category of the text area through two text categories, thereby improving the accuracy of determining the category of the text area.

[0087] Figure 5 1 is a schematic diagram of the structure of an image processing device provided according to an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case of how to process an image. The method can be executed by an image processing device. The device can be implemented in software and / or hardware and can be integrated into an electronic device that carries an image processing function, such as a server. Figure 5 As shown, the image processing device 500 includes:

[0088] A first category determination module 501 is used to perform text detection on a target text image to obtain a target text region in the target text image and a first text category of the target text region;

[0089] A second category determination module 502, used to classify the text content in the target text region to obtain a second text category of the target text region;

[0090] The target category determination module 503 is used to perform a fusion process on the first text category and the second text category of the target text region to obtain the target category of the target text region.

[0091] The technical solution provided by the embodiment of the present disclosure performs text detection on the target text image to obtain the target text area in the target text image and the first text category of the target text area, then classifies the text content in the target text area to obtain the second text category of the target text area, and then fuses the first text category and the second text category of the target text area to obtain the target category of the target text area. The above technical solution fuses the text category from the visual perspective and the text category from the semantic perspective to determine the target text category, so that the target text category finally obtained has a higher accuracy.

[0092] Further, the first category determination module 501 includes:

[0093] An image feature extraction unit, used for extracting image features of a target text image based on a feature extraction network in a visual segmentation model;

[0094] A candidate region determination unit, used to determine a candidate text region in a target text image based on a candidate region generation network in a visual segmentation model and according to image features;

[0095] A screening unit, used to screen candidate text regions using a multi-category non-maximum suppression algorithm;

[0096] The first category determination unit is used to process the filtered text area to obtain a target text area in the target text image and a first text category of the target text area.

[0097] Further, the second category determination module 502 includes:

[0098] A text content obtaining unit is used to identify the target text area based on the text recognition model to obtain the text content of the target text area;

[0099] The second category determination unit is used to classify the text content based on the text classification model to obtain a second text category of the target text area.

[0100] Furthermore, the target category determination module 503 includes:

[0101] a recognition unit, used to recognize whether the first text category of the target text area is the same as the second text category;

[0102] The target category selection unit is used to select a target category of the target text area from the first text category and the second text category according to the recognition result.

[0103] Furthermore, the target category selection unit is specifically used for:

[0104] If the recognition result is that the first text category is different from the second text category, a target category of the target text area is selected from the first text category and the second text category according to the probability value of the first text category and the probability value of the second text category.

[0105] Furthermore, the device also includes:

[0106] The storage module is used to associate and store the target text image with the location information of the target text area in the target text image, the text content and the target category.

[0107] Furthermore, the target text image is a target signboard image.

[0108] In the technical solution disclosed in the present invention, the acquisition, storage and application of text image data, etc. are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0110] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0111] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0112] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0113] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the intersection drawing method. For example, in some embodiments, the intersection drawing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the intersection drawing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the intersection drawing method in any other appropriate manner (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] To provide interaction with the sender, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the sender; and a keyboard and pointing device (e.g., a mouse or trackball) through which the sender can provide input to the computer. Other types of devices can also be used to provide interaction with the sender; for example, the feedback provided to the sender can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the sender can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a sender computer with a graphical sender interface or a web browser through which a sender can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0119] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0120] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0121] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Based on the feature extraction network in the visual segmentation model, the image features of the target text image are extracted; Based on the candidate region generation network in the visual segmentation model, determining the candidate text region in the target text image according to the image features; Using a multi-category non-maximum suppression algorithm to screen the candidate text area; Processing the filtered text area to obtain a target text area in the target text image and a first text category of the target text area; Classifying the text content in the target text area to obtain a second text category of the target text area; Performing a fusion process on the first text category and the second text category of the target text area to obtain a target category of the target text area; Among them, the specific implementation of the visual segmentation model, the text recognition model and the text classification model is as follows: the sample text image is input into the feature extraction network of the visual segmentation model to obtain the sample image features; the sample image features are input into the candidate region generation network to obtain the candidate text region; the candidate text region is screened using a multi-category non-maximum algorithm to obtain the screened text region; the screened text region is input into the result output network of the visual segmentation model, and the result output network includes three branches, namely, the position branch, the region branch and the category branch, to obtain the text box position of the sample text image, i.e., the sample text region position, the segmentation result, i.e., the segmented text box, i.e., the sample text region, and the first text category of the sample text region; and then the sample text region is input into the text Recognition model, obtain the text content of the sample text area; input the text content into the text classification model, obtain the second text category of the sample text area; determine the category loss according to the first text category predicted by the visual segmentation model, the second text category predicted by the text classification model, and the annotated category data, that is, fuse the category losses of the visual segmentation model and the text classification model, and determine the segmentation loss according to the annotated text box data and the segmentation result of the visual segmentation model, that is, the segmented text box; and determine the position loss according to the annotated text box position and the text box position predicted by the visual segmentation model, that is, the position of the sample text area; and then train the visual segmentation model, the text recognition model and the text classification model together according to the classification loss, segmentation loss and position loss.

2. The method according to claim 1, wherein: The step of classifying the text content in the target text area to obtain a second text category of the target text area includes: Based on the text recognition model, the target text area is recognized to obtain the text content of the target text area; Based on a text classification model, the text content is classified to obtain a second text category of the target text area.

3. The method according to claim 1, wherein: The fusing process of the first text category and the second text category of the target text area to obtain the target category of the target text area includes: Identifying whether the first text category of the target text area is the same as the second text category; According to the recognition result, a target category of the target text area is selected from the first text category and the second text category.

4. The method according to claim 3, wherein: The step of selecting the target category of the target text area from the first text category and the second text category according to the recognition result includes: If the recognition result is that the first text category is different from the second text category, a target category of the target text area is selected from the first text category and the second text category according to the probability value of the first text category and the probability value of the second text category.

5. The method according to claim 1, after fusing the first text category and the second text category of the target text area to obtain the target category of the target text area, further comprises: The target text image is associated with the position information, text content and target category of the target text area in the target text image and stored.

6. The method according to any one of claims 1 to 5, wherein: The target text image is a target signboard image.

7. An image processing device, comprising: The first category determination module includes: An image feature extraction unit, used for extracting image features of a target text image based on a feature extraction network in a visual segmentation model; A candidate region determination unit, configured to determine a candidate text region in the target text image based on a candidate region generation network in the visual segmentation model and according to the image features; A screening unit, used for screening the candidate text regions by adopting a multi-category non-maximum suppression algorithm; A first category determination unit is used to process the filtered text area to obtain a target text area in the target text image and a first text category of the target text area; A second category determination module, used for classifying the text content in the target text area to obtain a second text category of the target text area; A target category determination module, configured to fuse the first text category and the second text category of the target text area to obtain a target category of the target text area; Among them, the specific implementation of the visual segmentation model, the text recognition model and the text classification model is as follows: the sample text image is input into the feature extraction network of the visual segmentation model to obtain the sample image features; the sample image features are input into the candidate region generation network to obtain the candidate text region; the candidate text region is screened using a multi-category non-maximum algorithm to obtain the screened text region; the screened text region is input into the result output network of the visual segmentation model, and the result output network includes three branches, namely, the position branch, the region branch and the category branch, to obtain the text box position of the sample text image, i.e., the sample text region position, the segmentation result, i.e., the segmented text box, i.e., the sample text region, and the first text category of the sample text region; and then the sample text region is input into the text Recognition model, obtain the text content of the sample text area; input the text content into the text classification model, obtain the second text category of the sample text area; determine the category loss according to the first text category predicted by the visual segmentation model, the second text category predicted by the text classification model, and the annotated category data, that is, fuse the category losses of the visual segmentation model and the text classification model, and determine the segmentation loss according to the annotated text box data and the segmentation result of the visual segmentation model, that is, the segmented text box; and determine the position loss according to the annotated text box position and the text box position predicted by the visual segmentation model, that is, the position of the sample text area; and then train the visual segmentation model, the text recognition model and the text classification model together according to the classification loss, segmentation loss and position loss.

8. The device according to claim 7, wherein: The second category determination module comprises: A text content determination unit, used to identify the target text area based on a text recognition model to obtain the text content of the target text area; The second category determination unit is used to classify the text content based on a text classification model to obtain a second text category of the target text area.

9. The device according to claim 7, wherein: The target category determination module comprises: an identification unit, configured to identify whether the first text category of the target text area is the same as the second text category; The target category selection unit is used to select a target category of the target text area from the first text category and the second text category according to the recognition result.

10. The device according to claim 9, wherein: The target category selection unit is specifically used for: If the recognition result is that the first text category is different from the second text category, a target category of the target text area is selected from the first text category and the second text category according to the probability value of the first text category and the probability value of the second text category.

11. The apparatus according to claim 7, further comprising: The storage module is used to associate and store the target text image with the position information, text content and target category of the target text area in the target text image.

12. The device according to any one of claims 7 to 11, wherein: The target text image is a target signboard image.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the image processing method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the image processing method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the image processing method according to any one of claims 1 to 6.

Images