Title image recognition method and device, electronic equipment and storage medium
By identifying question images through a three-layer judgment method, combining image features, text features, and similarity, the problem of filtering non-question images in existing technologies is solved, thereby improving the security and data protection of the Q&A system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 深圳市星桐科技有限公司
- Filing Date
- 2022-11-09
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies are insufficient to effectively filter images that are not part of the question, leading to data leaks in the Q&A system. Users may use these non-question images to request answers, causing system vulnerabilities to be exploited.
A three-layer judgment method is adopted to identify the question image. First, it is based on image features, then it combines image and text features, and finally it is determined by the similarity with the preset question, which gradually improves the recognition accuracy. It includes a lightweight image binary classification model, an OCR module and fully connected network processing.
It enables precise filtering of images not related to the question, improving the security of the Q&A system and preventing data leakage.
Smart Images
Figure CN115761717B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and more particularly to a method, apparatus, electronic device, and storage medium for recognizing title images. Background Technology
[0002] When using the Q&A system, users can upload their questions as images. The system will then automatically return a question with a detailed answer that is most similar to the original question based on the uploaded image.
[0003] In practical applications, users might randomly take a picture of an image that is not part of the question, and these images do not require a solution. However, existing systems often struggle to filter out these non-question images, such as product advertisements, newspaper clippings, product manuals, etc. This makes it easy for vulnerabilities in the Q&A system to be exploited. Users can technically simulate generating non-question images, request the Q&A system, and scrape the system's legitimate questions and answer explanations, leading to data leaks.
[0004] Therefore, there is an urgent need for a method to identify title images and filter out non-title images. Summary of the Invention
[0005] In view of this, the present disclosure provides a method, apparatus, electronic device and storage medium for identifying title images, which can achieve accurate filtering of non-title images.
[0006] According to one aspect of this disclosure, a method for recognizing title images is provided, the method comprising:
[0007] Acquire the target image to be identified;
[0008] Based on the image features of the target image, determine whether the target image is a question image;
[0009] In response to determining that the target image is a title image based on the image features of the target image, the confidence level of the target image being a title image is determined based on the image features and text features of the target image;
[0010] Based on the confidence level, it is determined again whether the target image is the title image;
[0011] In response to determining that the target image is a question image based on the confidence level, the system further determines whether the target image is a question image based on the confidence level and the similarity between the target image and each preset question.
[0012] Based on the confidence level and the similarity between the target image and each preset question, the target image is determined to be a question image, and the question recognition result corresponding to the target image is determined.
[0013] According to another aspect of this disclosure, a device for recognizing title images is provided, the device comprising:
[0014] The acquisition module is used to acquire the target image to be identified;
[0015] The first judgment module is used to determine whether the target image is a question image based on the image features of the target image;
[0016] The second judgment module is used to respond to determining that the target image is a question image based on the image features of the target image, determining the confidence level that the target image is a question image based on the image features and text features of the target image, and further determining whether the target image is a question image based on the confidence level.
[0017] The third judgment module is used to determine whether the target image is a question image based on the confidence level, and to determine whether the target image is a question image again based on the confidence level and the similarity between the target image and each preset question.
[0018] The determination module is configured to, in response to the confidence level and the similarity between the target image and each preset question, determine that the target image is a question image and determine the question recognition result corresponding to the target image.
[0019] According to another aspect of this disclosure, an electronic device is provided, comprising:
[0020] Processor; and
[0021] Stored program memory,
[0022] The program includes instructions that, when executed by the processor, cause the processor to perform the image recognition method.
[0023] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions, wherein the computer instructions are used to cause a computer to execute the above-mentioned image recognition method.
[0024] In this disclosure, after acquiring the target image to be identified, a three-layer judgment can be performed to determine whether it is a question image. The first layer of judgment is based on the image features of the target image; the second layer is based on both the image and text features of the target image; and the third layer is based on the similarity between the target image and various preset questions, as well as the confidence score calculated in the second layer. The amount of information used in each of the three layers gradually increases, and the accuracy gradually improves. Therefore, using this disclosure, accurate filtering of non-question images can be achieved. Attached Figure Description
[0025] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:
[0026] Figure 1 A flowchart illustrating a method for recognizing title images according to exemplary embodiments of the present disclosure is shown.
[0027] Figure 2 A flowchart of the second determination process provided according to an exemplary embodiment of the present disclosure is shown;
[0028] Figure 3 A schematic diagram of a fully connected network provided according to an exemplary embodiment of the present disclosure is shown;
[0029] Figure 4 A schematic block diagram of a multimedia search device provided according to an exemplary embodiment of the present disclosure is shown;
[0030] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation
[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0032] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0033] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.
[0034] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0035] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0036] This disclosure provides a method for recognizing title images. This method can perform multiple recognitions on non-title images, further improving the accuracy of non-title image recognition. This method can be performed by a terminal, server, and / or other devices with processing capabilities. The method provided in the embodiments of this disclosure can be performed by any of the above-mentioned devices, or by multiple devices working together; this disclosure does not limit this.
[0037] The following will refer to Figure 1 The flowchart of the image recognition method is shown, and the method is introduced. The method includes the following steps 101-105.
[0038] Step 101: Obtain the target image to be identified.
[0039] In some applications, target images can be identified to determine whether they are question images. For example, when a user needs to look up the answer to a question, they can use a terminal to take a picture of a workbook or similar book and use the captured image as the target image. Another example is in image processing where the original image can be cropped, and each portion can be determined to be a question image.
[0040] This embodiment does not limit the specific application scenarios for recognizing question images.
[0041] Step 102: Based on the image features of the target image, determine whether the target image is the title image.
[0042] In one possible implementation, after acquiring the target image, a preliminary assessment can be made. Based on the image features of the target image, images that differ significantly from the title image can be initially filtered out.
[0043] Optionally, the processing in step 102 above can be as follows: input the target image into a lightweight image binary classification model, and determine the probability that the target image is the title image based on the image features of the target image in the lightweight image binary classification model; in response to the probability being greater than or equal to a preset probability threshold, determine that the target image is the title image.
[0044] The lightweight image binary classification model can employ models such as SqueezeNet, MobileNet, and ShuffleNet; this embodiment does not limit the specific model used. Employing a lightweight image binary classification model can improve system efficiency while ensuring the recall of normal question images.
[0045] In one possible implementation, a lightweight image binary classification model can be pre-trained to fully learn the image features of the question image, thereby enabling it to perform binary classification judgments based on these features, classifying the image as either "the question image" or "not the question image." This embodiment does not limit the specific training process.
[0046] The target image is preprocessed, such as by scaling to a fixed size or performing image normalization, to adapt it to a lightweight image binary classification model. The preprocessed target image is then input into the lightweight image binary classification model, where the image data is processed to determine the probability that the target image belongs to the question image, and then it is determined whether this probability is greater than a preset probability threshold.
[0047] When the probability is less than the probability threshold, the target image is considered not to be the question image, and a corresponding prompt message can be returned to remind the user that the target image is not the question image. Furthermore, no further processing is required.
[0048] When the probability is greater than or equal to the probability threshold, the target image is considered to be the image of the question, and the process can proceed to step 103 for further processing.
[0049] Step 103: In response to determining that the target image is a title image based on the image features of the target image, the confidence level of the target image being a title image is determined based on the image features and text features of the target image, and based on the confidence level, it is determined again whether the target image is a title image.
[0050] In one possible implementation, after the target image passes the first judgment described above, image features and text features can be used to further determine whether it is a title image.
[0051] The specific steps can be as follows:
[0052] Text recognition is performed on the target image to obtain the text feature vector of the target image;
[0053] Image feature extraction is performed on the target image to obtain the image feature vector of the target image;
[0054] The first fully connected network is used to extract features from the text feature vector and the image feature vector to obtain the intermediate feature vector;
[0055] The confidence level of the target image as the question image is determined based on the intermediate feature vector through a second fully connected network.
[0056] In one possible implementation, see Figure 2 The second processing flow shown involves inputting the target image into an OCR (Optical Character Recognition) module for text recognition. The OCR module can include a text detection module and a text recognition module. In the OCR module, the target image is input into the text detection module, which processes the target image and outputs the text line position information. Then, this text line position information and the target image are input into the text recognition module to extract the text feature vector corresponding to the text content in the target image. Optionally, after performing text recognition on the target image, the text content in the target image can also be obtained.
[0057] The target image is input into a feature extraction module (such as the backbone network of a MobileNet model), which outputs an image feature vector. This image feature vector and the aforementioned text feature vector are then input into a first fully connected network. The first fully connected network extracts features from the image and text feature vectors, outputting an intermediate feature vector. This intermediate feature vector is then input into a second fully connected network. The second fully connected network calculates the confidence score that the target image is the question image, thus determining whether it is indeed the question image.
[0058] When the confidence level is less than a preset confidence threshold, the target image is considered not to be the question image. In this case, a corresponding prompt can be returned to remind the user that the target image is not the question image. Furthermore, no further processing can be performed.
[0059] When the confidence level is greater than or equal to the preset confidence threshold, the target image is considered to be the title image, and the process can proceed to step 104 for further processing.
[0060] Optionally, after obtaining the intermediate feature vector, the intermediate feature vector can be further processed by a third fully connected network to extract features from the intermediate feature vector, thereby obtaining the target feature vector of the target image. This target feature vector carries both image feature information and text feature information, that is, it simultaneously represents image features and text features.
[0061] like Figure 3 The diagram shown illustrates a fully connected network. The fully connected network can include a first fully connected network, a second fully connected network, and a third fully connected network. The second fully connected network can be used to determine the confidence that the target image is the question image, and the third fully connected network can be used to determine the target feature vector.
[0062] The image feature vector and text feature vector are input into the first fully connected network. The resulting intermediate feature vector can then be input into the second and third fully connected networks, respectively. After processing by the second fully connected network, the output is the confidence score that the target image is the question image. After processing by the third fully connected network, the output is the target feature vector.
[0063] It should be noted that the OCR module described above can be replaced with other models that can be used for text recognition, such as models based on Attention; the feature extraction model described above can also be replaced with other models that can be used to extract image features, such as convolutional neural networks. This embodiment does not limit the specific model used.
[0064] The use of the lightweight MobileNet model reduces the amount of processing required, thus enabling the image recognition method provided in this disclosure to be applied to mobile devices as well, thereby expanding the scope of application of this disclosure.
[0065] Step 104: In response to determining whether the target image is a question image based on confidence, further determine whether the target image is a question image based on confidence and the similarity between the target image and each preset question.
[0066] In one possible implementation, question content can be pre-stored in the ES (Elastic Search) retrieval database. This question content may include the text content of each question (such as the question stem, answer content, etc.) and the corresponding question image.
[0067] When the target image passes the second judgment described above, the similarity between the target image and each preset question can be calculated. Optionally, this similarity may refer to a first similarity determined based on the target feature vector, and / or a second similarity determined based on the text content. The similarity between the target image and each preset question is determined as follows: determining the first similarity between the target feature vector of the target image and the target feature vector of each preset question, and / or determining the second similarity between the text content of the target image and the text content of each preset question.
[0068] For similarity calculation based on target feature vectors, the following are possible implementation methods:
[0069] By constructing the target feature vector in the same way as described above, the text feature vector and image feature vector of each preset question are extracted, and the target feature vector of the preset question is output. Therefore, after obtaining the target feature vector of the target image in step 103 above, each preset question can be traversed in the ES search library to calculate the first similarity between the target feature vector of the target image and the target feature vector of the preset question. For example, a cosine similarity algorithm can be used for calculation; this embodiment does not limit the specific vector similarity algorithm. Optionally, to improve processing efficiency, the question content and similarity of the k1 preset questions with the highest first similarity (k1 is an integer greater than 0) can be obtained and returned for subsequent processing.
[0070] For text content-based similarity calculation, the following are possible implementation methods:
[0071] The text content of the target image identified in the above process is obtained. Each preset question is then traversed in the ES search database to calculate the second similarity between the text content of the target image and the text content of the preset question. For example, the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm can be used for calculation; this embodiment does not limit the specific text similarity algorithm. Optionally, to improve processing efficiency, the question content and similarity of the k2 preset questions with the highest second similarity (k2 is an integer greater than 0) can be obtained and returned for subsequent processing.
[0072] Optionally, after calculating the similarity, the processing in step 104 above includes: based on the similarity between the target image and each preset question, determining the benchmark question among each preset question, and obtaining the target similarity corresponding to the benchmark question; and determining whether the target image is a question image based on the confidence level and the similarity corresponding to the benchmark question.
[0073] In one possible implementation, after obtaining the content of each preset question and its corresponding similarity, the confidence level of the target image calculated in step 103 can be obtained. Using the confidence level of the target image, the returned question content, and the corresponding similarity as input, a fine-ranking algorithm is used to precisely sort the returned preset questions, resulting in a sorted question sequence. Alternatively, the determined similarities can be simply sorted from largest to smallest to obtain a question sequence corresponding to the sorted similarity sequence.
[0074] In the question sequence, the question ranked first (i.e. the question that best matches the target image) can be used as the benchmark question, or the question ranked at the preset number (e.g., the 5th question) can be used as the benchmark question, and the similarity of the benchmark question can be obtained.
[0075] Furthermore, the similarity score of the benchmark question can be calculated using the following formula:
[0076] Pq = W1 * Pq1 + W2 * Pq2
[0077] In the formula, Pq is the similarity score of the benchmark question, Pq1 is the confidence score of the target image, Pq2 is the target similarity score corresponding to the benchmark question, W1 is the weight corresponding to the confidence score of the target image, and W2 is the weight corresponding to the similarity score of the benchmark question.
[0078] The similarity score of the benchmark question is compared with a preset score threshold. If the similarity score is greater than the score threshold, the target image is considered to be the question image; if the similarity score is not greater than the score threshold, the target image is considered to be a non-question image.
[0079] Step 105: In response to the determination of the target image as a question image based on the confidence level and the similarity between the target image and each preset question, the question recognition result corresponding to the target image is determined.
[0080] In one possible implementation, if the target image passes the third judgment, it can be determined that the target image is a question image. Then, the question content of the first preset number of questions in the question sequence can be obtained as the corresponding question recognition result.
[0081] If the target image is determined to be a non-question image in step 104, the corresponding prompt information can be returned as the corresponding question recognition result to remind the user that the target image is not a question image.
[0082] In this embodiment, after acquiring the target image to be identified, a three-layer judgment can be performed to determine whether it is a question image. The first layer of judgment is based on the image features of the target image; the second layer is based on both the image and text features of the target image; and the third layer is based on the similarity between the target image and various preset questions, as well as the confidence level calculated in the second layer. The amount of information used in each of the three layers gradually increases, and the accuracy gradually improves. Therefore, by employing this disclosure, accurate filtering of non-question images can be achieved.
[0083] This disclosure provides a device for recognizing title images, which is used to implement the aforementioned method for recognizing title images. For example... Figure 4 The schematic block diagram shown indicates that the image recognition device 400 includes: an acquisition module 401, a first judgment module 402, a second judgment module 403, a third judgment module 404, and a determination module 405.
[0084] The acquisition module 401 is used to acquire the target image to be identified;
[0085] The first judgment module 402 is used to determine whether the target image is a question image based on the image features of the target image;
[0086] The second judgment module 403 is used to respond to determining that the target image is a question image based on the image features of the target image, determining the confidence level that the target image is a question image based on the image features and text features of the target image, and further determining whether the target image is a question image based on the confidence level.
[0087] The third judgment module 404 is used to determine whether the target image is a question image based on the confidence level, and to determine whether the target image is a question image again based on the confidence level and the similarity between the target image and each preset question.
[0088] The determination module 405 is configured to determine, in response to the confidence level and the similarity between the target image and each preset question, that the target image is a question image and to determine the question recognition result corresponding to the target image.
[0089] Optionally, the first determination module 402 is used for:
[0090] The target image is input into a lightweight image binary classification model, and the probability that the target image is the title image is determined based on the image features of the target image in the lightweight image binary classification model;
[0091] In response to the probability being greater than or equal to a preset probability threshold, the target image is determined to be the title image.
[0092] Optionally, the second judgment module 403 is used for:
[0093] Text recognition is performed on the target image to obtain the text feature vector of the target image;
[0094] Image feature extraction is performed on the target image to obtain the image feature vector of the target image;
[0095] The text feature vector and the image feature vector are extracted using a first fully connected network to obtain an intermediate feature vector.
[0096] The confidence level that the target image is the question image is determined based on the intermediate feature vector through the second fully connected network.
[0097] Optionally, the second judgment module 403 is further configured to: after performing text recognition on the target image, obtain the text content in the target image; after obtaining the intermediate feature vector, perform feature extraction on the intermediate feature vector through a third fully connected network to obtain the target feature vector of the target image, wherein the target feature vector carries image feature information and text feature information;
[0098] The second judgment module 403 is used for:
[0099] Determine the first similarity between the target feature vector of the target image and the target feature vector of each preset question; and / or
[0100] Determine the second similarity between the text content of the target image and the text content of each preset question.
[0101] Optionally, the third determination module 404 is used for:
[0102] Based on the similarity between the target image and each preset question, a benchmark question is determined among the preset questions, and the target similarity corresponding to the benchmark question is obtained, wherein the similarity includes a first similarity determined based on the target feature vector and / or a second similarity determined based on the text content;
[0103] The similarity score of the benchmark questions is calculated using the following formula:
[0104] Pq = W1 * Pq1 + W2 * Pq2
[0105] In the formula, Pq is the similarity score of the benchmark question, Pq1 is the confidence score of the target image, Pq2 is the target similarity score corresponding to the benchmark question, W1 is the weight corresponding to the confidence score of the target image, and W2 is the weight corresponding to the target similarity score of the benchmark question.
[0106] In response to the similarity score being greater than a preset score threshold, the target image is determined to be the title image.
[0107] In this embodiment, after acquiring the target image to be identified, a three-layer judgment can be performed to determine whether it is a question image. The first layer of judgment is based on the image features of the target image; the second layer is based on both the image and text features of the target image; and the third layer is based on the similarity between the target image and various preset questions, as well as the confidence level calculated in the second layer. The amount of information used in each of the three layers gradually increases, and the accuracy gradually improves. Therefore, by employing this disclosure, accurate filtering of non-question images can be achieved.
[0108] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.
[0109] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.
[0110] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.
[0111] refer to Figure 5 The present invention describes a structural block diagram of an electronic device 500 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0112] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0113] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or text information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disks and optical discs. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.
[0114] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the above-described question image recognition method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the above-described question image recognition method by any other suitable means (e.g., by means of firmware).
[0115] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0116] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0117] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0120] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.
Claims
1. A method for recognizing title images, characterized in that, The method includes: Acquire the target image to be identified; Based on the image features of the target image, determine whether the target image is a question image; In response to determining that the target image is a title image based on the image features of the target image, the confidence level of the target image being a title image is determined based on the image features and text features of the target image; Based on the confidence level, it is determined again whether the target image is the title image; In response to determining that the target image is a question image based on the confidence level, the system further determines whether the target image is a question image based on the confidence level and the similarity between the target image and each preset question. Based on the confidence level and the similarity between the target image and each preset question, the target image is determined to be a question image, and the question recognition result corresponding to the target image is determined; The determination of the confidence level that the target image is the title image based on the image features and text features of the target image includes: Text recognition is performed on the target image to obtain the text feature vector of the target image; Image feature extraction is performed on the target image to obtain the image feature vector of the target image; The text feature vector and the image feature vector are extracted using a first fully connected network to obtain an intermediate feature vector. The confidence level that the target image is the question image is determined based on the intermediate feature vector through the second fully connected network.
2. The method according to claim 1, characterized in that, Determining whether the target image is a title image based on the image features of the target image includes: The target image is input into a lightweight image binary classification model, and the probability that the target image is the title image is determined based on the image features of the target image in the lightweight image binary classification model; In response to the probability being greater than or equal to a preset probability threshold, the target image is determined to be the title image.
3. The method according to claim 1, characterized in that, After performing text recognition on the target image, the method further includes: obtaining the text content in the target image; After obtaining the intermediate feature vector, the method further includes: extracting features from the intermediate feature vector through a third fully connected network to obtain the target feature vector of the target image, wherein the target feature vector carries image feature information and text feature information; The similarity between the target image and each preset question is determined in the following way: Determine the first similarity between the target feature vector of the target image and the target feature vector of each preset question; and / or Determine the second similarity between the text content of the target image and the text content of each preset question.
4. The method according to any one of claims 1-3, characterized in that, The step of determining whether the target image is a question image again based on the confidence level and the similarity between the target image and each preset question includes: Based on the similarity between the target image and each preset question, a benchmark question is determined among the preset questions, and the target similarity corresponding to the benchmark question is obtained, wherein the similarity includes a first similarity determined based on the target feature vector and / or a second similarity determined based on the text content; The similarity score of the benchmark questions is calculated using the following formula: Pq = W1*Pq1 + W2*Pq2 In the formula, Pq is the similarity score of the benchmark question, Pq1 is the confidence score of the target image, Pq2 is the target similarity score corresponding to the benchmark question, W1 is the weight corresponding to the confidence score of the target image, and W2 is the weight corresponding to the target similarity score of the benchmark question. In response to the similarity score being greater than a preset score threshold, the target image is determined to be the title image.
5. A device for recognizing title images, characterized in that, The device includes: The acquisition module is used to acquire the target image to be identified; The first judgment module is used to determine whether the target image is a question image based on the image features of the target image; The second judgment module is used to respond to determining that the target image is a question image based on the image features of the target image, determining the confidence level that the target image is a question image based on the image features and text features of the target image, and further determining whether the target image is a question image based on the confidence level. The third judgment module is used to determine whether the target image is a question image based on the confidence level, and to determine whether the target image is a question image again based on the confidence level and the similarity between the target image and each preset question. The determination module is configured to, in response to the confidence level and the similarity between the target image and each preset question, determine that the target image is a question image and determine the question recognition result corresponding to the target image; The second judgment module is used for: Image feature extraction is performed on the target image to obtain the image feature vector of the target image; Text recognition is performed on the target image to obtain the text feature vector of the target image; The text feature vector and the image feature vector are extracted using a first fully connected network to obtain an intermediate feature vector. The confidence level that the target image is the question image is determined based on the intermediate feature vector through the second fully connected network.
6. The apparatus according to any one of claims 5, characterized in that, The third judgment module is used for: Based on the similarity between the target image and each preset question, a benchmark question is determined among the preset questions, and the target similarity corresponding to the benchmark question is obtained, wherein the similarity includes a first similarity determined based on the target feature vector and / or a second similarity determined based on the text content; The similarity score of the benchmark questions is calculated using the following formula: Pq = W1*Pq1 + W2*Pq2 In the formula, Pq is the similarity score of the benchmark question, Pq1 is the confidence score of the target image, Pq2 is the target similarity score corresponding to the benchmark question, W1 is the weight corresponding to the confidence score of the target image, and W2 is the weight corresponding to the target similarity score of the benchmark question. In response to the similarity score being greater than a preset score threshold, the target image is determined to be the title image.
7. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-4.
8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.