Question area identification method and related device

Abstract

The invention discloses a question area identification method and a related device, and relates to the technical field of picture processing, and the question area identification method comprises the steps: obtaining a target picture containing a plurality of questions; performing text recognition on the target picture to obtain text content and position information of each text line in the target picture; utilizing a large language model to generate text content of each question according to the text content of each text line and / or the target picture; matching the text content of each text line with the text content of each question to obtain a text line belonging to each question; and according to the position information of the text line belonging to each question, determining an area where each question is located in the target picture. The question area identification method provided by the invention is simple and efficient, and the identification effect is good.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for identifying a topic area and a related device. Background Art

[0002] In certain fields (such as education and document processing), it is necessary to identify question regions within images containing multiple questions (e.g., test papers or exercise pictures). Question region recognition involves identifying the region within an image containing multiple questions, i.e., the question region for each question.

[0003] The current solution for question area recognition is to obtain a large number of training images with the question area marked, then use these training images to train a question area recognition model, and use the trained question area recognition model to perform question area recognition on the image to be recognized.

[0004] Although the current question area recognition scheme can realize the recognition of the question area, the process of the scheme is cumbersome and the recognition effect is poor. Summary of the Invention

[0005] In view of this, the present application provides a method and related apparatus for identifying a question area, which is used to solve the problems of the current question area identification scheme having a cumbersome process and poor identification effect. The technical solution is as follows:

[0006] In a first aspect, the present application provides a method for identifying a topic area, comprising:

[0007] Get a target image containing several questions;

[0008] Performing text recognition on the target image to obtain text content and position information of each text line in the target image;

[0009] Using a large language model, generating text content for each question based on the text content of each text line and / or the target image;

[0010] Obtaining a text line belonging to each topic by matching the text content of each text line with the text content of each topic;

[0011] The region where each question is located in the target image is determined according to the position information of the text line belonging to each question.

[0012] In a possible implementation, the topic area identification method further includes:

[0013] According to the area where each question is located in the target image, a regional selection box for each question is marked on the target image.

[0014] In one possible implementation, using a large language model to generate text content for each question based on the text content of each text line and / or the target image includes:

[0015] A multimodal large language model is used to generate a topic text list based on the text content of each text line and in combination with the target image, wherein the topic text list includes the number and text content of each topic.

[0016] In a possible implementation, obtaining the text line belonging to each topic by matching the text content of each text line with the text content of each topic includes:

[0017] For each question, determine the similarity between the text content of each text line and the text content of the question, and obtain the text similarity between each text line and the question;

[0018] A text line whose text similarity with the topic is greater than a preset similarity threshold is determined as a text line belonging to the topic.

[0019] In a possible implementation, determining the region in the target image where each title is located based on the position information of the text line belonging to each title includes:

[0020] For each question, the minimum enclosing rectangle of all text lines belonging to the question is determined based on the position information of the text lines belonging to the question, and the area where the question is located is obtained.

[0021] In a possible implementation, the position information of any text line in the target image is represented by the coordinates of the upper left corner and the lower right corner of the text line;

[0022] The step of determining the minimum enclosing rectangle of all text lines belonging to the topic based on the position information of the text lines belonging to the topic includes:

[0023] Obtain the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate and maximum vertical coordinate from the upper left corner coordinates and lower right corner coordinates of several text lines belonging to the question;

[0024] A rectangle with (minimum horizontal coordinate, minimum vertical coordinate) as the upper left vertex and (maximum horizontal coordinate, maximum vertical coordinate) as the lower right vertex is determined as the minimum enclosing rectangle of all text lines belonging to the question.

[0025] A second aspect of the present application provides a topic area identification device, comprising: an image acquisition module, a text recognition module, a topic text generation module, a text matching module, and a topic area determination module;

[0026] The image acquisition module is used to acquire target images containing several topics;

[0027] The text recognition module is used to perform text recognition on the target image to obtain the text content and position information of each text line in the target image;

[0028] The topic text generation module is used to generate the text content of each topic based on the text content of each text line and / or the target image using a large language model;

[0029] The text matching module is used to obtain a text line belonging to each question by matching the text content of each text line with the text content of each question;

[0030] The topic area determination module is configured to determine the area where each topic is located in the target image based on the position information of the text line belonging to each topic.

[0031] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is configured to execute the computer program so as to enable the electronic device to implement the steps of any one of the above-mentioned methods for identifying a topic area.

[0034] In a fourth aspect, the present application provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned question area identification methods.

[0035] A fifth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the steps of any one of the above-mentioned methods for identifying a topic area.

[0036] By means of the above technical solution, the question area recognition method provided by the present application, after obtaining a target image containing several questions, first obtains the text content and position information of each text line in the target image by performing text recognition on the target image, then uses a large language model to generate the text content of each question according to the text content of each text line and / or the target image, then obtains the text line belonging to each question by matching the text content of each text line with the text content of each question, and finally determines the question area of ​​each question in the target image according to the position information of the text line belonging to each question. The question area recognition method provided by the present application, based on traditional text recognition, combined with the semantic understanding ability of the large language model, can accurately determine the text line belonging to each question, and then accurately determine the question area of ​​each question according to the position of the text line belonging to each question. Since the question area recognition method provided by the present application is based on traditional text recognition and combines the large language model to process the target image to obtain the question area, without obtaining a large amount of training data to train the model, the question area recognition process is simplified. In summary, the question area recognition method provided by the present application is simple and efficient, and has a good recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0038] Figure 1 A schematic diagram of a system architecture involved in this application;

[0039] Figure 2 A schematic diagram of the hardware structure of a terminal provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the hardware structure of a server provided in an embodiment of the present application;

[0041] Figure 4 A flowchart of a method for identifying a topic area according to an embodiment of the present application;

[0042] Figure 5 This is a structural diagram of the topic area identification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0044] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0046] In one possible implementation, Figure 1 As shown, the system architecture involved in this application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The example includes one server. Terminal 101 may obtain a target image including several questions and send the target image to server 102. Server 102 may identify the area where each question is located in the target image and further mark the area selection box of each question in the target image based on the recognition result.

[0047] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities and can obtain a target image including a plurality of questions, identify the region where each question is located in the target image, and further mark the region selection box of each question in the target image based on the recognition result.

[0048] Next, the product form of the above-mentioned terminal will be described.

[0049] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0050] Figure 2 A schematic diagram of an optional hardware structure of a terminal is shown.

[0051] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will understand that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0052] The input unit 230 can be used to receive input digital or character information and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect user touch operations on or near it (for example, operations performed on or near the touch screen using a finger, joint, stylus, or any other suitable object) and drive corresponding connected devices according to a pre-set program. The touch screen can detect user touch actions on the touch screen, convert the touch actions into touch signals and transmit them to the processor 270, and can receive and execute commands sent by the processor 270; the touch signals include at least touch point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, touch screens can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 231, the input unit 230 may also include other input devices. Specifically, the other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, a joystick, and the like.

[0053] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0054] Memory 220 can be used to store instructions and data. Memory 220 primarily includes an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files and text. The instruction storage area can store software units such as the operating system, applications, and instructions required for at least one function, or subsets or extensions thereof. It may also include non-volatile random access memory (RAM). It provides processor 270 with management functions for the hardware, software, and data resources within the computing and processing device, supporting control software and applications. It is also used to store multimedia files and running programs and applications.

[0055] The processor 270 is the terminal's control center, connecting all components of the terminal using various interfaces and circuits. By executing instructions stored in memory 220 and accessing data stored in memory 220, it performs various terminal functions and processes data, thereby providing overall control of the terminal. Optionally, the processor 270 may include one or more processing units. Preferably, the processor 270 integrates an application processor and a modem processor, with the application processor primarily processing the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 270. In some embodiments, the processor and memory may be implemented on a single chip; in other embodiments, they may be implemented on separate chips. The processor 270 may also generate corresponding operational control signals and send them to corresponding components of the computing and processing device. It may also read and process data in the software, particularly the data and programs in memory 220, to enable the various functional modules therein to perform their corresponding functions, thereby controlling the corresponding components to operate as instructed.

[0056] The memory 220 may be used to store software codes related to the question area identification method. The processor 270 may execute the software codes in the memory 220 and may also schedule other units (such as the input unit 230 and the display unit 240 ) to implement corresponding functions.

[0057] The RF unit 210 (optional) can be used to send and receive information or receive and send signals during a call. For example, it receives downlink information from the base station and passes it to the processor 270 for processing; in addition, it sends uplink data to the base station. Typically, the RF unit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, and the like. Furthermore, the RF unit 210 can communicate with network devices and other devices via wireless communication. This wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), and the like.

[0058] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices and receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional and can be replaced by other communication interfaces, such as a network port.

[0059] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0060] The terminal further includes an external interface 280 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to other devices for communication, or to connect a charger for charging the terminal.

[0061] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0062] Next, the product form of the above server will be described.

[0063] Figure 3 A structural diagram of the above server is provided, such as Figure 3As shown, the server may include a bus 301 , a processing device 302 , a communication interface 303 , and a storage device 304 . The processing device 302 , the storage device 304 , and the communication interface 303 communicate with each other via the bus 301 .

[0064] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0065] The processing device 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0066] The storage device 304 may include volatile memory, such as random access memory (RAM). The storage device 304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0067] The storage device 304 may be used to store software codes related to the question area identification method. The processing device 302 may call the software codes stored in the storage device 304 or schedule other units to implement corresponding functions.

[0068] The processor 270 in the above-mentioned terminal and the processing device 302 in the server can be a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, it can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0069] Existing methods for identifying question areas require collecting a large number of pictures containing questions and marking the question areas on the collected pictures. After obtaining a large number of training pictures with the question areas marked, these training pictures are used to train a model. After the model is trained, the trained model is used to identify the question areas on the pictures to be identified. It can be seen that the existing methods for identifying question areas are cumbersome. In addition, since the existing solutions do not consider the semantic level of the questions, the recognition effect is poor.

[0070] In order to solve the problems existing in the existing question area recognition methods, the present application proposes a question area recognition method which is relatively simple, efficient and has a good recognition effect. The question area recognition method provided by the present application is introduced through the following embodiments.

[0071] The question area identification method provided in the embodiment of the present application can be applied to a variety of application scenarios, such as question bank construction scenario (identifying the area where the question is located, and then extracting the question to build a question bank), photo search scenario (identifying the question area from the question picture uploaded by the user and matching the answer to the question bank), question retrieval scenario (identifying the question area, and then retrieving similar questions), etc.

[0072] See also Figure 4 , which shows a flow chart of a method for identifying a topic area according to an embodiment of the present application. The method for identifying a topic area may include:

[0073] Step S401: Obtain a target picture including several questions.

[0074] In this embodiment, the target image may be an image containing several questions obtained by photographing, scanning, or screenshotting, for example, an image obtained by photographing a test paper, an image obtained by scanning an exercise book, and so on.

[0075] Step S402: Perform text recognition on the target image to obtain the text content and position information of each text line in the target image.

[0076] In one possible implementation, an optical character recognition (OCR) model (such as a PaddleOCR model) may be used to perform text recognition on a target image to obtain the text content and position information of each text line in the target image.

[0077] Step S403: Generate the text content of each question based on the text content of each text line and / or the target image using the large language model.

[0078] This embodiment uses the semantic understanding capability of a large language model to perform semantic understanding and analysis on the text content of each text line and / or target image to generate the text content of each question.

[0079] Step S404: obtaining the text line belonging to each question by matching the text content of each text line with the text content of each question.

[0080] For each question in the target image, the text line belonging to the question is obtained by matching the text content of each text line in the target image with the text content of the question.

[0081] Step S405: Determine the area where each question is located based on the position information of the text line belonging to each question.

[0082] After obtaining the text lines associated with each question, the region of the target image where the question is located, i.e., the question region of the target image, can be determined based on the positional information of the text lines associated with the question. It should be noted that the question region of a question can include all parts of the question stem, or it can include the question stem and the answer.

[0083] Optionally, after determining the area where each question is located in the target image, a regional selection box for each question may be marked in the target image according to the area where each question is located.

[0084] The method for identifying a topic area provided in an embodiment of the present application, after obtaining a target image containing several topics, first obtains the text content and position information of each text line in the target image by performing text recognition on the target image, then uses a large language model to generate the text content of each topic based on the text content of each text line and / or the target image, then obtains the text line belonging to each topic by matching the text content of each text line with the text content of each topic, and finally determines the topic area of ​​each topic in the target image based on the position information of the text line belonging to each topic. The method for identifying a topic area provided in an embodiment of the present application, based on traditional text recognition, combines the semantic understanding ability of a large language model to accurately determine the text line belonging to each topic, and then accurately determines the topic area of ​​each topic based on the position of the text line belonging to each topic. Since the method for identifying a topic area provided in an embodiment of the present application is based on traditional text recognition and combines a large language model to process the target image to obtain the topic area, without obtaining a large amount of training data to train the model, the process of identifying a topic area is simplified. In summary, the method for identifying a topic area provided in an embodiment of the present application is simple and efficient, and has a good recognition effect.

[0085] In another embodiment of the present application, a specific implementation process of generating the text content of each title in the target image using a large language model is introduced.

[0086] There are many ways to use a large language model to generate the text content of each title in the target image. This embodiment provides the following three implementation methods.

[0087] The first implementation method is to use a large language model to generate the text content of each title in the target image based on the text content of each text line in the target image.

[0088] Specifically, using the large language model, the process of generating the text content of each title in the target image based on the text content of each text line in the target image may include:

[0089] Step a1: Obtain a preconfigured first prompt format template.

[0090] Among them, the first prompt format template includes a first task instruction and a text line information slot. The first task instruction is used to prompt the large language model to split the information in the text line information slot according to the topic and output the text content of each topic.

[0091] Step a2: Fill the text content of each text line into the text line information slot of the first prompt format template to obtain the first prompt instruction prompt.

[0092] Step a3: Input the first prompt instruction prompt into the large language model to obtain the text content of each question output by the large model.

[0093] The first prompt instruction prompt is input into the large language model, and the large language model divides the text content of each text line according to the topic and outputs the text content of each topic.

[0094] The second implementation method is to use a multimodal large language model to generate the text content of each question in the target image based on the target image.

[0095] Specifically, the process of generating the text content of each title in the target image using the multimodal large language model according to the target image may include:

[0096] Step b1: Obtain a preconfigured second prompt format template.

[0097] Among them, the second prompt format template includes a second task instruction and a picture information slot. The second task instruction is used to prompt the multimodal large language model to identify the text content of each question from the picture information in the picture information slot.

[0098] Step b2: Fill the target image into the image information slot of the second prompt format template to obtain the second prompt instruction prompt.

[0099] Step b3: input the second prompt instruction prompt into the multimodal large language model to obtain the text content of each question output by the multimodal large language model.

[0100] The second prompt instruction prompt is input into the multimodal large language model, and the multimodal large language model parses the target image according to the second task instruction to identify the text content of each question in the target image and output it.

[0101] The third implementation method: Use a multimodal large language model to generate the text content of each question in the target image based on the text content of each text line and the target image.

[0102] Specifically, the process of using a multimodal large language model to generate the text content of each question in a target image based on the text content of each text line and in combination with the target image may include:

[0103] Step c1: Obtain a preconfigured third prompt format template.

[0104] Among them, the third prompt format template includes a third task instruction, a text information slot and a picture information slot. The third task instruction is used to prompt the multimodal large language model to generate the text content of each question based on the text information in the text information slot and the picture information in the picture information slot.

[0105] Step c2: Fill the text content of each text line into the text information slot of the third prompt format template, and fill the target image into the image information slot of the third prompt format template to obtain a third prompt instruction prompt.

[0106] Step c3: input the third prompt instruction prompt into the multimodal large language model to obtain the text content of each question output by the multimodal large language model.

[0107] The third prompt instruction prompt is input into the multimodal large language model. The multimodal large language model parses the text content of each text line and the target image according to the third task instruction, generates the text content of each question and outputs it.

[0108] In one possible implementation, the large language model may be instructed to output a list of topic texts sorted by number, where the list includes the topic number and text content of each topic (e.g., topic 1: text content 1, topic 2: text content 2, ...).

[0109] Through any of the above implementation methods, the text content of each title in the target image can be obtained.

[0110] After obtaining the text content of each text line in the target image and the text content of each question, the text line belonging to each question can be obtained by matching the text content of each text line with the text content of each question. In another embodiment of the present application, the specific implementation process of obtaining the text line belonging to each question by matching the text content of each text line with the text content of each question is introduced.

[0111] The process of obtaining the text line belonging to each topic by matching the text content of each text line with the text content of each topic may include:

[0112] Step d1: For each question, determine the similarity between the text content of each text line and the text content of the question, and obtain the text similarity between each text line and the question.

[0113] There are many ways to determine the similarity between the text content of any text line and the text content of the question. This embodiment provides the following two ways.

[0114] The first implementation method is: first, convert the text content of the text line into a vector to obtain the text representation vector of the text line, and then convert the text content of the question into a vector to obtain the text representation vector of the question, and then calculate the cosine similarity between the text representation vector of the text line and the text representation vector of the question as the similarity between the text content of the text line and the text content of the question.

[0115] The second implementation method is to determine the Levenshtein distance between the text content of the text line and the text content of the question, and use the inverse of the obtained Levenshtein distance as the similarity between the text content of the text line and the text content of the question. Determining the Levenshtein distance between the text content of the text line and the text content of the question means determining the minimum number of editing operations (such as insertion, deletion, and replacement) between the text content of the text line and the text content of the question.

[0116] It should be noted that this embodiment is not limited to the above two methods for determining the similarity between the text content of a text line and the text content of the question. Other methods that can determine the similarity between two text contents are also applicable to this application.

[0117] Step d2: Determine the text lines whose text similarity with the topic is greater than a preset similarity threshold as the text lines belonging to the topic.

[0118] A similarity threshold can be preset. For any text line, after obtaining the similarity between the text content of the text line and the text content of the question, it can be determined whether the similarity is greater than the preset similarity threshold. If the similarity is greater than the preset similarity threshold, it is determined that the text line belongs to the question; otherwise, it is determined that the text line does not belong to the question.

[0119] After determining the text line belonging to each question, the area where each question is located can be determined based on the position information of the text line belonging to each question. In another embodiment of the present application, the specific implementation process of determining the area where each question is located based on the position information of the text line belonging to each question is introduced.

[0120] The specific implementation process of determining the area where each question is located based on the position information of the text lines belonging to each question may include: for each question, based on the position information of the text lines belonging to the question, determining the minimum enclosing rectangle of all text lines belonging to the question, and obtaining the area where the question is located.

[0121] In a possible implementation, the position information of any text line in the target image may be represented by the coordinates of the upper left corner and the lower right corner of the text line.

[0122] For any question, based on the position information of the text lines belonging to the question, the process of determining the minimum enclosing rectangle of all text lines belonging to the question may include: obtaining the minimum horizontal coordinate x from the upper left corner coordinates and the lower right corner coordinates of the text lines belonging to the question min , minimum ordinate y min , maximum horizontal coordinate x max and the maximum ordinate y max ; will be (minimum horizontal coordinate x min , the minimum ordinate y min ) is the upper left vertex, with (the maximum horizontal coordinate x max , the maximum vertical coordinate y max ) as the lower right vertex, is determined as the minimum enclosing rectangle for all text lines belonging to the question. It should be noted that the origin is (0, 0) in the upper left corner of the target image, with the x-axis pointing rightward and the y-axis pointing downward.

[0123] Based on the topic area identification method provided in the above embodiment, the topic area identification method will be introduced below with reference to a specific example.

[0124] In one application scenario, it is necessary to identify the question area in a test paper image. The identification process of the question area in the test paper image is as follows:

[0125] Step e1: Get the test paper image.

[0126] Among them, the test paper picture contains several questions.

[0127] Step e2: Use the OCR model to recognize the test paper image to obtain the text content and position information of each text line in the test paper image.

[0128] For ease of description and understanding, let's assume that there is a question in the test paper image. The question content is only one line, "1. Calculate the value of 1+1". Using the OCR model to recognize the test paper image, we can obtain the text content of the first text line "1. Calculate the value of 1+1" and the position information of the first text line (50, 100, 300, 150). Among them, (50, 100) is the coordinate of the upper left corner of the first text line, and (300, 150) is the coordinate of the lower right corner of the first text line.

[0129] Step e3: Using a large multimodal language model, the complete text content of each question is generated based on the text recognition results of each text line and combined with the test paper image.

[0130] According to the process of steps c1 and c2 above, the text content of the first text line "1. Calculate the value of 1+1" and the test paper picture can be organized into a third prompt prompt and input into the multimodal large language model. The multimodal large language model outputs the number "1" and the complete text content of the question "1. Calculate the value of 1+1".

[0131] Step e4: Obtain the text line belonging to each topic by matching the text content of each text line with the text content of each topic.

[0132] Calculate the similarity between the text content of the first text line "1. Calculate the value of 1+1" and the text content of the question numbered "1" "1. Calculate the value of 1+1". After calculation, the similarity between the text content of the first text line and the text content of the question numbered "1" is 1, which is greater than the preset similarity threshold of 0.1. It can be determined that the first text line belongs to the question numbered "1".

[0133] Step e5: Determine the region where each topic is located based on the position of the text line belonging to each topic.

[0134] Since there is only one question in the test paper image, and the question has only one line of text, the area where the first text line is located is the area where the question is located. That is, the rectangular area with (50, 100) as the upper left vertex and (300, 150) as the lower right vertex is the area where the question is located.

[0135] Optionally, after determining the area where each question is located, you can mark the area selection box for each question on the test paper image based on the area where each question is located. For the above example, you can mark the rectangular area with (50, 100) as the upper left vertex and (300, 150) as the lower right vertex on the test paper image.

[0136] Through the above process, the area where each question is located in the test paper image can be framed.

[0137] The embodiments of the present application also provide a device for executing the method for identifying a topic area provided in the above embodiments.

[0138] See also Figure 5 , Figure 5 This is a structural diagram of a question area identification device provided in an embodiment of the present application. The question area identification device may include: an image acquisition module 501, a text recognition module 502, a question text generation module 503, a text matching module 504 and a question area determination module 505.

[0139] The picture acquisition module 501 is used to acquire target pictures containing several topics.

[0140] The text recognition module 502 is used to perform text recognition on the target image to obtain the text content and position information of each text line in the target image.

[0141] The topic text generation module 503 is used to generate the text content of each topic based on the text content of each text line and / or the target image using the large language model.

[0142] The text matching module 504 is configured to obtain a text line belonging to each question by matching the text content of each text line with the text content of each question.

[0143] The topic area determination module 505 is configured to determine the area where each topic is located in the target image based on the position information of the text line belonging to each topic.

[0144] In a possible implementation, the topic area identification device may further include: a topic area marking module.

[0145] The question area marking module is used to mark the area selection box of each question on the target image according to the area where each question is located in the target image.

[0146] In one possible implementation, the question text generation module 503 generates the text content of each question based on the text content of each text line and / or the target image using the large language model, specifically for:

[0147] Using a multimodal large language model, a question text list is generated based on the text content of each text line and combined with the target image. The question text list includes the number and text content of each question.

[0148] In a possible implementation, the text matching module 504 includes: a text similarity determination module and a text line attribution determination module.

[0149] The text similarity determination module is used to determine the similarity between the text content of each text line and the text content of the question for each question, so as to obtain the text similarity between each text line and the question.

[0150] The text line attribution determination module is used to determine a text line whose text similarity with the topic is greater than a preset similarity threshold as a text line belonging to the topic.

[0151] In one possible implementation, the topic area determination module 505 is specifically configured to:

[0152] For each question, the minimum enclosing rectangle of all text lines belonging to the question is determined based on the position information of the text lines belonging to the question, and the area where the question is located is obtained.

[0153] In a possible implementation, the position information of any text line in the target image is represented by the coordinates of the upper left corner and the lower right corner of the text line.

[0154] When determining the minimum enclosing rectangle of all text lines belonging to the topic based on the position information of the text lines belonging to the topic, the topic area determination module 505 is specifically configured to:

[0155] Obtain the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate and maximum vertical coordinate from the upper left corner coordinates and lower right corner coordinates of several text lines belonging to the question;

[0156] A rectangle with (minimum horizontal coordinate, minimum vertical coordinate) as the upper left vertex and (maximum horizontal coordinate, maximum vertical coordinate) as the lower right vertex is determined as the minimum enclosing rectangle of all text lines belonging to the question.

[0157] The question area identification device provided in the embodiment of the present application, based on traditional text recognition, combined with the semantic understanding ability of the large language model, can accurately determine the text line belonging to each question, and then accurately determine the question area of ​​each question according to the position of the text line belonging to each question. Since the question area identification device provided in the embodiment of the present application is based on traditional text recognition and combined with the large language model to process the icon image to obtain the question area in the image, without obtaining a large amount of training data to train the model, the question area identification process is simplified. In summary, the question area identification device provided in the embodiment of the present application is simple and efficient, and has a good recognition effect.

[0158] An embodiment of the present application further provides an electronic device, which may include: at least one processor and a memory connected to the processor.

[0159] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application; the memory may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0160] The memory is used to store the computer program, and the processor is used to execute the computer program, so that the electronic device can implement the steps of the question area identification method provided in the above embodiment.

[0161] An embodiment of the present application further provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the question area identification method provided in the above embodiment.

[0162] An embodiment of the present application further provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the method for identifying a topic area provided in the above embodiment.

[0163] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0164] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0165] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0166] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A method for identifying a topic area, characterized in that: include: Get a target image containing several questions; Performing text recognition on the target image to obtain text content and position information of each text line in the target image; Using a large language model, generating text content for each question based on the text content of each text line and / or the target image; Obtaining a text line belonging to each topic by matching the text content of each text line with the text content of each topic; The region where each question is located in the target image is determined according to the position information of the text line belonging to each question.

2. The method for identifying a topic area according to claim 1, wherein: Also includes: According to the area where each question is located in the target image, a regional selection box for each question is marked on the target image.

3. The method for identifying a topic area according to claim 1, wherein: The method of using the large language model to generate the text content of each question based on the text content of each text line and / or the target image includes: A multimodal large language model is used to generate a topic text list based on the text content of each text line and in combination with the target image, wherein the topic text list includes the number and text content of each topic.

4. The method for identifying a topic area according to claim 1, wherein: The step of obtaining a text line belonging to each topic by matching the text content of each text line with the text content of each topic includes: For each question, determine the similarity between the text content of each text line and the text content of the question, and obtain the text similarity between each text line and the question; A text line whose text similarity with the topic is greater than a preset similarity threshold is determined as a text line belonging to the topic.

5. The method for identifying a topic area according to claim 1, wherein: The determining, based on the position information of the text line belonging to each question, the area where each question is located in the target image includes: For each question, the minimum enclosing rectangle of all text lines belonging to the question is determined based on the position information of the text lines belonging to the question, and the area where the question is located is obtained.

6. The method for identifying a topic area according to claim 5, wherein: The position information of any text line in the target image is represented by the coordinates of the upper left corner and the lower right corner of the text line; The step of determining the minimum enclosing rectangle of all text lines belonging to the topic based on the position information of the text lines belonging to the topic includes: Obtain the minimum horizontal coordinate, minimum vertical coordinate, maximum horizontal coordinate and maximum vertical coordinate from the upper left corner coordinates and lower right corner coordinates of several text lines belonging to the question; A rectangle with (minimum horizontal coordinate, minimum vertical coordinate) as the upper left vertex and (maximum horizontal coordinate, maximum vertical coordinate) as the lower right vertex is determined as the minimum enclosing rectangle of all text lines belonging to the question.

7. A question area recognition device, characterized in that: include: Image acquisition module, text recognition module, title text generation module, text matching module and title area determination module; The image acquisition module is used to acquire target images containing several topics; The text recognition module is used to perform text recognition on the target image to obtain the text content and position information of each text line in the target image; The topic text generation module is used to generate the text content of each topic based on the text content of each text line and / or the target image using a large language model; The text matching module is used to obtain a text line belonging to each question by matching the text content of each text line with the text content of each question; The topic area determination module is configured to determine the area where each topic is located in the target image based on the position information of the text line belonging to each topic.

8. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so that the electronic device can implement the steps of the method for identifying a topic area according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the question area identification method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the method for identifying a topic area according to any one of claims 1 to 6.

Images