Model training method and electronic equipment

Abstract

The present application discloses a model training method and electronic device, the method comprising: obtaining a sample document image and a first paragraph region corresponding to the sample document image; indenting the first paragraph region based on a preset indentation ratio to obtain a second paragraph region; determining a target paragraph segmentation map of the sample document image based on the second paragraph region; invoking an initial image segmentation model to process the sample document image to obtain a predicted paragraph segmentation map of the sample document image; and updating model parameters of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain an image segmentation model. Based on the method described in the present application, the situation of paragraph region adhesion can be reduced, and the overall performance of layout analysis can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular to model training methods and electronic devices. Background Art

[0002] Optical Character Recognition (OCR) is the process of analyzing and identifying text images to obtain text and layout information. To ensure that paragraph information and layout information are not lost, layout analysis can be used during OCR to extract text from images, allowing the recognized text to be displayed according to its layout. Layout analysis, in this context, refers to the process of segmenting document images into different paragraph regions (such as text lines, titles, tables, and images). For example, image segmentation models can be used to perform layout segmentation on document images.

[0003] However, for images taken by a camera, due to problems such as shooting angle or shooting light, the regularity of the image will be not high, so when the image segmentation model is used for processing, the segment area will be stuck (such as Figure 1 As shown in the figure, paragraph area 1 and paragraph area 2 are stuck together, which greatly reduces the overall performance of layout analysis. Therefore, how to reduce the situation of paragraph area sticking is a problem to be solved. Summary of the Invention

[0004] This application provides a model training method and electronic device that can reduce the adhesion of paragraph areas and improve the overall performance of layout analysis.

[0005] The first aspect provides a model training method, which includes: obtaining a sample document image and a first paragraph area corresponding to the sample document image; indenting the first paragraph area based on a preset indentation ratio to obtain a second paragraph area; determining a target paragraph segmentation map of the sample document image based on the second paragraph area; calling an initial image segmentation model to process the sample document image to obtain a predicted paragraph segmentation map of the sample document image; updating the model parameters of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain an image segmentation model.

[0006] In the method described in the first aspect, after obtaining the sample document image and the paragraph area corresponding to the sample document image, the marked paragraph area is readjusted by indenting, and the image segmentation model is trained using the sample document image and the adjusted paragraph area. This can improve the accuracy of the model in layout segmentation, reduce the adhesion of paragraph areas, and help improve the overall performance of layout analysis.

[0007] In one possible implementation, indenting the first paragraph region based on a preset indent ratio to obtain a second paragraph region includes: obtaining first coordinate information of the first paragraph region; determining the area and perimeter of the first paragraph region based on the first coordinate information; determining a preliminary indent distance based on the area, perimeter, and preset indent ratio of the first paragraph region; and indenting the first paragraph region based on the preliminary indent distance to obtain the second paragraph region. Based on this method, it is possible to separate the adjoining paragraph regions as much as possible, thereby improving the accuracy of layout segmentation.

[0008] In one possible implementation, indenting the first paragraph region based on the preliminary indent distance to obtain a second paragraph region includes: if the preliminary indent distance is less than a first value, indenting the first paragraph region based on the preliminary indent distance to obtain a second paragraph region. This approach ensures that single-line paragraphs are not over-indented, avoids situations where the text within the paragraph region is not fully enclosed, and improves the effectiveness of the indentation process.

[0009] In one possible implementation, the method further includes: if the preliminary indent distance is greater than or equal to a second value and the preliminary indent distance is less than or equal to a third value, indenting the first paragraph region based on the preliminary indent distance to obtain a second paragraph region; the first value is less than the second value. Based on this method, it is possible to ensure that the indent distance of the paragraph region does not exceed the height of a single line of text and also ensure that the indent distance of the paragraph region is reasonable.

[0010] In one possible implementation, the method further includes: if the preliminary indent distance is greater than a third value and less than or equal to a fourth value, adjusting the preliminary indent distance to a first indent distance, and indenting the first paragraph region based on the first indent distance to obtain a second paragraph region; the first indent distance is less than the preliminary indent distance, the first indent distance is greater than or equal to the third value, and the first indent distance is less than a preset single-line text height. Based on this method, excessive indentation of paragraph regions can be avoided, which is conducive to improving the effectiveness of indentation processing.

[0011] In one possible implementation, the method further includes: if the initial indent distance is greater than or equal to a first value and less than a second value, adjusting the initial indent distance to a second indent distance, and indenting the first paragraph region based on the second indent distance to obtain a second paragraph region; the second indent distance is greater than the initial indent distance, less than or equal to the second value, and less than a preset single-line text height. Based on this approach, it is possible to avoid situations where the indent distance of a paragraph region is not maximized, thereby improving the effectiveness and rationality of indentation processing.

[0012] In one possible implementation, the method further includes: if the preliminary indent distance is greater than the fourth value, adjusting the preliminary indent distance to a third indent distance, and indenting the first paragraph region based on the third indent distance to obtain a second paragraph region; the third indent distance is less than the preliminary indent distance, the second indent distance is greater than or equal to the second value, and the third indent distance is less than the preset single-line text height. Based on this method, while ensuring that the indent distance does not exceed the single-line text height, the indentation processing method for large paragraphs is simplified, ensuring the effectiveness and efficiency of the indentation processing.

[0013] A second aspect provides an electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code. The computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the model training method in any possible implementation of the first aspect.

[0014] The third aspect provides a model training device, which can be an electronic device, a device in an electronic device, or a device that can be used in combination with an electronic device. The model training device can also be a chip system. The model training device can execute the method described in the first aspect. The functions of the model training device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more units or modules corresponding to the above functions. The unit or module can be software and / or hardware. The operations and beneficial effects performed by the model training device can refer to the methods and beneficial effects described in the first aspect above, and the repeated parts will not be repeated.

[0015] The fourth aspect provides a computer storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the model training method in any possible implementation of the first aspect.

[0016] The fifth aspect provides a computer program product, which, when running on a computer, enables the computer to execute the model training method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a document image and a marked paragraph area provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0019] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of another document image and a marked paragraph area provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of a preliminary retraction distance provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of a target paragraph segmentation map (mask) of a sample document image provided in an embodiment of the present application;

[0024] Figure 8 A flowchart of another model training method provided in an embodiment of the present application;

[0025] Figure 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0027] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0028] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0029] In order to reduce the adhesion of paragraph areas and improve the overall performance of layout analysis, the present application provides a model training method. In a specific implementation, the above-mentioned model training method can be performed by an electronic device 100. The electronic device 100 can be a terminal device or a server. For example, the terminal device can be a laptop, a tablet computer, a desktop computer, etc., but is not limited to this; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server, etc., but is not limited to this.

[0030] The hardware structure of the electronic device 100 is introduced below. Figure 2 , Figure 2 Schematic diagram of the hardware structure of the electronic device 100 provided in an embodiment of the present application.

[0031] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0032] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0033] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0034] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0035] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or is cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. The processor 110 calls the instructions or data stored in the memory, causing the electronic device 100 to execute the model training method executed by the electronic device in the following method embodiment.

[0036] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0037] The charging management module 140 is configured to receive charging input from a charger, which may be a wireless charger or a wired charger.

[0038] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. In some other embodiments, the power management module 141 can also be set in the processor 110.

[0039] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0040] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0041] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0042] The modem processor includes a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a medium- or high-frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is passed to the application processor.

[0043] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), BLE broadcasting, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0044] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0045] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0046] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0047] Electronic device 100 can implement a camera function using an ISP, camera 193, a video codec, a GPU, a display 194, and an application processor. The ISP processes data fed back by camera 193. Camera 193 is used to capture still images or video. The digital signal processor processes digital signals, and can process not only digital image signals but also other digital signals. The video codec compresses or decompresses digital video. Electronic device 100 may support one or more video codecs.

[0048] NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn.

[0049] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0050] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function), etc. The data storage area can store data (such as audio data) created during the use of the electronic device 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as a flash memory device, etc.

[0051] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0052] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0053] Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Receiver 170B, also known as an "earpiece," is used to convert audio electrical signals into sound signals. Microphone 170C, also known as a "microphone" or "microphone," is used to convert sound signals into electrical signals. Headphone jack 170D is used to connect wired headphones. Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A may be provided on display screen 194. Gyroscope sensor 180B may be used to determine the motion posture of electronic device 100. Air pressure sensor 180C is used to measure air pressure. Magnetic sensor 180D includes a Hall sensor. Acceleration sensor 180E may detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes). Distance sensor 180F is used to measure distance. Proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector. Ambient light sensor 180L is used to sense ambient light brightness. Fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect the temperature. The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The bone conduction sensor 180M can obtain vibration signals. The buttons 190 include a power button, a volume button, etc. The motor 191 can generate vibration prompts. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0054] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100. Figure 3 This is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application. A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0055] The application layer can include a series of application packages. Figure 3 As shown, the application layer may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0056] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0057] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0058] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0059] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0060] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0061] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0062] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0063] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0064] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0065] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0066] The system library can include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0067] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0068] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats.

[0069] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0070] A 2D graphics engine is a drawing engine for 2D drawings.

[0071] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0072] The following is a detailed description of the model training method provided by this application. Figure 4 , Figure 4 This is a flow chart of a model training method provided in an embodiment of the present application. Figure 4 As shown, the model training method includes the following steps S401 to S405: Figure 4 The method execution subject shown includes an electronic device (such as Figure 2 electronic device 100 in FIG. 1 ). Alternatively, Figure 4 The method execution subject shown includes a chip, chip system, or processor in an electronic device. Or, Figure 4 The execution subject of the method shown may also be a logic module or software that can realize all or part of the functions of the electronic device. The embodiment of the present application does not limit the execution subject of the model training method. Figure 4 Take electronic equipment as an example to illustrate the execution subject.

[0073] S401: The electronic device obtains a sample document image and a first paragraph area corresponding to the sample document image.

[0074] In the embodiment of the present application, the sample document image refers to an image of a paper type or a book type, such as a newspaper photo taken by a camera. Before training the model, the electronic device needs to obtain one or more sample document images, and each sample document image is marked with one or more first paragraph areas (such as Figure 1 As shown). The one or more first paragraph regions marked in each sample document image may be marked using a traditional image segmentation model (such as a semantic segmentation U-NET model, a semantic segmentation SegNet model, etc.), or may be marked using other methods, which are not limited here.

[0075] In one possible implementation, the longest side of the sample document image is a preset length. That is, after acquiring the sample document image, the electronic device uniformly adjusts the longest side to the preset length, ensuring the uniformity of the sample document images and improving the accuracy of the model. For example, the longest side of the sample document image can be adjusted to 960 pixels.

[0076] S402, the electronic device indents the first paragraph area based on a preset indentation ratio to obtain a second paragraph area. In an embodiment of the present application, since there may be adhesion between one or more first paragraph areas marked in the sample document image, if the model is trained using the adhered paragraph areas, the adhered paragraph areas will be regarded as one paragraph area, which will greatly reduce the accuracy of layout segmentation and the performance of layout analysis. Therefore, the electronic device needs to indent each first paragraph area so that the adhered first paragraph areas are separated. For example, the electronic device Figure 1 Each first paragraph area in is indented according to the preset indent ratio, and the result is as follows Figure 5 The multiple second paragraph areas (such as paragraph area a, paragraph area b, etc.) shown in .

[0077] In one possible implementation, when the electronic device indents the first paragraph area based on a preset indentation ratio to obtain the second paragraph area, the specific implementation method may be: obtaining the first coordinate information of the first paragraph area; determining the area of ​​the first paragraph area and the perimeter of the first paragraph area based on the first coordinate information; determining a preliminary indentation distance based on the area of ​​the first paragraph area, the perimeter of the first paragraph area and the preset indentation ratio; and indenting the first paragraph area based on the preliminary indentation distance to obtain the second paragraph area. That is, if Figure 6As shown, the initial indent distance can be understood as the distance between the boundary of the first paragraph area (i.e., paragraph area 1) and the boundary of the second paragraph area (i.e., paragraph area a). The first paragraph area is indented toward the center of the area by this initial indent distance to form the second paragraph area. The first coordinate information of the first paragraph area refers to the coordinates of each vertex of the first paragraph area; the preset indent ratio can be fixed or dynamically changed, which is not limited here.

[0078] Specifically, the electronic device can calculate the initial retraction distance using the following formula (1):

[0079]

[0080] Wherein, S is the area of ​​the first paragraph region, C is the perimeter of the first paragraph region, m is the preset indentation ratio (eg, 5%), and L0 is the initial indentation distance.

[0081] Optionally, when the electronic device indents the first paragraph area based on the preliminary indentation distance to obtain the second paragraph area, the specific implementation method may include the following: Figure 8 For details of steps S805 to S809, please refer to Figure 8 The description of steps S805 to S809 is omitted here. Of course, when the electronic device indents the first paragraph area based on the preliminary indent distance to obtain the second paragraph area, the specific implementation method may also include the following: Figure 8 One or more of steps S805 to S809, Figure 8 Taking the specific implementation method of the electronic device indenting the first paragraph area based on the preliminary indentation distance to obtain the second paragraph area as an example, the implementation method includes steps S805 to S809.

[0082] S403: The electronic device determines a target paragraph segmentation graph of the sample document image based on the second paragraph region.

[0083] In the embodiment of the present application, the target paragraph segmentation map of the sample document image can be a segmentation map represented by a mask. The target paragraph segmentation map here can be considered as the result of the optimal layout segmentation of the sample document image. Figure 7 As shown, the pixel points of the second paragraph area in the mask are set to the first label (i.e., 1, displayed in white), and the pixel points other than the second paragraph area in the mask are set to the second label (i.e., 0, displayed in black). Through this mask, the second paragraph area in the sample document image can be distinguished from other areas, which facilitates the observation of the paragraph structure in the sample document image.

[0084] S404: The electronic device calls the initial image segmentation model to process the sample document image to obtain a predicted paragraph segmentation map of the sample document image.

[0085] In the embodiments of the present application, the initial image segmentation model can be a semantic segmentation model, such as a U-NET model, a SegNet model, a DeepLab model, etc., without limitation herein. The electronic device inputs the sample document into the initial image segmentation model for processing, and then outputs a predicted paragraph segmentation map of the sample document image.

[0086] S405: The electronic device updates the model parameters of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain an image segmentation model.

[0087] In an embodiment of the present application, the electronic device continuously optimizes the initial image segmentation model so that the predicted paragraph segmentation map is as close as possible to the target paragraph segmentation map; when the gap between the target paragraph segmentation map and the predicted paragraph segmentation map reaches a preset threshold, the training of the initial image segmentation model can be terminated, and the image segmentation model is obtained.

[0088] In one possible implementation, the electronic device updates the model parameters of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain the image segmentation model. The specific implementation method may be: determining the loss value of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map; and updating the model parameters of the initial image segmentation model based on the loss value to obtain the image segmentation model.

[0089] The specific implementation method for the electronic device to determine the loss value of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map can be calculated using a loss function, such as a cross entropy function, a mean square error loss function, etc., which is not limited here. In addition, the method for updating the model parameters of the initial image segmentation model can adopt a stochastic gradient descent method, an adaptive gradient algorithm, etc., which is not limited here.

[0090] Optionally, the method further includes: obtaining a first document image to be processed; and invoking the image segmentation model to process the first document image to obtain a first paragraph segmentation map for the first document image, wherein the first paragraph segmentation map includes one or more third paragraph regions. In other words, after the electronic device has trained the image segmentation model, it can apply the image segmentation model by simply inputting the document image to the image segmentation model to output a paragraph segmentation map for the document image. In this case, the first paragraph segmentation map includes one or more third paragraph regions without any overlap between the paragraph regions, significantly improving the overall performance of the layout analysis.

[0091] exist Figure 4 In the described method, after obtaining the sample document image and the paragraph area corresponding to the sample document image, the marked paragraph area is readjusted by indenting, and the image segmentation model is trained using the sample document image and the indented paragraph area. This can improve the accuracy of the model in layout segmentation, reduce the adhesion of paragraph areas, and help improve the overall performance of layout analysis.

[0092] See Figure 8 , Figure 8 This is a flow chart of another model training method provided in the embodiment of the present application. Figure 8 As shown, the model training method includes the following steps S801 to S812: Figure 8 The method execution subject shown includes an electronic device (such as Figure 2 electronic device 100 in FIG. 1 ). Alternatively, Figure 8 The method execution subject shown includes a chip in an electronic device. Figure 8 Take electronic equipment as an example to illustrate the execution subject.

[0093] S801: The electronic device obtains a sample document image and a first paragraph area corresponding to the sample document image.

[0094] S802: The electronic device obtains first coordinate information of the first paragraph area.

[0095] S803: The electronic device determines the area and perimeter of the first paragraph region based on the first coordinate information.

[0096] S804: The electronic device determines a preliminary indentation distance based on the area of ​​the first paragraph area, the perimeter of the first paragraph area, and a preset indentation ratio.

[0097] The specific implementation of step S801 may refer to the specific implementation of step S401 above, and the specific implementation of steps S802 to S804 may refer to a possible implementation of step S402 above, which will not be described in detail here.

[0098] S805: If the preliminary indentation distance is less than the first value, the electronic device indents the first paragraph area based on the preliminary indentation distance to obtain a second paragraph area.

[0099] Based on this method, it can be ensured that a single-line paragraph will not be over-indented, avoiding the situation where the text in the paragraph area cannot be completely included, and improving the effectiveness of the indentation processing.

[0100] For example, assuming the first value is 0.5 pixels and the height of a single line of text is 10 pixels, when the electronic device calculates a preliminary indentation distance of 0.3 pixels (i.e., less than 0.5 pixels), the first paragraph area can be considered a single-line paragraph. Further indentation may result in the text in the first paragraph area not being fully enclosed. Therefore, the electronic device can directly indent the first paragraph area by 0.3 pixels (i.e., the preliminary indentation distance) to obtain the second paragraph area.

[0101] S806: If the preliminary indentation distance is greater than or equal to the second value and the preliminary indentation distance is less than or equal to the third value, the electronic device indents the first paragraph area based on the preliminary indentation distance to obtain a second paragraph area.

[0102] The first value is smaller than the second value. Based on this approach, it is possible to ensure that the indentation distance of the paragraph area does not exceed the height of a single line of text, and also to ensure the rationality and effectiveness of the indentation distance of the text paragraph.

[0103] For example, assuming the first value is 0.5 pixels, the second value is 3 pixels, the third value is 6 pixels, and the height of a single line of text is 10 pixels. When the electronic device calculates a preliminary indentation distance of 5 pixels (i.e., greater than 3 pixels and less than 6 pixels), it can be considered that the first paragraph area is a small paragraph, and the preliminary indentation distance is between 3 pixels and 6 pixels, which does not exceed the height of a single line of text and is a relatively reasonable indentation distance. Therefore, the electronic device can directly indent the first paragraph area by 5 pixels (i.e., the preliminary indentation distance) to obtain the second paragraph area.

[0104] S807: If the preliminary indentation distance is greater than the third value and is less than or equal to the fourth value, the electronic device adjusts the preliminary indentation distance to the first indentation distance, and indents the first paragraph area based on the first indentation distance to obtain a second paragraph area.

[0105] The first indent distance is smaller than the initial indent distance, is greater than or equal to the third value, and is smaller than the preset single-line text height. This approach can avoid excessive indentation of the paragraph area, thereby improving the effectiveness of indentation processing.

[0106] For example, assuming the third value is 6 pixels, the fourth value is 15 pixels, the preset single-line text height is 10 pixels, and the first indent distance is 6 pixels. When the preliminary indent distance calculated by the electronic device is 8 pixels (i.e., greater than 6 pixels and less than 15 pixels), it can be considered that the first paragraph area is a medium-sized paragraph. At this time, the preliminary indent distance is between 6 pixels and 15 pixels. If the preliminary indent distance is directly used to indent the first paragraph area, there may be a situation where the indent is excessive and exceeds the single-line text height, resulting in the text in the first paragraph area not being fully included. Therefore, the electronic device can adjust the preliminary indent distance to 6 pixels (i.e., the first indent distance), and indent the first paragraph area by 6 pixels (i.e., the first indent distance) to obtain the second paragraph area.

[0107] S808. If the preliminary indentation distance is greater than or equal to the first value and is less than the second value, the electronic device adjusts the preliminary indentation distance to a second indentation distance, and indents the first paragraph area based on the second indentation distance to obtain a second paragraph area.

[0108] The second indent distance is greater than the initial indent distance, is less than or equal to the second value, and is less than the preset single-line text height. Based on this approach, it is possible to avoid the situation where the indent distance of the paragraph area is not maximized, thereby improving the effectiveness and rationality of the indentation process.

[0109] For example, assuming the first value is 0.5 pixels, the second value is 3 pixels, the preset single-line text height is 10 pixels, and the second indent distance is 3 pixels. When the preliminary indent distance calculated by the electronic device is 2 pixels (i.e., greater than 0.5 pixels and less than 3 pixels), it can be considered that the first paragraph area is a small paragraph. At this time, the preliminary indent distance is between 0.5 pixels and 3 pixels. If the preliminary indent distance is directly used to indent the first paragraph area, the indent distance may not be maximized. While ensuring that the indent distance does not exceed the preset single-line text height, it can be further indented. Therefore, the electronic device can adjust the preliminary indent distance to 3 pixels (i.e., the second indent distance), and indent the first paragraph area by 3 pixels (i.e., the second indent distance) to obtain the second paragraph area.

[0110] S809: If the preliminary indentation distance is greater than the fourth value, the electronic device adjusts the preliminary indentation distance to a third indentation distance, and indents the first paragraph area based on the third indentation distance to obtain a second paragraph area.

[0111] The third indent distance is smaller than the initial indent distance, the second indent distance is greater than or equal to the second value, and the third indent distance is smaller than the preset single-line text height. Based on this approach, while ensuring that the indent distance does not exceed the single-line text height, the indentation processing method for large paragraphs is simplified, ensuring the effectiveness and efficiency of the indentation processing.

[0112] For example, assuming the second value is 3 pixels, the fourth value is 15 pixels, the preset single-line text height is 10 pixels, and the second indent distance is 3 pixels. When the preliminary indent distance calculated by the electronic device is 18 pixels (i.e., greater than 15 pixels), it can be considered that the first paragraph area is a large paragraph. At this time, the first paragraph area is indented using the preliminary indent distance, or the first paragraph area is indented using a fixed third indent distance. The final indented second paragraph area is not much different. In order to simplify the indentation processing method, while ensuring that the indent distance does not exceed the single-line text height, the electronic device can uniformly adjust the preliminary indent distance to 3 pixels (i.e., the third indent distance), and indent the first paragraph area by 3 pixels (i.e., the third indent distance) to obtain the second paragraph area.

[0113] In general, electronic devices need to determine the size of a paragraph based on the initial indent distance, and then use different methods to adjust or retain the initial indent distance, allowing single-line paragraphs and small and medium-sized paragraphs to be indented more, while at the same time not exceeding the height of a single line of text. As long as the paragraph area does not completely lose the entire line area, the matching result can be obtained based on the text line information. Therefore, using different strategies for different paragraph sizes to effectively indent the area helps to reduce the problem of paragraph area adhesion.

[0114] It should be noted that steps S805 to S809 are a specific implementation method for the electronic device in step S402 above to indent the first paragraph area based on the preliminary indent distance to obtain the second paragraph area. Of course, when the electronic device indents the first paragraph area based on the preliminary indent distance to obtain the second paragraph area, the specific implementation method may also include one or more steps in steps S805 to S809. The embodiment of the present application is described by taking the example of the electronic device indenting the first paragraph area based on the preliminary indent distance to obtain the second paragraph area, and the specific implementation method includes all the steps in steps S805 to S809.

[0115] S810: The electronic device determines a target paragraph segmentation graph of the sample document image based on the second paragraph region.

[0116] S811: The electronic device calls the initial image segmentation model to process the sample document image to obtain a predicted paragraph segmentation map of the sample document image.

[0117] S812: The electronic device updates the model parameters of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain an image segmentation model.

[0118] The specific implementation of steps S810 to S812 may refer to the specific implementation of steps S403 to S405 above, and will not be described in detail here.

[0119] exist Figure 8 In the described method, after obtaining a sample document image and the paragraph area corresponding to the sample document image, the size of the paragraph is calculated and determined based on the preliminary indentation distance. Different strategies are used to effectively indent paragraph areas of different sizes. The image segmentation model is trained using the sample document image and the indented paragraph area, which can improve the accuracy of the model in layout segmentation, reduce the adhesion of paragraph areas, and help improve the overall performance of layout analysis.

[0120] See Figure 9 , Figure 9 A structural diagram of a model training device according to an embodiment of the present application is shown. Figure 9 The device shown can be used to perform the above Figure 4 The method embodiment described herein partially or completely implements the functions of the electronic device, wherein the device may also be a chip system. Figure 9 The model training device 900 shown may include an acquisition unit 901, a processing unit 902, and an update unit 903.

[0121] An acquiring unit 901 is configured to acquire a sample document image and a first paragraph region corresponding to the sample document image;

[0122] The processing unit 902 is configured to indent the first paragraph region based on a preset indent ratio to obtain a second paragraph region;

[0123] The processing unit 902 is further configured to determine a target paragraph segmentation map of the sample document image based on the second paragraph region;

[0124] The processing unit 902 is further configured to call the initial image segmentation model to process the sample document image to obtain a predicted paragraph segmentation map of the sample document image;

[0125] The updating unit 903 is configured to update the model parameters of the initial image segmentation model based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain an image segmentation model.

[0126] In one possible implementation, the processing unit 902, when indenting the first paragraph area based on a preset indentation ratio to obtain the second paragraph area, can be specifically used to: obtain first coordinate information of the first paragraph area; determine the area of ​​the first paragraph area and the perimeter of the first paragraph area based on the first coordinate information; determine a preliminary indentation distance based on the area of ​​the first paragraph area, the perimeter of the first paragraph area and the preset indentation ratio; and indent the first paragraph area based on the preliminary indentation distance to obtain the second paragraph area.

[0127] In one possible implementation, the processing unit 902, when indenting the first paragraph area based on the preliminary indentation distance to obtain the second paragraph area, can be specifically used to: if the preliminary indentation distance is less than the first value, indenting the first paragraph area based on the preliminary indentation distance to obtain the second paragraph area.

[0128] In one possible implementation, the processing unit 902 is further used to: if the preliminary indentation distance is greater than or equal to the second value and the preliminary indentation distance is less than or equal to the third value, then indent the first paragraph area based on the preliminary indentation distance to obtain a second paragraph area; the first value is less than the second value.

[0129] In one possible implementation, the processing unit 902 is also used to: if the preliminary indentation distance is greater than the third value and the preliminary indentation distance is less than or equal to the fourth value, adjust the preliminary indentation distance to the first indentation distance, and indent the first paragraph area based on the first indentation distance to obtain the second paragraph area; the first indentation distance is less than the preliminary indentation distance, and the first indentation distance is greater than or equal to the third value, and the first indentation distance is less than the preset single-line text height.

[0130] In one possible implementation, the processing unit 902 is also used to: if the preliminary indentation distance is greater than or equal to the first value and the preliminary indentation distance is less than the second value, adjust the preliminary indentation distance to the second indentation distance, and indent the first paragraph area based on the second indentation distance to obtain the second paragraph area; the second indentation distance is greater than the preliminary indentation distance, and the second indentation distance is less than or equal to the second value, and the second indentation distance is less than the preset single-line text height.

[0131] In one possible implementation, the processing unit 902 is also used to: if the preliminary indentation distance is greater than the fourth value, adjust the preliminary indentation distance to a third indentation distance, and indent the first paragraph area based on the third indentation distance to obtain a second paragraph area; the third indentation distance is smaller than the preliminary indentation distance, and the second indentation distance is greater than or equal to the second value, and the third indentation distance is smaller than the preset single-line text height.

[0132] The embodiments of the present invention and Figure 4 The method embodiments shown are based on the same concept and have the same technical effects. For specific principles, please refer to Figure 4 The description of the illustrated embodiment is omitted here.

[0133] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0134] The present application also provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

[0135] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0136] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0137] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample document image and a first paragraph area corresponding to the sample document image; Obtaining first coordinate information of the first paragraph area; determining the area of ​​the first paragraph region and the perimeter of the first paragraph region based on the first coordinate information; determining a preliminary indentation distance based on the area of ​​the first paragraph region, the perimeter of the first paragraph region, and a preset indentation ratio; If the preliminary indentation distance is less than a first value, indenting the first paragraph area based on the preliminary indentation distance to obtain a second paragraph area; If the preliminary indentation distance is greater than or equal to a second value and the preliminary indentation distance is less than or equal to a third value, indenting the first paragraph area based on the preliminary indentation distance to obtain a second paragraph area; the first value is less than the second value; If the preliminary indent distance is greater than the third value and the preliminary indent distance is less than or equal to the fourth value, the preliminary indent distance is adjusted to the first indent distance, and the first paragraph area is indented based on the first indent distance to obtain a second paragraph area; the first indent distance is less than the preliminary indent distance, the first indent distance is greater than or equal to the third value, and the first indent distance is less than a preset single-line text height; determining a target paragraph segmentation map of the sample document image based on the second paragraph region; Invoking an initial image segmentation model to process the sample document image to obtain a predicted paragraph segmentation map of the sample document image; The model parameters of the initial image segmentation model are updated based on the target paragraph segmentation map and the predicted paragraph segmentation map to obtain an image segmentation model.

2. The method according to claim 1, characterized in that The method further comprises: If the preliminary indent distance is greater than or equal to the first value and the preliminary indent distance is less than the second value, the preliminary indent distance is adjusted to the second indent distance, and the first paragraph area is indented based on the second indent distance to obtain a second paragraph area; the second indent distance is greater than the preliminary indent distance, and the second indent distance is less than or equal to the second value, and the second indent distance is less than the preset single-line text height.

3. The method according to claim 2, characterized in that The method further comprises: If the preliminary indent distance is greater than the fourth value, the preliminary indent distance is adjusted to the third indent distance, and the first paragraph area is indented based on the third indent distance to obtain a second paragraph area; the third indent distance is less than the preliminary indent distance, and the second indent distance is greater than or equal to the second value, and the third indent distance is less than the preset single-line text height.

4. An electronic device, characterized in that: include: One or more processors, one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device performs the method as described in any one of claims 1-3.

5. A computer storage medium, characterized in that The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 3.

6. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 3.

Images