Image processing method, device, electronic device, and computer-readable storage medium

Abstract

The present application relates to an image processing method, apparatus, electronic device, storage medium, and computer program product. The method comprises: determining a plurality of character regions in an image to be processed; performing preprocessing of the character regions in a character recognition process using a first processing thread to obtain a preprocessing result; performing inference processing in a character recognition process using a second processing thread on the preprocessing result to obtain an inference result; and performing post-processing in a character recognition process using a third processing thread on the inference result to obtain a character recognition result for the image to be processed; wherein the second processing thread is executed in parallel with at least one of the first and third processing threads. This method can improve image processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology, computer vision has been widely applied in various fields, including retail, manufacturing, healthcare, autonomous driving, and agriculture. Optical character recognition (OCR) is a key branch of computer vision technology. OCR involves analyzing and processing image files of textual materials to obtain text and layout information. Specifically, OCR identifies text within an image and produces text recognition results. However, the efficiency of the current OCR process is relatively low. Summary of the Invention

[0003] Embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the efficiency of image processing.

[0004] An image processing method, comprising:

[0005] Determine a number of character regions in the image to be processed;

[0006] Performing preprocessing in character recognition processing on the character area through the first processing thread to obtain a preprocessing result;

[0007] Performing inference processing in the character recognition processing on the preprocessing result through the second processing thread to obtain an inference result;

[0008] Performing post-processing of the character recognition process on the inference result through the third processing thread to obtain a character recognition result of the image to be processed;

[0009] The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

[0010] An image processing device, comprising:

[0011] A character region determination module, used for determining a number of character regions in the image to be processed;

[0012] A preprocessing module, configured to perform preprocessing in character recognition processing on the character area through a first processing thread to obtain a preprocessing result;

[0013] An inference module, configured to perform inference processing in the character recognition processing on the preprocessing result through a second processing thread to obtain an inference result;

[0014] A post-processing module, configured to perform post-processing of the character recognition process on the inference result through a third processing thread to obtain a character recognition result of the image to be processed;

[0015] The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

[0016] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Determine a number of character regions in the image to be processed;

[0018] Performing preprocessing in character recognition processing on the character area through the first processing thread to obtain a preprocessing result;

[0019] Performing inference processing in the character recognition processing on the preprocessing result through the second processing thread to obtain an inference result;

[0020] Performing post-processing of the character recognition process on the inference result through the third processing thread to obtain a character recognition result of the image to be processed;

[0021] The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0023] Determine a number of character regions in the image to be processed;

[0024] Performing preprocessing in character recognition processing on the character area through the first processing thread to obtain a preprocessing result;

[0025] Performing inference processing in the character recognition processing on the preprocessing result through the second processing thread to obtain an inference result;

[0026] Performing post-processing of the character recognition process on the inference result through the third processing thread to obtain a character recognition result of the image to be processed;

[0027] The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

[0028] A computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0029] Determine a number of character regions in the image to be processed;

[0030] Performing preprocessing in character recognition processing on the character area through the first processing thread to obtain a preprocessing result;

[0031] Performing inference processing in the character recognition processing on the preprocessing result through the second processing thread to obtain an inference result;

[0032] Performing post-processing of the character recognition process on the inference result through the third processing thread to obtain a character recognition result of the image to be processed;

[0033] The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

[0034] The above-mentioned image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, for several character regions in the image to be processed, performs preprocessing in character recognition processing on the character regions through a first processing thread, performs inference processing on the preprocessing results through a second processing thread, and performs post-processing on the inference results through a third processing thread to obtain the character recognition results of the image to be processed, and the second processing thread is parallel to at least one of the first processing thread and the third processing thread. During the image processing process, the preprocessing, inference processing, and post-processing in the character recognition process are respectively executed by different processing threads, and the thread performing the inference processing is parallel to at least one of the threads performing the preprocessing and the threads performing the post-processing, thereby realizing parallel processing in the character recognition process, which is beneficial to improving image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 A diagram showing an application environment of an image processing method in one embodiment;

[0037] Figure 2 is a flowchart of an image processing method in one embodiment;

[0038] Figure 3 is a flowchart of an image processing method in another embodiment;

[0039] Figure 4 A schematic diagram of the time consumption of character recognition processing in one embodiment;

[0040] Figure 5 A schematic diagram of sequential execution time consumption in one embodiment;

[0041] Figure 6 A schematic diagram of pipeline execution time consumption in one embodiment;

[0042] Figure 7 A schematic diagram of the average time consumption for character recognition processing in one embodiment;

[0043] Figure 8 A schematic diagram of average time consumption for character recognition processing in another embodiment;

[0044] Figure 9 is a structural block diagram of an image processing device in one embodiment;

[0045] Figure 10 is a diagram of the internal structure of an electronic device in one embodiment;

[0046] Figure 11 FIG. 4 is a diagram showing the internal structure of an electronic device in another embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can send the image to be processed to the server 104 through the network. After receiving the image to be processed, the server 104 can determine several character areas in the image to be processed. The server 104 performs pre-processing in the character recognition process on the several character areas in the image to be processed through a first processing thread, performs inference processing on the pre-processing results through a second processing thread, and performs post-processing on the inference results through a third processing thread to obtain the character recognition result of the image to be processed. The second processing thread is parallel to at least one of the first processing thread and the third processing thread. The server 104 can also return the character recognition result of the image to be processed to the terminal 102. In addition, the image processing method can also be implemented separately by the server 104 or the terminal 102, that is, the image processing method is implemented by the server 104 alone, or the image processing method is implemented by the terminal 102 alone.

[0049] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0050] In one embodiment, Figure 2 As shown, an image processing method is provided, which is applied to Figure 1 The server or terminal in the example is used as an example to illustrate the process, including the following steps:

[0051] Step 202: Determine a number of character regions in the image to be processed.

[0052] The image to be processed is an image that needs to be processed, specifically an image that needs to be processed for character recognition. The image to be processed can be captured by a terminal, and there is text in the image to be processed. By performing character recognition on the image to be processed, the text can be identified from the image to be processed. The character area is the area in the image to be processed where characters are detected. In specific applications, the character area can be an circumscribed rectangular frame that represents the characters in the image. There can be multiple character areas in the image to be processed, so character recognition can be performed based on the character areas to identify text from the image to be processed.

[0053] Specifically, the server can determine several character regions in the image to be processed. In a specific implementation, the server can obtain the image to be processed and perform character detection on the image to be processed, thereby detecting several character regions from the image to be processed and locating the characters in the image to be processed. Specifically, the characters can be marked in the form of an external rectangular frame. In addition, the server can also directly obtain the character regions in the image to be processed. The processing of performing character region detection on the image to be processed can be implemented in advance. The server directly obtains several character regions in the image to be processed to perform character recognition processing.

[0054] Step 204 : Preprocessing the character recognition process on the character area through the first processing thread to obtain a preprocessing result.

[0055] Among them, a thread is the smallest unit that the operating system can perform computational scheduling. The first processing thread is used to perform preprocessing in the character recognition process. The character recognition process can be an OCR process, which aims to automatically extract and recognize text in a target image or video through computer vision technology. It is one of the most widely used technologies in the field of computer vision. The specific processing of the preprocessing can be determined according to the algorithm adopted by the character recognition process. Specifically, it can include various processes such as cropping the text area from the original image according to the coordinates of the character area, size normalization, value normalization, and binarization processing.

[0056] Specifically, the server can perform preprocessing in the character recognition process on the character area in the image to be processed through the first processing thread, that is, repeatedly execute the preprocessing steps in the character recognition process through the first processing thread to preprocess several character areas in the image to be processed respectively to obtain preprocessing results.

[0057] Step 206 : Perform inference processing in the character recognition process on the preprocessing result through the second processing thread to obtain an inference result.

[0058] Inference processing is the specific process of character recognition based on preprocessing results. For example, character recognition can be performed on preprocessing results using a pretrained artificial neural network model. For example, various models such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and LSTM (Long Short Term Memory) can be used to implement inference processing in character recognition and obtain inference results.

[0059] Specifically, the server can obtain the preprocessing result obtained by the first processing thread through the second processing thread, and the second processing thread can perform inference processing in the character recognition processing on the preprocessing result to facilitate character inference on the preprocessing result to obtain the inference result.

[0060] Step 208 , performing post-processing of the character recognition process on the inference result through the third processing thread to obtain a character recognition result of the image to be processed; wherein the second processing thread is parallel to at least one of the first processing thread and the third processing thread.

[0061] Among them, the specific processing of post-processing can be determined according to the algorithm adopted by the character recognition processing, and can specifically include various processing such as traversing the model output probability vector, dictionary mapping, etc. The character recognition result is the processing result obtained after performing character recognition processing on the image to be processed, and can specifically be the text in the image to be processed. The second processing thread is parallel to at least one of the first processing thread and the third processing thread. Specifically, the second processing thread can be parallel to the first processing thread, that is, while the first processing thread performs pre-processing, the second processing thread performs inference processing on the pre-processing result obtained by the first processing thread. The second processing thread can be parallel to the third processing thread, that is, while the second processing thread performs inference processing, the third processing thread performs post-processing on the inference processing result obtained by the second processing thread. The first processing thread, the second processing thread and the third processing thread can all be executed in parallel, that is, pre-processing is performed by the first processing thread, inference processing is performed by the second processing thread, and post-processing is performed by the third processing thread. In this way, a pipeline processing method for character processing can be implemented, which is conducive to improving image processing efficiency.

[0062] Specifically, the server can post-process the inference results obtained by the second processing thread through the third processing thread to obtain a character recognition result for the image to be processed. The first processing thread, the second processing thread, and the third processing thread are different threads. The second processing thread can be processed in parallel with at least one of the second processing thread and the third processing thread, thereby improving image processing efficiency.

[0063] In the above-mentioned image processing method, for several character regions in the image to be processed, a first processing thread performs preprocessing of the character regions in the character recognition process, a second processing thread performs inference processing on the preprocessing results, and a third processing thread performs postprocessing on the inference results to obtain a character recognition result for the image to be processed. The second processing thread is parallel to at least one of the first and third processing threads. During the image processing process, the preprocessing, inference processing, and postprocessing of the character recognition process are respectively performed by different processing threads, and the thread performing the inference processing is parallel to at least one of the threads performing the preprocessing and the threads performing the postprocessing, thereby achieving parallel processing in the character recognition process, which is beneficial to improving image processing efficiency.

[0064] In one embodiment, Figure 3 As shown, the character area is preprocessed in the character recognition process through the first processing thread to obtain a preprocessing result, including:

[0065] In step 302, a first processing thread is used to perform preprocessing in character recognition processing on character regions of a batch processing quantity obtained sequentially from a character region sequence to obtain a batch preprocessing result; the character region sequence is obtained by sorting the character regions according to region size.

[0066] Among them, the character area sequence is obtained by sorting the character areas according to the area size. Multiple character areas can be detected in the image to be processed, and the sizes of the character areas are different. For the character detection frame, the height of the character detection frame is the same, but the width can be different, that is, the more characters in the area, the larger the width of the corresponding character detection frame. The area size of the character area can refer to the width of the character area. Specifically, the character areas in the image to be processed can be sorted according to the width of the character area to obtain a character area sequence. For example, the character area sequence can be obtained by sorting from small to large according to the area size, or by sorting from large to small according to the area size. The number of batch processing refers to the number of character areas processed in batches each time the character areas are processed in batches, which can be regarded as the batch size. The number of batch processing can be set according to actual needs to ensure the stability of image processing and improve processing efficiency. The batch preprocessing result is the preprocessing result obtained by preprocessing the character areas of the batch processing number.

[0067] Specifically, the server can use a first processing thread to sequentially obtain a batch processing quantity of character regions from a character region sequence, and perform preprocessing in a character recognition process on the obtained character regions in batches to obtain batch preprocessing results. In a specific application, after determining a number of character regions in an image to be processed, the character regions can be sorted according to the area size of each character region to obtain a character region sequence. The server can use a first processing thread to sequentially obtain a batch processing quantity of character regions from a character region sequence, thereby obtaining character regions that need to be batch preprocessed. The server uses the first processing thread to batch preprocess the obtained character regions to obtain batch preprocessing results.

[0068] Furthermore, the second processing thread performs inference processing in the character recognition processing on the preprocessing result to obtain an inference result, including:

[0069] Step 304 : Perform inference processing in character recognition processing on the batch preprocessing results through the second processing thread to obtain batch inference results.

[0070] Specifically, for the batch preprocessing results obtained by the first processing thread through batch preprocessing of the character area, the server can perform batch inference processing on the batch preprocessing results through the second processing thread to obtain batch inference results.

[0071] Furthermore, the inference result is post-processed in the character recognition process by the third processing thread to obtain the character recognition result of the image to be processed, including:

[0072] Step 306 : Post-process the batch inference results in character recognition processing through the third processing thread to obtain a character recognition result of the image to be processed.

[0073] Specifically, for the batch inference results obtained by the second processing thread through batch inference processing, the server can perform post-processing in character recognition processing on the batch inference results through the third processing thread to obtain the character recognition results of the image to be processed.

[0074] In this embodiment, a character area sequence is obtained by sorting the character areas according to area size, and a batch processing quantity of character areas is obtained from the character area sequence, and character recognition processing is performed in batches. Batch character recognition processing can be performed on the character areas in the image to be processed, which can reduce the number of character recognition processing times and improve image processing efficiency.

[0075] In one embodiment, a first processing thread is used to perform preprocessing in character recognition processing on a batch processing quantity of character areas obtained sequentially from a character area sequence to obtain batch preprocessing results, including: obtaining a batch processing quantity of batch character areas sequentially from a character area sequence through the first processing thread; performing preprocessing in character recognition processing on the batch character areas respectively through the first processing thread to obtain preprocessing results corresponding to the batch character areas; performing data expansion processing on the preprocessing results corresponding to the batch character areas through the first processing thread to obtain batch preprocessing results.

[0076] The "batch processing quantity" refers to the number of character regions processed in each batch during batch processing, and can be considered the batch size. The batch processing quantity can be set based on actual needs to ensure image processing stability and improve processing efficiency. A batch of character regions is a batch of data consisting of the number of character regions processed in the batch.

[0077] Specifically, the server obtains a character area sequence obtained by sorting the character areas according to area size through the first processing thread, and obtains batch character areas of batch processing quantity in sequence from the character area sequence. After obtaining the batch character areas, the server performs preprocessing in the character recognition process on the batch character areas respectively through the first processing thread to obtain preprocessing results corresponding to the batch character areas. Specifically, the preprocessing in the character recognition process can be performed on each character area in the batch character area, and the preprocessing results corresponding to the batch character areas are obtained based on the preprocessing results of each character area. When the second processing thread performs inference processing, it is necessary to ensure that the data size of the inference processing is uniform, and the data size of the preprocessing results corresponding to the batch character areas is consistent with the size of the corresponding character areas. At this time, the preprocessing results corresponding to the batch character areas need to be data expanded to make the data size of the preprocessing results corresponding to the batch character areas uniform. Specifically, the server performs data expansion processing on the preprocessing results corresponding to the batch character areas through the first processing thread to obtain batch preprocessing results.

[0078] In a specific implementation, the server may perform data padding on the preprocessing results corresponding to the batch character areas through the first processing thread to expand the preprocessing results corresponding to the batch character areas to the same size. For example, the server may perform 0-padding on the preprocessing results corresponding to the batch character areas through the first processing thread. Specifically, the preprocessing results corresponding to the batch character areas may be 0-padding according to the largest character area in the batch character areas, so that the size of the preprocessing results corresponding to the batch character areas is the same as the area size of the largest character area in the batch character areas.

[0079] In this embodiment, after the first processing thread obtains the batch processing quantity of batch character areas from the character area sequence, preprocessing is performed, and data expansion processing is performed on the preprocessing results corresponding to the batch character areas to obtain batch preprocessing results of uniform size, which facilitates subsequent batch inference processing of the batch preprocessing results and is beneficial to improving image processing efficiency.

[0080] In one embodiment, the batch inference results are post-processed in the character recognition process by a third processing thread to obtain the character recognition result of the image to be processed, including: splitting the batch inference results by the third processing thread to obtain regional inference results corresponding to the batch character areas; and post-processing the regional inference results by the third processing thread to obtain the character recognition result of the image to be processed.

[0081] The batch inference results are the inference results obtained by the second processing thread through batch inference processing, including the inference results corresponding to each character region in the batch character region. The regional inference results are the inference results corresponding to each character region in the batch character region.

[0082] Specifically, the server uses a third processing thread to split the obtained batch inference results to obtain regional inference results corresponding to the batch character regions. In a specific implementation, the inference results corresponding to each character region are of the same size, so the batch inference results can be equally split according to the batch processing quantity to obtain preprocessing results corresponding to the batch character regions. The server uses the third processing thread to post-process the regional inference results in the character recognition process, thereby performing post-processing on each character region in the batch character region, obtaining a character recognition result for each character region, and thus obtaining a character recognition result for the image to be processed.

[0083] In this embodiment, after the batch inference results obtained are split by the third processing thread, post-processing in the character recognition process is performed separately, thereby realizing batch character recognition processing of the character area, which is beneficial to improving image processing efficiency.

[0084] In one embodiment, the image processing method further includes: matching the character regions in the character region sequence with the character recognition results through a second processing thread to obtain a character recognition result that matches the character regions in the image to be processed.

[0085] Among them, when performing batch character recognition processing on each character area, each character area is sorted according to the area size. The actual processing order of each character area is different from the distribution position of the character area in the image to be processed. After the character recognition processing is completed, it is necessary to match each character recognition result with each character area in the image to be processed to ensure the accuracy of the character recognition result.

[0086] Specifically, the server can use the second processing thread to match the character regions in the character region sequence with the character recognition results. For example, the server can use the second processing thread to match each character region with the character recognition results based on index information of each character region in the character region sequence, thereby obtaining a character recognition result that matches the character region in the image to be processed, thereby ensuring the correspondence between the character recognition result and each character region. The index information can be generated when the character regions are sorted to form the character region sequence.

[0087] In this embodiment, each character recognition result is matched with each character region in the image to be processed, thereby ensuring the accuracy of the character recognition result.

[0088] In one embodiment, the second processing thread is a main thread, and the first processing thread and the third processing thread are child threads.

[0089] Among them, the main thread refers to the thread running in the foreground, of which there is generally only one. The second processing thread is the main thread, so that the reasoning processing is performed through the main thread to ensure the processing efficiency of the longer-consuming reasoning processing. The child thread refers to the thread running in the background, of which the number is not fixed. The first processing thread and the third processing thread are child threads. Specifically, the server sets the second processing thread to execute in the main thread, and sets the first processing thread and the third processing thread to execute in the child thread, so that the longer-consuming reasoning processing is set to execute in the main thread with stronger computing power, avoiding blocking problems during parallel processing of character recognition, thereby ensuring image processing efficiency.

[0090] In one embodiment, after the character area is preprocessed in the character recognition process by the first processing thread to obtain the preprocessing result, the method further includes: storing the preprocessing result in a preprocessing result queue by the first processing thread.

[0091] The preprocessing result queue is a pre-set storage queue for storing preprocessing results. Specifically, after the server preprocesses the character area through the first processing thread, the server stores the preprocessing results in the preprocessing result queue through the first processing thread.

[0092] Furthermore, the preprocessing result is subjected to inference processing in character recognition processing through the second processing thread to obtain an inference result, including: obtaining the preprocessing result from the preprocessing result queue through the second processing thread, and performing inference processing in character recognition processing, and storing the obtained inference result in the inference result queue.

[0093] The inference result queue is a pre-configured storage queue for storing inference results. Specifically, the server obtains preprocessing results from the preprocessing result queue via the second processing thread, and performs inference processing on the obtained preprocessing results as in the character recognition process to obtain an inference result. The server then stores the obtained inference results in the inference result queue via the second processing thread.

[0094] Furthermore, the inference result is post-processed in the character recognition process through the third processing thread to obtain the character recognition result of the image to be processed, including: obtaining the inference result from the inference result queue through the third processing thread, and performing post-processing in the character recognition process, and storing the obtained character recognition result in the recognition result queue.

[0095] The recognition result queue is a pre-configured storage queue for character recognition results. Specifically, the server retrieves the inference processing results from the inference result queue via a third processing thread and performs post-processing on the retrieved inference processing results as part of the character recognition process to obtain a character recognition result. The server then stores the obtained character recognition results in the recognition result queue via the third processing thread.

[0096] Furthermore, the image processing method further includes: obtaining, by a second processing thread, a character recognition result of the image to be processed from a recognition result queue.

[0097] Specifically, after the third processing thread stores the obtained character recognition results in the recognition result queue, the server can also read the character recognition results of the image to be processed from the recognition result queue through the second processing thread, so that the character recognition results of the image to be processed can be output or displayed.

[0098] In this embodiment, the processing results of the first processing thread, the second processing thread and the third processing thread can be stored in the corresponding preset storage queues respectively, which can avoid the blocking problem during parallel processing of character recognition, thereby ensuring the processing efficiency of parallel processing of character recognition and ensuring the processing efficiency of the image.

[0099] This application also provides an application scenario, which applies the above-mentioned image processing method. Specifically, the application of the image processing method in this application scenario is as follows:

[0100] The present embodiment involves performing OCR on an image. OCR, or optical character recognition algorithm, is designed to automatically extract and recognize characters in a target image or video through computer vision technology. It is one of the most widely used technologies in the field of computer vision. Generally, an OCR algorithm consists of two parts: a detection process, i.e., locating the character area, and a recognition process, i.e., extracting the characters in the character area, i.e., mapping pixel values ​​into character strings. Generally, after inputting an image to be recognized, an OCR detection is first performed on the entire image to obtain N character areas, which can be specifically displayed in the form of an external rectangular frame; then the OCR recognition algorithm is used to recognize these N character areas one by one to obtain the final result. When the size of the input image is similar, the OCR detection can obtain all detection frames at one time, so the time consumed by the detection step is roughly similar; however, since the OCR detection step is performed on the character areas one by one, the time consumed by the OCR detection step is positively correlated with the number of characters in the input image. Currently, OCR recognition can be implemented based on CNNs. A complete OCR recognition process generally includes three major modules: preprocessing, model inference, and postprocessing. Preprocessing and postprocessing can be performed on the CPU (Central Processing Unit), while inference is performed on the GPU (Graphics Processing Unit). When there are many detection boxes, a large amount of data is transmitted between the CPU and GPU, which wastes a lot of time and reduces CPU and GPU device utilization. Moreover, for images with many characters, such as documents and newspaper photos, OCR recognition takes too long, resulting in low image processing efficiency.

[0101] like Figure 4 As shown in the figure, for the time analysis of an OCR process, the time consumed for a single character area is about 10.2ms. Among them, preprocessing takes 0.84ms, accounting for 8% of the time; inference processing takes 4.77ms, accounting for 47% of the time; and post-processing takes 4.59ms, accounting for 45% of the time. For images with a lot of text, such as a document image with 67 text detection boxes, if they are recognized one by one, it takes about 683ms, and the OCR processing efficiency is low. At the same time, because different operations in OCR recognition run on different devices, only one device is working at the same time, resulting in low device utilization.

[0102] The image processing method of this embodiment is based on the OCR recognition process of batch pipeline processing. Through reasonable scheduling, the preprocessing of different character areas, that is, pre-processing, model reasoning processing, and post-processing operations are carried out simultaneously, which can effectively reduce the overall time consumption and improve equipment utilization. Specifically, if there are 3 character detection frames in a certain OCR task. For a single OCR recognition, the time spent on character recognition in preprocessing, model reasoning processing, and post-processing is the same, which is t. Figure 5 As shown in , if OCR recognition is performed one by one in sequence, the time taken for a single OCR recognition operation is 3t, and the total time taken for the three character detection frames is 9t. Figure 6 As shown in the figure, if a pipeline operation is used, the OCR recognition operations of different character detection frames are performed simultaneously. Specifically, when performing the inference processing of character detection frame 1, the preprocessing of character detection frame 2 is performed at the same time; when performing the post-processing of character detection frame 1, the inference processing of character detection frame 2 and the preprocessing of character detection frame 3 are performed at the same time. This ensures that the equipment is always in a full-load state and shortens the total time to 5t. More generally, assume that the time consumption of preprocessing, model inference processing, and post-processing is t1, t2, and t3 respectively, where the maximum value is tm, and there are N character areas in total. Then the total time consumption of sequential execution is N*(t1+t2+t3), and the total time consumption of pipeline execution is t1+t2+t3+(N-1)*tm. When N is greater than 3, the processing time consumption of OCR recognition can be effectively reduced and the processing efficiency of OCR recognition can be improved.

[0103] Furthermore, this embodiment implements pipeline execution of OCR recognition based on a multi-thread mechanism. Specifically, considering that the inference processing takes the longest time, in order to avoid queue blocking, the inference processing can be carried out in the main thread, while the pre-processing and post-processing can be placed in the sub-thread. In specific implementation, any operation can also be placed in the main thread and other operations can be placed in the sub-thread. Specifically, the specific processing of pre-processing in the character recognition process can be pre-set, such as cropping the text area in the original image according to the detection frame coordinates, size normalization, value normalization, etc.; for post-processing, it can specifically include traversing the model output probability vector, dictionary mapping, etc. The server establishes three queues, queue A is used to store the pre-processed pre-processing results, queue B is used to store the inference results output by the model, and queue C is used to store the character recognition results obtained by post-processing.

[0104] Furthermore, the pre-processing thread PR is initialized. The input of PR is the original image and N character detection frames. The operation performed by PR is to traverse each character detection frame, perform pre-processing operations one by one and store the results in queue A. The end condition of PR processing can be that the pre-processing of N character detection frames is completed. The operation performed by the main thread is to perform N loops. The operation of a single loop is to take a model input from queue A, that is, to obtain the pre-processing result from queue A, perform model inference processing, and store the inference result in queue B. If queue A is empty, the main thread blocks and waits. The post-processing thread PO is initialized. The input of PO is queue B and the number of detection frames N. The operation performed by PO is to read the model output from queue B, that is, to obtain the inference result. After performing the post-processing operation, the character recognition result is stored in queue C. The end condition of PO can be that the number of post-processing times reaches N. In addition, after the post-processing loop ends, the main thread can also take out N outputs from queue C and return to obtain the OCR recognition result. As Figure 7 As shown in the figure, in a specific application, the total OCR processing time decreased from 683ms to 384ms, a decrease of approximately 44%. The average time taken for each OCR process was 0.02ms for pre-processing, accounting for nearly 0% of the total time; 4.97ms for inference, accounting for 99% of the total time; and 0.04ms for post-processing, accounting for 1% of the total time.

[0105] Furthermore, OCR recognition can be implemented using a batch pipeline. This can be done using a CNN model, which supports batch inference. As the batch size (BS) increases, the time required increases much less than linearly. For example, if a single inference takes 10ms, performing two inferences with a BS of 1 for two text regions would take 20ms, while performing one inference with a BS of 2 might reduce the time to 15ms. Therefore, batch processing combined with a pipeline can further improve processing efficiency.

[0106] Due to the limitations of the CNN architecture, inputs within a batch must maintain the same size. Therefore, batch processing requires a size selection issue. The OCR recognition algorithm requires maintaining a consistent height of the input character regions, for example, 32 pixels. In practice, this can be achieved by setting a fixed value W as the uniform width. Regions that are insufficient in width are expanded using zero padding, while those that exceed the width are compressed using interpolation. However, determining W is difficult. If W is too large, a large number of regions will need to be expanded, and the increased time spent on expansion will offset the time saved by batch processing, affecting OCR processing efficiency. If W is too small, a large number of regions will be compressed, causing characters to stick together, resulting in poor recognition and reduced accuracy. Alternatively, the width of the longest region within a batch can be selected as the uniform width for the current batch. This ensures that no text regions are compressed. However, due to the random distribution of text region widths, each batch contains both long and short text regions. Excessive expansion of short regions increases OCR processing time.

[0107] Based on this, all the character detection frames in the image to be processed can be sorted by width from small to large first, and then batched in order with a certain batch size, such as a batch size of 4 or 8, to ensure that the widths of the character areas within the same batch are close, thereby avoiding large-scale expansion. Specifically, the specific processing of pre-processing in the character recognition process can be set in advance, such as including cropping the text area in the original image according to the detection frame coordinates, size normalization, value normalization, etc.; for post-processing, it can specifically include traversing the model output probability vector, dictionary mapping, etc. The server establishes three queues, queue A is used to store the pre-processed results, queue B is used to store the inference results output by the model, and queue C is used to store the character recognition results obtained by post-processing.

[0108] Furthermore, the server sorts all character detection frames by width from smallest to largest, retaining the original index information. For example, the width of the shortest frame ranked 1 after sorting is the original 18th frame. The preprocessing thread PR is initialized. PR's input is the original image, N character detection frames, and the batch size BS. PR sequentially retrieves BS character detection frames. If there are fewer than BS character detection frames, it retrieves the actual number of character detection frames. The preprocessing thread PR iterates over the BS character detection frames, performing preprocessing operations on each one. PR targets the character detection frame with the largest width, zero-padding each preprocessing result, and then stores the batch data in queue A. PR's processing can terminate when all N character detection frames have been preprocessed. The main thread executes N loops. Each loop takes a model input from queue A, retrieves the preprocessing result from queue A, performs model inference, and stores the inference result in queue B. If queue A is empty, the main thread blocks and waits. Initialize the post-processing thread PO. The input of PO is queue B and the number of detection boxes N. The operation performed by PO is to read the model output from queue B, that is, to obtain the inference result. PO splits the batch output into individual character region outputs, and after performing post-processing operations on each character region output, stores the character recognition results in queue C. The termination condition of PO can be that the number of post-processing times reaches N. After the post-processing loop ends, the main thread can also take out N outputs from queue C and reorder them according to the original index so that the final character recognition result corresponds to the original detection box in order and returns it to obtain the OCR recognition result.

[0109] like Figure 8 As shown in the figure, in a specific application, the total OCR processing time dropped from 683ms to 131ms, a decrease of approximately 80%. The average time taken for each OCR process was 0.04ms for pre-processing, accounting for nearly 2% of the total time; 1.83ms for inference, accounting for 93% of the total time; and 0.09ms for post-processing, accounting for 5% of the total time.

[0110] The image processing method implemented in this embodiment enables the electronic device to simultaneously process different recognition processes of different character areas through a pipelined reasoning processing flow, thereby increasing the recognition speed by increasing the utilization efficiency of the device, thereby improving the image processing efficiency.

[0111] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0112] Based on the same inventive concept, embodiments of the present application also provide an image processing device for implementing the aforementioned image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image processing device embodiments can be found in the above-described limitations on the image processing method and will not be further elaborated here.

[0113] In one embodiment, Figure 9 As shown, an image processing device 900 is provided, comprising: a character region determination module 902, a pre-processing module 904, an inference module 906 and a post-processing module 908, wherein:

[0114] A character region determination module 902 is configured to determine a plurality of character regions in the image to be processed;

[0115] A preprocessing module 904 is configured to perform preprocessing in character recognition processing on the character region through a first processing thread to obtain a preprocessing result;

[0116] An inference module 906 is configured to perform an inference process in the character recognition process on the preprocessing result through a second processing thread to obtain an inference result;

[0117] The post-processing module 908 is used to perform post-processing in the character recognition processing on the inference result through the third processing thread to obtain the character recognition result of the image to be processed; wherein, the second processing thread is parallel to at least one of the first processing thread and the third processing thread.

[0118] In one embodiment, the preprocessing module 904 is further used to perform preprocessing in character recognition processing on the character areas of batch processing quantity obtained in sequence from the character area sequence through the first processing thread to obtain batch preprocessing results; the character area sequence is obtained by sorting the character areas according to area size; the inference module 906 is further used to perform inference processing in character recognition processing on the batch preprocessing results through the second processing thread to obtain batch inference results; the post-processing module 908 is further used to perform post-processing in character recognition processing on the batch inference results through the third processing thread to obtain character recognition results of the image to be processed.

[0119] In one embodiment, the preprocessing module 904 includes a batch area acquisition module, a batch preprocessing module and a data expansion module; wherein: the batch area acquisition module is used to sequentially obtain a batch processing quantity of batch character areas from a character area sequence through a first processing thread; the batch preprocessing module is used to perform preprocessing in character recognition processing on the batch character areas through the first processing thread to obtain preprocessing results corresponding to the batch character areas; the data expansion module is used to perform data expansion processing on the preprocessing results corresponding to the batch character areas through the first processing thread to obtain batch preprocessing results.

[0120] In one embodiment, the post-processing module 908 is also used to split the batch inference results through the third processing thread to obtain regional inference results corresponding to the batch character areas; and to perform post-processing in the character recognition processing on the regional inference results through the third processing thread to obtain the character recognition results of the image to be processed.

[0121] In one embodiment, a recognition result matching module is further included, which is used to match the character areas in the character area sequence with the character recognition results through the second processing thread to obtain a character recognition result that matches the character area in the image to be processed.

[0122] In one embodiment, the second processing thread is a main thread, and the first processing thread and the third processing thread are child threads.

[0123] In one embodiment, it also includes a preprocessing result storage module, which is used to store the preprocessing results in the preprocessing result queue through the first processing thread; the inference module 906 is also used to obtain the preprocessing results from the preprocessing result queue through the second processing thread, and perform inference processing in the character recognition processing, and store the obtained inference results in the inference result queue; the post-processing module 908 is also used to obtain the inference results from the inference result queue through the third processing thread, and perform post-processing in the character recognition processing, and store the obtained character recognition results in the recognition result queue; it also includes a recognition result acquisition module, which is used to obtain the character recognition results of the image to be processed from the recognition result queue through the second processing thread.

[0124] Each module in the above-mentioned image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in the electronic device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0125] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the electronic device is used to store image processing data. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0126] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown. The electronic device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals via wired or wireless means, and the wireless means can be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements an image processing method. The display unit of the electronic device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0127] Those skilled in the art will understand that Figure 10 and Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0128] The present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of the image processing method.

[0129] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0131] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0132] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image processing method, characterized in that: include: Determine a number of character regions in the image to be processed; performing preprocessing in character recognition processing on a batch processing quantity of character regions sequentially obtained from a character region sequence by a first processing thread, to obtain a batch preprocessing result; the character region sequence is obtained by sorting the character regions according to region size; Performing the inference processing in the character recognition processing on the batch preprocessing results through the second processing thread to obtain batch inference results; Performing post-processing in the character recognition process on the batch inference results through a third processing thread to obtain a character recognition result of the image to be processed; The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

2. The method according to claim 1, characterized in that The method of performing preprocessing in character recognition processing on the character regions of a batch processing quantity obtained in sequence from the character region sequence by the first processing thread to obtain batch preprocessing results includes: Obtaining, by the first processing thread, a batch processing quantity of batch character regions from the character region sequence in sequence; By using the first processing thread, preprocessing in the character recognition process is performed on the batch character areas respectively to obtain preprocessing results corresponding to the batch character areas; The first processing thread performs data expansion processing on the preprocessing results corresponding to the batch character areas to obtain batch preprocessing results.

3. The method according to claim 2, characterized in that The first processing thread performs data expansion processing on the preprocessing results corresponding to the batch character areas to obtain batch preprocessing results, including: The preprocessing results corresponding to the batch character areas are filled with data through the first processing thread to expand the preprocessing results corresponding to the batch character areas to the same size.

4. The method according to claim 2, characterized in that The post-processing of the character recognition process on the batch inference results by the third processing thread to obtain the character recognition result of the image to be processed includes: Splitting the batch inference results through a third processing thread to obtain regional inference results corresponding to the batch character regions; The regional inference results are respectively subjected to post-processing in the character recognition process through the third processing thread to obtain the character recognition result of the image to be processed.

5. The method according to claim 1, wherein The method further comprises: Matching the character regions in the character region sequence with the character recognition results through the second processing thread to obtain a character recognition result that matches the character regions in the image to be processed; The second processing thread is a main thread, and the first processing thread and the third processing thread are sub-threads.

6. The method according to any one of claims 1 to 5, characterized in that After performing preprocessing in character recognition processing on the character regions of the batch processing quantity obtained in sequence from the character region sequence by the first processing thread and obtaining the batch preprocessing result, the method further includes: Storing the batch preprocessing results in a preprocessing result queue through the first processing thread; The step of performing the inference processing in the character recognition processing on the batch preprocessing results by the second processing thread to obtain an inference result includes: Obtaining the batch preprocessing results from the preprocessing result queue through a second processing thread, performing inference processing in the character recognition process, and storing the obtained batch inference results in the inference result queue; The post-processing of the character recognition process on the batch inference results by the third processing thread to obtain the character recognition result of the image to be processed includes: Obtaining the batch inference results from the inference result queue through a third processing thread, performing post-processing in the character recognition process, and storing the obtained character recognition results in the recognition result queue; The method further comprises: The character recognition result of the image to be processed is obtained from the recognition result queue through the second processing thread.

7. An image processing device, characterized in that include: A character region determination module, used for determining a number of character regions in the image to be processed; a preprocessing module configured to perform preprocessing in character recognition processing on a batch processing quantity of character regions sequentially obtained from a character region sequence by sorting the character regions according to region size, to obtain a batch preprocessing result, through a first processing thread; an inference module, configured to perform, through a second processing thread, an inference process in the character recognition process on the batch preprocessing results to obtain batch inference results; a post-processing module, configured to perform post-processing in the character recognition process on the batch inference results through a third processing thread to obtain a character recognition result of the image to be processed; The second processing thread is executed in parallel with at least one of the first processing thread and the third processing thread.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images