Image information extraction method and device, electronic equipment and storage medium

Abstract

The invention provides an image information extraction method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the model compression processing of an optical character recognition model, so as to obtain a lightweight optical character recognition model; performing text recognition on the input image by using the lightweight model at the edge equipment end to obtain text content and position information, then judging whether to trigger cloud processing according to a text recognition result, and sending the input image and the text recognition result to the cloud during triggering; the cloud server performs multi-modal information extraction through a large language model to obtain structured information, so that the problems that in the prior art, an optical character recognition model is difficult to adapt to the computing power of edge equipment, and end-cloud cooperation lacks a reasonable triggering mechanism, so that response delay or resource waste is caused, and the efficiency is high can be solved. And edge equipment cannot directly deploy a large model to realize high-precision structured information extraction.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image information extraction method and apparatus, electronic device and storage medium. Background Technology

[0002] Image information extraction is a key technology for the integration of artificial intelligence and edge computing, and is widely used in scenarios such as identity verification, ticket processing, and industrial inspection. This technology constructs a complete system from image preprocessing to semantic parsing by working in conjunction with optical character recognition (OCR) and large language models, covering key steps such as text localization, feature extraction, and context modeling. With the development of deep learning, existing methods mostly adopt an end-to-end OCR model combined with a large language model to achieve high-precision information extraction. The former relies on convolutional neural networks to extract text features, while the latter is based on the Transformer architecture for semantic parsing. However, this technical system faces many challenges when deployed on edge devices.

[0003] Existing methods generally employ uniform quantization strategies, lacking systematic analysis of the sensitivity of parameter matrix quantization. Common lightweight optimization techniques for optical character recognition, while reducing the number of parameters through model quantization and pruning, suffer significant accuracy degradation and require extensive fine-tuning with labeled data. Furthermore, large cloud-based language models demand substantial memory and high power consumption, while edge devices have limited memory and power consumption, creating a clear mismatch. This results in slow response times for existing edge-cloud collaborative solutions in complex layout recognition scenarios, failing to meet the real-time requirements of scenarios like smart access control. The industry urgently needs a lightweight solution that requires minimal training data and can dynamically adapt to the computing power of edge devices. Summary of the Invention

[0004] This disclosure provides an image information extraction method, apparatus, electronic device, and storage medium. Its main objective is to at least partially address one of the technical problems in related technologies.

[0005] According to a first aspect of this disclosure, an image information extraction method is provided, comprising: The optical character recognition model is compressed to obtain a lightweight optical character recognition model. At the edge device, the lightweight optical character recognition model is used to perform text recognition on the input image to obtain the text content and its location information in the image. Based on the results of the text recognition, determine whether to trigger cloud processing; When cloud processing is triggered, the input image and the text recognition result are sent to the cloud server; On a cloud server, a large language model is used to extract multimodal information from the input image and the text recognition results to obtain structured information.

[0006] Optionally, the model compression processing of the optical character recognition model includes: The optical character recognition model is then pruned and quantized.

[0007] Optionally, the pruning process includes: The importance of each layer is evaluated based on the norm of the parameter matrix, and the sparsity rate is dynamically allocated. The dynamic allocation of the sparsity rate is adjusted according to the ratio of the number of pruned parameters to the total number of parameters in the model.

[0008] Optionally, the quantization process includes: The quantization sensitivity is evaluated based on the condition number of the weight matrix and the Lipschitz constant of the activation matrix, and different quantization bits are assigned to different layers according to the evaluation results to achieve mixed-precision quantization.

[0009] Optionally, determining whether to trigger cloud processing based on the text recognition result includes: The confidence level of the text recognition result is compared with a preset threshold, or the result is based on the output of the task complexity evaluation module.

[0010] Optionally, the large language model is a cross-modal large language model, which can process image and text inputs simultaneously and perform joint reasoning by combining text location information.

[0011] According to a second aspect of this disclosure, an image information extraction apparatus is provided, comprising: The compression unit is used to perform model compression processing on the optical character recognition model to obtain a lightweight optical character recognition model. The recognition unit is used at the edge device to perform text recognition on the input image using the lightweight optical character recognition model to obtain the text content and its position information in the image; The judgment unit is used to determine whether to trigger cloud processing based on the result of the text recognition. The sending unit is used to send the input image and the text recognition result to the cloud server when cloud processing is triggered; The extraction unit is used on a cloud server to extract multimodal information from the input image and the text recognition results using a large language model, thereby obtaining structured information.

[0012] Optionally, the compression unit is also used for: The optical character recognition model is then pruned and quantized.

[0013] Optionally, the pruning process includes: The importance of each layer is evaluated based on the norm of the parameter matrix, and the sparsity rate is dynamically allocated. The dynamic allocation of the sparsity rate is adjusted according to the ratio of the number of pruned parameters to the total number of parameters in the model.

[0014] Optionally, the quantization process includes: The quantization sensitivity is evaluated based on the condition number of the weight matrix and the Lipschitz constant of the activation matrix, and different quantization bits are assigned to different layers according to the evaluation results to achieve mixed-precision quantization.

[0015] Optionally, the decision unit is also used for: The confidence level of the text recognition result is compared with a preset threshold, or the result is based on the output of the task complexity evaluation module.

[0016] Optionally, the large language model is a cross-modal large language model, which can process image and text inputs simultaneously and perform joint reasoning by combining text location information.

[0017] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0020] The image information extraction method, apparatus, electronic device, and storage medium disclosed herein compress an optical character recognition model to obtain a lightweight optical character recognition model. This lightweight model is then used on an edge device to perform text recognition on an input image to obtain text content and location information. Subsequently, based on the text recognition result, it is determined whether cloud processing should be triggered. If triggered, the input image and text recognition result are sent to the cloud. The cloud server then uses a large language model to extract multimodal information to obtain structured information. Therefore, this method solves the problems in existing technologies where the optical character recognition model is difficult to adapt to the computing power of edge devices, the lack of a reasonable triggering mechanism for edge-cloud collaboration leads to response delays or resource waste, and edge devices cannot directly deploy large models to achieve high-precision structured information extraction. This method achieves the technical effect of adapting the optical character recognition model to the computing power of edge devices, optimizing edge-cloud resource allocation, ensuring real-time processing at the edge, and improving information extraction accuracy.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart illustrating an image information extraction method provided in this embodiment of the present disclosure; Figure 2 This is a schematic diagram of the structure of an image information extraction device provided in an embodiment of the present disclosure; Figure 3 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] The image information extraction method, apparatus, electronic device, and storage medium of the present disclosure are described below with reference to the accompanying drawings.

[0025] Figure 1 This is a schematic flowchart of an image information extraction method provided in an embodiment of the present disclosure.

[0026] like Figure 1As shown, the method includes the following steps: Step 101: Perform model compression processing on the optical character recognition model to obtain a lightweight optical character recognition model.

[0027] In the embodiments of this disclosure, to adapt the optical character recognition model to the limited computing power, memory, and storage resources of edge devices and meet the real-time processing requirements of the edge, model compression processing needs to be performed on the initial optical character recognition model. This processing reduces the resource consumption and computing power consumption during model operation by reducing the model parameter size, optimizing the model calculation process, or simplifying the network structure, ultimately resulting in a lightweight optical character recognition model that can be deployed on edge devices. As one implementation method, the model compression processing can adopt a post-training optimization method that does not rely on a large amount of labeled data, avoiding complex training processes while controlling accuracy loss and ensuring that the lightweight model is adapted to the operating environment of edge devices.

[0028] This step obtains a lightweight optical character recognition model through model compression, which effectively solves the problem that traditional optical character recognition models cannot be deployed on edge devices due to their large number of parameters and high computing power requirements. This provides a foundation for efficient text recognition at the edge and ensures that the lightweight model still has the recognition performance to meet the needs of practical applications.

[0029] Step 102: At the edge device, the lightweight optical character recognition model is used to perform text recognition on the input image to obtain the text content and its position information in the image.

[0030] In the embodiments of this disclosure, at an edge device with limited computing power and storage resources, a lightweight optical character recognition model is used to perform text recognition on an input image containing text information. This recognition process detects text regions in the input image, parses characters, and extracts information through the model, ultimately outputting two core results simultaneously: the text content itself contained in the image, and the positional information of each text content in the input image coordinate system. This provides basic data support for subsequent determination of whether to trigger cloud processing. As one implementation method, the input image may include scene images such as identity documents or tickets from which key information needs to be extracted. The text positional information can be represented as the boundary coordinates of the text regions to ensure the accuracy of the information.

[0031] By running a lightweight optical character recognition model locally on the edge device to complete text recognition, the problem of excessive network bandwidth consumption and processing delay caused by directly uploading the original image to the cloud is avoided. At the same time, the text content and location information acquired synchronously provide the necessary basis for subsequent cloud processing and judgment, effectively ensuring the real-time performance and data validity of text recognition at the edge and adapting to the operating capabilities of the edge device.

[0032] Step 103: Based on the result of the text recognition, determine whether to trigger cloud processing.

[0033] In the embodiments of this disclosure, after obtaining the text recognition result (including the text content and its position information in the image), a judgment logic is constructed based on the key attributes reflected by the result (such as the completeness of the text information, the reliability of the recognition result, the complexity of the text layout, etc.) to determine whether it is necessary to call the computing resources of the cloud server for subsequent processing. This judgment process aims to distinguish between scenarios where the edge device can independently complete information extraction and scenarios where a large cloud model is needed to improve the processing accuracy, avoiding unnecessary cloud interaction while ensuring the quality of information extraction in complex scenarios. As one implementation method, the judgment can be made based on the degree of missing text content, the ambiguity of the position information, or a preset recognition confidence threshold. When the text recognition result has obvious missing parts, ambiguity, or a confidence level lower than the threshold, it is determined that cloud processing is triggered.

[0034] This step dynamically determines whether to trigger cloud processing based on the text recognition results, effectively avoiding the accuracy problems caused by "full edge processing" and the resource waste and delay problems caused by "full cloud processing". It achieves a reasonable allocation of resources between the edge and the cloud, ensuring real-time processing efficiency in simple scenarios and providing a triggering mechanism for high-precision information extraction in complex scenarios.

[0035] Step 104: When cloud processing is triggered, the input image and the text recognition result are sent to the cloud server.

[0036] In the embodiments of this disclosure, when it is determined that cloud processing needs to be triggered based on the text recognition results, the input image and the aforementioned text recognition results (including text content and corresponding location information) need to be transmitted together to the cloud server. The input image is the original image to be processed, used to provide complete visual information support for the large cloud model. The text recognition results provide the cloud with basic text and location data obtained from preliminary edge processing. The combination of these two avoids the cloud from repeatedly performing basic text recognition operations, laying a data foundation for subsequent multimodal information extraction. As one implementation method, the data transmission process can adopt a secure transmission method that meets the communication requirements between edge devices and the cloud, ensuring the integrity and security of the input image and text recognition results during transmission, and preventing data loss or leakage.

[0037] This step synchronously transmits the input image and text recognition results when cloud processing is triggered. This saves the computing power and time cost of re-performing text recognition in the cloud, and provides complete visual and text data for the large cloud model, ensuring the efficiency and accuracy of subsequent multimodal information extraction. At the same time, it adapts to the data interaction requirements under the edge-cloud collaborative architecture and avoids cloud processing deviations caused by single data transmission.

[0038] Step 105: On the cloud server, use a large language model to extract multimodal information from the input image and the text recognition results to obtain structured information.

[0039] In the embodiments of this disclosure, on a cloud server with sufficient computing power, a large language model is used to perform multimodal information extraction on the received input image and the text recognition results (including text content and corresponding location information) generated at the edge. This process leverages the large language model's ability to fuse and analyze visual and text data, combining the visual features of the input image with the semantic information of the text recognition results to identify, verify, and standardize the key information carried by the image, ultimately outputting structured information with a unified format that can be directly applied, completing the transformation from unstructured image data to standardized information. As one implementation method, the large language model can have cross-modal information processing capabilities, and the structured information can be presented in a field-based format adapted to specific business scenarios, facilitating direct subsequent use.

[0040] This step uses a cloud-based large language model to fuse two types of data for multimodal information extraction. This solves the problem that edge devices cannot achieve high-precision structured information extraction due to computing power limitations. It also reduces redundant calculations in the cloud by relying on text recognition results. At the same time, it combines the input image to ensure the accuracy and completeness of information extraction. The final output structured information can directly meet the needs of practical applications, improving the practicality and reliability of the overall information extraction process.

[0041] The image information extraction method disclosed herein compresses an optical character recognition (OCR) model to obtain a lightweight OCR model. This lightweight model is then used on an edge device to perform text recognition on the input image, obtaining text content and location information. The method then determines whether to trigger cloud processing based on the text recognition results. If triggered, the input image and text recognition results are sent to the cloud. The cloud server then uses a large language model to extract multimodal information to obtain structured information. Therefore, this method solves the problems in existing technologies where the OCR model is difficult to adapt to the computing power of edge devices, the lack of a reasonable triggering mechanism for edge-cloud collaboration leads to response delays or resource waste, and edge devices cannot directly deploy large models to achieve high-precision structured information extraction. This method achieves the technical effect of adapting the OCR model to the computing power of edge devices, optimizing edge-cloud resource allocation, ensuring real-time processing at the edge, and improving information extraction accuracy.

[0042] As a specific implementation of this disclosure, based on the basic scheme, the model compression processing of the optical character recognition model is further defined as including: pruning and quantization processing of the optical character recognition model.

[0043] Specifically, the pruning process for the optical character recognition model first assesses the importance of the weight matrices of each layer (e.g., using the norm index of the weight matrix to determine the contribution of filters to the model output). Based on the assessment results, sparsity is dynamically assigned to different layers—lower sparsity is assigned to layers with high contribution and strong importance to retain core parameters, while higher sparsity is assigned to layers with low contribution and high redundancy. Then, channel-level structured pruning is performed on each layer according to the sparsity, removing unimportant filters and corresponding connections to reduce the number of model parameters and computational load. The quantization process first analyzes the quantization sensitivity of the parameter matrices of each linear layer of the model. Based on the sensitivity differences, appropriate quantization bits are assigned to different layers. The weights and activation values ​​originally stored in high-precision floating-point form in the model are converted into low-precision discrete values ​​through a quantization mapping function (e.g., some layers are converted to 4-bit or 8-bit integer values). At the same time, the quantization scaling factor and truncation threshold are adjusted to control quantization error, ultimately completing the model compression that combines pruning and quantization.

[0044] By removing redundant parameters from the model through pruning and reducing parameter storage precision through quantization, the two work together to more efficiently compress the size and computational requirements of the optical character recognition model. Compared with a single compression method, it is more comprehensively adapted to the limited memory and computing power of edge devices. At the same time, dynamic pruning based on importance and quantization based on sensitivity can effectively control the loss of accuracy while greatly compressing the model, ensuring the text recognition performance at the edge.

[0045] As a specific implementation of this disclosure, based on the basic scheme, the pruning process is further defined to include: evaluating the importance of each layer based on the norm of the parameter matrix and dynamically allocating the sparsity rate, wherein the dynamic allocation of the sparsity rate is adjusted according to the ratio of the number of pruned parameters to the total number of parameters in the model.

[0046] Specifically, when pruning the optical character recognition model, the importance of each layer of the model is first evaluated based on the norm of the parameter matrix. Specifically, the optimized average F-norm (calculated as the square root of the sum of squares of all elements in the parameter matrix divided by the total number of parameters) can be used. This norm can eliminate the interference of differences in the number of parameters in different layers on the importance assessment, and more objectively reflect the contribution of each layer to the model's recognition performance—the larger the average F-norm, the richer the effective feature information contained in its parameters, and the higher the layer's importance; conversely, the lower the importance, the lower the importance. Then, sparsity is dynamically allocated based on this importance assessment result: a lower sparsity rate (e.g., 10%-20%) is allocated to highly important layers with a large average F-norm (such as key feature extraction layers of the model), removing only a small number of redundant parameters; a higher sparsity rate (e.g., 60%-70%) is allocated to low-importance layers with a small average F-norm (such as some transition layers), significantly simplifying redundant structures. During the allocation process, it is necessary to calculate the total number of pruned parameters in each layer according to the current sparsity rate in real time, and use the ratio of this to the total number of parameters in the model as the basis for adjustment: if the ratio is lower than the preset compression target (e.g., 50%), the sparsity rate of the low importance layers is appropriately increased; if the ratio is higher than the target, the sparsity rate of some medium importance layers is reduced until the ratio of the number of pruned parameters to the total number of parameters meets the preset compression requirements, and finally the channel-level structured pruning is completed, removing unimportant filters and corresponding connections in each layer.

[0047] By accurately assessing the importance of layers using norms, excessive pruning of core layer parameters caused by uniform sparsity is avoided, reducing performance loss. At the same time, the sparsity is dynamically adjusted based on the parameter ratio, which can precisely control the degree of model compression and ensure that the number of pruned model parameters is highly adapted to the memory and computing power requirements of edge devices, so as to retain the model recognition accuracy to the greatest extent while achieving lightweight design.

[0048] As a specific implementation of this disclosure, based on the basic scheme, the quantization process is further defined to include: evaluating the quantization sensitivity based on the condition number of the weight matrix and the Lipschitz constant of the activation matrix, and allocating different quantization bits to different layers according to the evaluation results to achieve mixed precision quantization.

[0049] Specifically, when quantizing the optical character recognition model, the condition number is first calculated for the weight matrix of each linear layer (such as convolutional layers and fully connected layers): the maximum and minimum singular values ​​of the weight matrix are obtained through singular value decomposition, and the ratio of the two is the condition number. The larger the condition number, the more sensitive the weight matrix is ​​to quantization perturbations, and small parameter changes can easily lead to drastic fluctuations in output features. At the same time, the Lipshitz constant is calculated for the input activation matrix of each layer. This constant is the spectral norm of the activation matrix (i.e., the maximum singular value). The larger its value, the more susceptible the mapping relationship of the activation matrix is ​​to quantization, and the worse the stability of the inference result. Based on the above evaluation results, quantization bits are allocated: for the layers containing weight matrices with a condition number greater than a preset threshold (e.g., 10) and activation matrices with a Lipshitz constant greater than a preset threshold (e.g., 5), 8 quantization bits are allocated to reduce the impact of perturbations; for layers where both the condition number and Lipshitz constant are less than the corresponding thresholds, 4 quantization bits are allocated to simplify storage. During the quantization process, the quantization scaling factor (calculated as the ratio of the difference between the upper and lower bounds of the parameter to the number of discrete values ​​that can be represented by the target bits) and the truncation function are dynamically adjusted in combination with the parameter range of each layer to ensure that key parameter information is not lost during low-precision conversion, and ultimately achieve mixed-precision quantization.

[0050] By accurately locating the quantization sensitivity of each layer using condition number and Lipshitz constant, excessive loss of core layer accuracy caused by uniform quantization is avoided. At the same time, by differentiating bit allocation, parameter storage is compressed to the maximum extent while ensuring model performance, making the quantized model more suitable for the limited memory resources of edge devices, thus balancing lightweight design and recognition accuracy.

[0051] As a specific implementation of this disclosure, based on the basic solution, the determination of whether to trigger cloud processing based on the text recognition result is further defined, including: a comparison of the confidence level of the text recognition result with a preset threshold, or the output result of the task complexity evaluation module.

[0052] Specifically, when determining whether to trigger cloud processing based on text recognition results, the first method is confidence comparison: The lightweight optical character recognition model, while outputting text content and location information, simultaneously generates the recognition confidence score for each text segment (this confidence score is a quantitative rating of the model's reliability of the recognition result, typically ranging from 0 to 1). A confidence threshold is pre-set according to the actual application scenario requirements (e.g., a threshold of 0.8 for identity information extraction and 0.7 for invoice information extraction). The confidence score of each text segment is compared with the preset threshold one by one. If at least one text segment has a confidence score lower than the threshold, or if the overall average confidence score is lower than the threshold, then cloud processing needs to be triggered. The second method relies on the task complexity assessment module: This module pre-builds text complexity assessment dimensions (including text layout density, number of text languages, and whether the text is blurred / skewed / occluded, etc.). It analyzes the text layout and content features in the text recognition results and outputs a complexity score of 0-10 (a higher score indicates a more complex task). If the score is higher than the preset complexity threshold (e.g., 6 points), then cloud processing needs to be triggered. In practical applications, either of the two judgment methods mentioned above can trigger the cloud processing flow.

[0053] By comparing confidence levels, the reliability of edge recognition results can be intuitively judged, avoiding the waste of resources caused by uploading simple tasks with accurate recognition to the cloud; the task complexity assessment module can accurately identify complex scenarios that are difficult for edge devices to handle, ensuring the accuracy of information extraction. The combination of the two judgment methods makes the triggering logic more comprehensive, which can not only meet the real-time requirements of simple scenarios, but also meet the accuracy requirements of complex scenarios.

[0054] As a specific implementation of this disclosure, based on the basic scheme, the large language model is further defined as a cross-modal large language model, which can process image and text input simultaneously and perform joint reasoning by combining text location information.

[0055] Specifically, when performing multimodal information extraction on the cloud server, the cross-modal large language model employs a built-in visual processing branch and a text processing branch: the visual processing branch extracts features from the input image through a convolutional neural network or a visual Transformer module, first dividing the image into fixed-size feature blocks to generate a visual feature map containing global semantics and local regional details; the text processing branch performs word segmentation and semantic encoding on the text recognition results transmitted from the edge, converting the text content into text semantic features in vector form, and converting the text's position information in the image (such as the boundary coordinates with the top left corner of the image as the origin) into a spatial feature vector, and binding the spatial feature vector to the corresponding regional features in the visual feature map through a coordinate mapping algorithm, establishing a one-to-one correspondence between "text semantics - image region". During the joint inference phase, the model uses a cross-modal attention mechanism to guide the text semantic features and the bound image region features: on the one hand, it locates key regions in the image based on the text semantics (such as the number region in the image corresponding to the text "ID number"), and on the other hand, it supplements ambiguous information in the text with the help of image region features (such as the ambiguous text "1" and "7" can be identified by the visual outline of the characters in the image), and finally outputs structured information with field-based and uniform format.

[0056] By enabling joint reasoning between images and text through a cross-modal large language model, we can avoid semantic ambiguity problems that cannot be solved by a single text modality, and accurately locate key areas of images by leveraging text location information, thereby reducing interference from invalid visual information. Compared with traditional single-modal models, we can significantly improve the accuracy of structured information extraction in complex scenes (such as blurred or tilted images), while reducing the processing cost of redundant image features in the cloud.

[0057] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0058] Corresponding to the image information extraction method described above, this disclosure also proposes an image information extraction device. Since the device embodiments of this disclosure correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to the method embodiments described above, and will not be repeated here.

[0059] Figure 2 This is a schematic diagram of the structure of an image information extraction device provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, it includes: Compression unit 21 is used to perform model compression processing on the optical character recognition model to obtain a lightweight optical character recognition model; The recognition unit 22 is used at the edge device end to perform text recognition on the input image using the lightweight optical character recognition model to obtain the text content and its position information in the image; The judgment unit 23 is used to determine whether to trigger cloud processing based on the result of the text recognition; The sending unit 24 is used to send the input image and the text recognition result to the cloud server when cloud processing is triggered; Extraction unit 25 is used to extract multimodal information from the input image and the text recognition result on a cloud server using a large language model to obtain structured information.

[0060] The image information extraction device disclosed herein compresses an optical character recognition model to obtain a lightweight optical character recognition model. This lightweight model is then used on an edge device to perform text recognition on the input image, obtaining text content and location information. The device then determines whether to trigger cloud processing based on the text recognition results. If triggered, the input image and text recognition results are sent to the cloud, where a cloud server uses a large language model to extract multimodal information and obtain structured information. Therefore, this device solves the problems in existing technologies where the optical character recognition model is difficult to adapt to the computing power of edge devices, the lack of a reasonable triggering mechanism for edge-cloud collaboration leads to response delays or resource waste, and edge devices cannot directly deploy large models to achieve high-precision structured information extraction. This achieves the technical effect of adapting the optical character recognition model to the computing power of edge devices, optimizing edge-cloud resource allocation, ensuring real-time processing at the edge, and improving information extraction accuracy.

[0061] Furthermore, in one possible implementation of this embodiment, the compression unit 21 is also used for: The optical character recognition model is then pruned and quantized.

[0062] Furthermore, in one possible implementation of this embodiment, the pruning process includes: The importance of each layer is evaluated based on the norm of the parameter matrix, and the sparsity rate is dynamically allocated. The dynamic allocation of the sparsity rate is adjusted according to the ratio of the number of pruned parameters to the total number of parameters in the model.

[0063] Furthermore, in one possible implementation of this embodiment, the quantization process includes: The quantization sensitivity is evaluated based on the condition number of the weight matrix and the Lipschitz constant of the activation matrix, and different quantization bits are assigned to different layers according to the evaluation results to achieve mixed-precision quantization.

[0064] Furthermore, in one possible implementation of this embodiment, the determining unit 23 is also used for: The confidence level of the text recognition result is compared with a preset threshold, or the result is based on the output of the task complexity evaluation module.

[0065] Furthermore, in one possible implementation of this embodiment, the large language model is a cross-modal large language model, which can process image and text inputs simultaneously and perform joint reasoning by combining text location information.

[0066] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0067] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0068] Figure 3 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0069] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 302 or a computer program loaded from storage unit 308 into RAM (Random Access Memory) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An I / O (Input / Output) interface 305 is also connected to the bus 304.

[0070] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0071] The computing unit 301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as image information extraction methods. For example, in some embodiments, the image information extraction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the aforementioned image information extraction method by any other suitable means (e.g., by means of firmware).

[0072] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0073] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0075] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0076] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0077] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0078] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0079] The various numerical designations such as "first," "second," etc., used in this disclosure are merely for ease of description and are not intended to limit the scope of the embodiments of this disclosure, nor do they indicate a sequential order.

[0080] At least one of the features described in this disclosure can also be described as one or more, and multiple features can be two, three, four or more, and this disclosure does not impose any limitations. In the embodiments of this disclosure, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no sequential order or size order among the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0081] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image information extraction method characterized by comprising: include: The optical character recognition model is compressed to obtain a lightweight optical character recognition model. At the edge device, the lightweight optical character recognition model is used to perform text recognition on the input image to obtain the text content and its location information in the image. Based on the results of the text recognition, determine whether to trigger cloud processing; When cloud processing is triggered, the input image and the text recognition result are sent to the cloud server; On a cloud server, a large language model is used to extract multimodal information from the input image and the text recognition results to obtain structured information.

2. The method of claim 1, wherein, The process of compressing the optical character recognition model includes: The processing includes pruning and quantization of the optical character recognition model.

3. The method of claim 2, wherein, The pruning process includes: The importance of each layer is evaluated based on the norm of the parameter matrix, and the sparsity rate is dynamically allocated. The dynamic allocation of the sparsity rate is adjusted according to the ratio of the number of pruned parameters to the total number of parameters in the model.

4. The method of claim 2, wherein, The quantization process includes: The quantization sensitivity is evaluated based on the condition number of the weight matrix and the Lipschitz constant of the activation matrix, and different quantization bits are assigned to different layers according to the evaluation results to achieve mixed-precision quantization.

5. The method of claim 1, wherein, The step of determining whether to trigger cloud processing based on the text recognition result includes: The confidence level of the text recognition result is compared with a preset threshold, or the result is based on the output of the task complexity evaluation module.

6. The method of claim 1, wherein, The large language model is a cross-modal large language model that can process image and text inputs simultaneously and perform joint reasoning by combining text location information.

7. An image information extraction apparatus characterized by comprising: include: The compression unit is used to perform model compression processing on the optical character recognition model to obtain a lightweight optical character recognition model. The recognition unit is used at the edge device to perform text recognition on the input image using the lightweight optical character recognition model to obtain the text content and its position information in the image; The judgment unit is used to determine whether to trigger cloud processing based on the result of the text recognition. The sending unit is used to send the input image and the text recognition result to the cloud server when cloud processing is triggered; The extraction unit is used on a cloud server to extract multimodal information from the input image and the text recognition results using a large language model, thereby obtaining structured information.

8. An electronic device, comprising: include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images