Information extraction method and related device

Abstract

The invention provides an information extraction method and a related device. The method comprises the following steps: determining data of each first text region in a first image; secondly, determining first feature data of each first text region according to the data of each first text region; then, determining data of a second text area according to the information extraction instruction; determining second feature data of the second text area according to the data of the second text area; each piece of first feature data is matched with the second feature data, and the confidence degree of each first text area and the second text area is determined; and determining the first text region with the confidence higher than a preset confidence threshold as a third text region, and extracting third text information of each third text region. According to the method, on the basis that a model does not need to be customized again, needed information can be extracted for any type of images, and the development cost is greatly reduced while the accuracy of the extracted information is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information extraction method and related devices. Background Art

[0002] In daily life, there is a large amount of image data. Generally, optical character recognition (OCR) is used to collect text information in graphic data, and deep learning models are used to structure the information. However, existing deep learning models are generally based on prior knowledge, that is, the model is trained on the basis of knowing which key information needs to be extracted from the image data to be extracted, and the information extraction scope of the model is limited. It can be considered that the information category that can be extracted by the model is a closed set.

[0003] In practice, there are a large number of open set tasks, that is, we do not know in advance what key information needs to be extracted from the image data to be extracted. At this time, the original model is used for information extraction, which is not effective for extracting information categories that do not exist in the closed set. Retraining different models for different image data will result in excessively high development costs. Summary of the invention

[0004] In view of this, the present application provides an information extraction method and related devices, which can extract the required information for any category of images by adding prompt text without re-customizing the model, thereby greatly reducing the development cost while ensuring the accuracy of the extracted information.

[0005] In a first aspect, an embodiment of the present application provides an information extraction method, the method comprising:

[0006] Determine first text information, first spatial position information, and first image information of each first text area in the first image;

[0007] Determine first feature data of each first text region according to the first text information, the first spatial position information and the first image information of each first text region;

[0008] Determine second text information, second spatial position information, and second image information of a second text area according to the information extraction instruction;

[0009] determining second feature data of the second text area according to the second text information, the second spatial position information and the second image information;

[0010] Matching each first feature data with the second feature data to determine the confidence level of each first text region and the second text region;

[0011] The first text regions whose confidence level is higher than a preset confidence threshold are determined as third text regions, and third text information of each third text region is extracted.

[0012] In a second aspect, an embodiment of the present application provides an information extraction device, the device comprising:

[0013] A first area unit, used to determine first text information, first spatial position information and first image information of each first text area in the first image;

[0014] A first feature unit, configured to determine first feature data of each first text region according to the first text information, the first spatial position information and the first image information of each first text region;

[0015] A second area unit, used to determine second text information, second spatial position information and second image information of a second text area according to the information extraction instruction;

[0016] A second feature unit, configured to determine second feature data of the second text area according to the second text information, the second spatial position information and the second image information;

[0017] an associating classification unit, used for matching each first feature data with the second feature data, and determining the confidence level of each first text region and the second text region;

[0018] The extraction unit is used to determine the first text regions whose confidence is higher than a preset confidence threshold as third text regions, and extract third text information from each third text region.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps of any method in the first aspect of the embodiment of the present application.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in any method of the first aspect of the embodiment of the present application.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0022] It can be seen that, through the above information extraction method and related devices, first, the first text information, first spatial position information and first image information of each first text area in the first image are determined; secondly, the first feature data of each first text area is determined according to the first text information, the first spatial position information and the first image information of each first text area; then, the second text information, the second spatial position information and the second image information of the second text area are determined according to the information extraction instruction; then, the second feature data of the second text area is determined according to the second text information, the second spatial position information and the second image information; then, each first feature data and the second feature data are matched to determine the confidence of each first text area and the second text area; finally, the first text area with a confidence higher than a preset confidence threshold is determined as a third text area, and the third text information of each third text area is extracted. The required information can be extracted for any category of images without re-customizing the model, which greatly reduces the development cost while ensuring the accuracy of the extracted information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0024] Figure 1 A system architecture diagram of an information extraction method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram for comparing a traditional bill image and a new bill image provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a first image provided in an embodiment of the present application;

[0027] Figure 4 A flowchart of an information extraction method provided in an embodiment of the present application;

[0028] Figure 5A flowchart of another information extraction method provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0030] Figure 7 A block diagram of the functional units of an information extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0033] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship. The "plurality" appearing in the embodiments of the present application refers to two or more.

[0034] In the embodiments of the present application, "at least one item" or similar expressions refer to any combination of these items, including any combination of single items or plural items, and refer to one or more, and multiple refers to two or more. For example, at least one item of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.

[0035] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] The following is an explanation of the relevant contents, concepts, meanings, technical issues, technical solutions, beneficial effects, etc. involved in the embodiments of the present application.

[0038] 1. Related terms

[0039] Optical Character Recognition (OCR): The core principle of OCR technology is to convert the image text on paper documents into computer-processable text through a series of complex algorithms and steps. This process usually includes steps such as image preprocessing, character segmentation, feature extraction and text recognition. Image preprocessing: At this stage, OCR technology will perform a series of processing on the scanned paper document image to improve the accuracy of subsequent character recognition. These processing operations include denoising, binarization, tilt correction, etc., which are aimed at eliminating interference factors in the image and making the text clearer and easier to recognize. Character segmentation: After the preprocessing is completed, OCR technology will segment the text in the image. Since the text on paper documents is usually arranged continuously, they need to be segmented into individual characters or words for subsequent recognition. This step requires the use of image processing technology to identify the boundaries between the characters and accurately segment them. Feature extraction: After the character segmentation is completed, OCR technology will extract the feature information of each character or word. This feature information includes the shape, size, strokes, etc. of the character, which is an important basis for subsequent text recognition. By extracting these feature information, OCR technology can distinguish different characters or words and prepare for subsequent recognition. Text recognition: After feature extraction is completed, OCR technology will match the extracted feature information with the known character library to obtain the recognition result. This process requires the use of pattern recognition algorithms to classify and identify feature information. Finally, OCR technology will output the recognized text according to the layout and format of the original document to form an editable text format.

[0040] Structured data: refers to data stored in a clear format and rules, which can be stored and managed through tables, databases or other programmable data models. Structured data has clear fields and attributes, and each field has a specific data type and value range. This type of data is easy to organize, store and analyze, and can be effectively processed and mined through various algorithms and tools.

[0041] 2. Application scenarios and electronic devices

[0042] 1. Application scenarios

[0043] The embodiments of the present application can be applied to the following application scenarios, including but not limited to converting text information in images of cards, certificates, and bills into structured data for storage or use. No specific limitation is made here.

[0044] 2. Electronic equipment

[0045] The electronic device in the embodiment of the present application may be a portable electronic device that also includes other functions such as a personal digital assistant and / or a music player function, such as a mobile phone, a tablet computer, a wearable electronic device with a wireless communication function (such as a smart watch), etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with an IOS system, an Android system, a Microsoft system, or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer (Laptop), etc. It should also be understood that in some other embodiments, the above-mentioned electronic device may not be a portable electronic device, but a desktop computer, a server, etc.

[0046] 3. Example

[0047] See also Figure 1 , Figure 1 This is a system architecture diagram of an information prompt method provided in an embodiment of the present application. Figure 1 As shown, it includes an image acquisition module 110 , an image recognition module 120 , a prompt text module 130 , a feature calculation module 140 , an association classification module 150 and a post-processing module 160 .

[0048] Among them, the image acquisition module 110 is used to acquire the first image and transmit the first image to the image recognition module 120. The first image is an image that needs to be used for information extraction, including but not limited to bill images, card images and other images, which are not specifically limited here. The first image may include multiple text areas. For the sake of easy distinction, each text area in the first image is referred to as a first text area.

[0049] Among them, the image recognition module 120 can have a built-in OCR algorithm for identifying the first image from the image acquisition module 110 and determining the information characteristics of each first text area in the first image. Specifically, the image recognition module 120 can include an OCR model. The OCR model can preprocess the image. The preprocessing can include denoising, binarization, rotation correction, etc., which are used to improve the image quality and readability of the text. The OCR model may include a convolutional neural network structure for detecting areas containing text from the image. The OCR model may also include recursive neural networks, long short-term memory networks and other structures for converting the image in the detected text area into editable text. In this way, the information characteristics of each first text area in the first image can be determined.

[0050] The prompt text module 130 is used to determine the second text information according to the information extraction instruction, construct a second text area, and determine the information features of the second text area. It should be noted that the dimension of the information features of the second text area is consistent with the dimension of the information features of each of the first text areas, and the second text area is constructed to correspond to each first text area, so as to maintain consistency in subsequent feature processing.

[0051] Among them, the feature calculation module 140 is used to perform cross-module feature calculation on the information features of each first text area and the information features of the second text area, and transmit the calculated features to the associated classification module 150. Specifically, the feature calculation module 140 can adopt a transformer architecture to realize processing between features.

[0052] The associated classification module 150 is used to determine whether the second text region matches each first text region based on the calculated features, and classify them into two situations: matching and mismatching. Specifically, the associated classification module 150 can be a classifier that performs classification based on confidence.

[0053] Among them, the post-processing module 160 is used to determine the first text area that needs to be extracted information according to the matching situation obtained by the associated classification module 150, and extract the text information of this part of the first text area to facilitate subsequent data structuring. For the sake of distinction, the first text area that needs to be extracted information is referred to as the third text area, and the text information of the third text area is referred to as the third text information. Specifically, the post-processing module 160 can adopt an OCR detection algorithm, which will not be elaborated here.

[0054] In a possible embodiment, a prompt text module 130 may be added to the basic architecture of the layoutLM series models to obtain the system architecture of the embodiment of the present application. The layoutLM series models may include LayoutLM v1, LayoutLM v2, LayoutLM v3, LayoutXLM, etc., which will not be elaborated here.

[0055] Through the above system architecture, by adding a second text, the required information can be extracted for any category of images without re-customizing the model, thereby greatly reducing the development cost while ensuring the accuracy of the extracted information.

[0056] It should be noted that the above description is a summary of each functional module, and the specific implementation can refer to the subsequent description.

[0057] 3. A method for information extraction

[0058] In the application scenario of information extraction, the existing method is generally to conduct special model training according to needs, that is, to determine in advance which categories of text information in the image can be extracted. For example, for information extraction of outpatient invoices, it is necessary to first determine the fields in the outpatient invoice. After the model is trained, the model has a good information extraction effect for the outpatient invoice. However, in daily practice, there may be many new fields in the outpatient invoice, such as handwritten notes, etc. These parts are not fixed and difficult to determine in advance. If the old model is used for information extraction, the extraction effect will be poor, and retraining the model requires a lot of development costs, and each time a new field appears, it needs to be retrained, which is a waste of data resources.

[0059] It should be noted that the above model may be a named entity recognition (NER) model, and the NER model may include a distributed vector input, a context encoding structure, and a label decoding structure.

[0060] Among them, distributed vector input can include word-level vector input, character-level vector input, etc., among which, word-level vector input can be realized by adopting continuous bag-of-words model structure, skip-gram model, etc.; among them, character-level vector input can be realized by adopting convolutional neural network CNN to extract character-level representation of words, and then connecting the character representation vector with the word embedding, and then sending it to the context encoder of recurrent neural network. In another implementation method, character-level vector input can be realized by extracting character-level representation using bidirectional long short-term memory network.

[0061] Among them, the context coding structure is mainly implemented by convolutional neural network, recurrent neural network, recursive neural network and transformer. For convolutional neural network, it includes embedding layer, convolutional coding module, maximum pooling layer, decoding layer, etc.; for recurrent neural network, it includes bidirectional recurrent network, etc.; for recursive neural network, it includes algorithms for calculating the hidden state vector of each node and classifying each node according to these hidden vectors, etc.; for transformer, it includes stacked self-attention and point-by-point fully connected layers to construct the basic blocks of encoder and decoder.

[0062] Among them, the label decoding structure can be implemented by using a multi-layer perceptron + softmax layer, a conditional random field (CRF), a recursive neural network and a pointer network, etc., which will not be elaborated here.

[0063] For example, if Figure 2 As shown, Figure 2 A comparison diagram of a traditional bill image and a new bill image provided in the embodiment of the present application shows that the categories included in the traditional bill image are "name", "payment fee", and "drug name", while the new bill image is based on the traditional bill image and adds handwritten instructions for taking each drug. The method of taking "drug a" is "once in the morning and once in the evening, two tablets each time", and the method of taking "drug b" is "three times a day, after meals, 30 ml each time". Since the categories of traditional bill images are fixed, when model training is performed based on the categories included in the traditional bill image and information extraction is performed using the trained model, the new bill image with handwritten text does not have a good extraction effect. In this scenario, it is necessary to develop an information extraction method that can extract the required information for any category of image without re-customizing the model. It should be noted that the reason why the above-mentioned new bill image is "new" is that handwritten text information is added on the basis of the traditional bill image, or other printed text information is added on the basis of the traditional bill image, which is not specifically limited here.

[0064] To solve the above problems, the embodiments of the present application provide an information extraction method and related devices, which can extract the required information for any category of images by adding prompt text without re-customizing the model, thereby greatly reducing the development cost while ensuring the accuracy of the extracted information.

[0065] The technical solutions, beneficial effects, concepts, etc. involved in the embodiments of the present application are described in detail below.

[0066] In the embodiment of the present application, the object that needs to be extracted is the first image, and the information that needs to be extracted is text information. The first image can be an image of a bill containing structured data, such as an outpatient invoice, an inpatient invoice, a shopping receipt, an airline itinerary, a value-added tax invoice, etc. The so-called structured data means that there are fields and corresponding text information, similar to a key-value pair. For example, the name "axx" corresponding to the "name" field is a structured data. There may be one or more first text areas in the above-mentioned first image, each first text area corresponds to a first text area data, and each first text area data includes first text information, first spatial position information and first image information.

[0067] Among them, the first spatial position information of each first text area can be obtained through the OCR detection algorithm, such as the length, width and position of each first text area in the first image, etc., which can be specifically the vertex coordinates of the circumscribed rectangular box of each first text area; in the case of no circumscribed rectangular box, it is possible to determine which characters belong to the same text based on the distance between the characters, so that after determining each text, a virtual circumscribed rectangular box of each text can be determined, and the fixed-point coordinates of each virtual circumscribed rectangular box can be determined. It should be noted that the virtual circumscribed rectangular box is a virtual concept and is not displayed in the actual first image. Its purpose is to facilitate the determination of the first spatial position information of each first text area.

[0068] The first image may be divided into image information of each first text area according to the first spatial position information of each first text area. For easy distinction, the image information of each first text area is referred to as first image information of each first text area.

[0069] The image information of each first text region may be recognized by an OCR recognition algorithm to obtain the first text information of each first text region, which will not be elaborated herein.

[0070] For example, the first image is an airline itinerary, such as Figure 3 As shown, Figure 3 A schematic diagram of a first image provided for an embodiment of the present application shows that there are a total of 9 first text areas. From top to bottom and from left to right, the first first text area "xx Airlines" and the second first text area "xx-y Place" can be determined. The third first text area "Business Class 2024.07.23", the fourth first text area "Passenger Name A", the fifth first text area "Ticket Number 12345xxx", the sixth first text area "Total Amount 1810CNY", the seventh first text area "Air Ticket Price 1690CNY", the eighth first text area "Fuel Surcharge 70CNY", and the ninth first text area "Airport Construction Fee 50CNY".

[0071] It can be seen that by determining the first text information, the first spatial position information and the first image information of each first text area in the first image, a data basis can be provided for determining the multimodal features of each first text area, thereby facilitating the subsequent determination of the matching status of the second text area with each first text area.

[0072] The embodiment of the present application introduces the concept of first feature data, which is a multimodal feature. The first feature data can be determined based on the first text information, the first spatial position information and the first image information of each first text area, including the first text feature of each first text area and the first image feature of each first text area.

[0073] Among them, the first text feature of each first text area can be determined according to the first spatial position information of each first text area and the first text information of each first text area, so that the first text feature includes the position information, size information, etc. of the corresponding first text area.

[0074] Specifically, each first text information can be processed with a unified sequence length, and each first text information can be mapped to a unified sequence length. Texts with insufficient length can be padded. The padding processing here can be to fill the sequence length that does not reach the above-mentioned unified sequence length with empty characters to the above-mentioned unified sequence length. At the same time, the bounding box of each first text area is introduced into the spatial position information of each first text area through normalization processing, and merged with the first text information. Finally, the first text information containing the spatial position information of the corresponding first text area and the unified sequence length is input into a pre-trained text feature extraction model, and each first text feature is obtained according to the output of the pre-trained text feature extraction model, wherein the above-mentioned text feature extraction model can include an embedding layer to convert the above-mentioned first text information into a high-dimensional vector, which will not be elaborated here.

[0075] In a possible embodiment, the first image information of each first text region can be input into a pre-trained image feature extraction model, and the first image features of each first text region can be obtained according to the output of the pre-trained image feature extraction model. Specifically, the image feature extraction model can adopt a Faster R-CNN model, etc., which will not be described in detail here.

[0076] In a possible embodiment, each first image information may be linearly mapped to obtain each first image feature. Specifically, each first image information may be scaled to a uniform size, and then each first image information may be divided into blocks of a fixed size, and each first image feature may be obtained by flattening and serializing through linear mapping, thereby reducing computational overhead.

[0077] It can be seen that in this way, the multimodal features of each first text region can be obtained, which is convenient for subsequently determining the matching status of each first text region with the second text region.

[0078] In order to implement a general information extraction method, the embodiment of the present application introduces the concept of second text area data. The second text area corresponds to the second text area data, and the second text area data includes second text information, second spatial position information and second image information.

[0079] Among them, the second text area data corresponds to each first text area in format. First, the second text information can be determined based on the information extraction instruction. The information extraction instruction can be an instruction input by the user. The second text information can be a question, a description, a task description, or even a part of the conversation history record. It can contain keywords, constraints, example input and output, such as "how to take medicine", "what other fees are there besides airport construction fees", etc., which are not specifically limited here. It should be noted that the second text information in the embodiment of the present application is intended for information extraction rather than information generation. For example, the second text information can be "name", "form number" and other texts representing the keywords of the information to be extracted, which is equivalent to asking a question about the first image, that is, whether the result corresponding to a certain second text information exists in the first image.

[0080] Since unified multimodal feature modeling is required, the image information of the first image can be used as the second image information, and the second spatial position information can be determined based on the image information of the first image. For example, the circumscribed edge rectangle of the entire first image can be used as the second spatial position information. In this way, the second text area data can be constructed based on the second text information, the second image information and the second spatial position information, so that subsequent features can be aligned in dimension.

[0081] In a possible embodiment, before determining the second text area data according to the information extraction instruction, the first image can also be identified, and the type of the first image and the text information data of the first image can be determined. Here, the type of the first image can include air travel bills, outpatient bills, etc., which are not described in detail here. The text information data here can include all the text information of the first image. Then, the first extraction field corresponding to the type of the first image is determined. For example, the first extraction field corresponding to the "ID card" type includes "name", "gender", "birth", "address", and "citizen ID card number". The first extraction field is a field inherent to this type. Then, the fifth text information corresponding to each common extraction field in the text information data can be determined, that is, the specific name corresponding to the "name" field and the specific gender corresponding to the "gender" field. It is not described in detail here. Then, the sixth text information other than the fifth text information in the text information data is determined. For example, the ID card is a lost ID card. After being picked up by someone, the text "picked up at address xx" is written with an erasable pen. At this time, "picked up at address xx" is the sixth text information. Finally, the candidate prompt text can be determined according to the semantics of the sixth text information, and the candidate prompt text is displayed. For example, the candidate prompt texts may be “where the ID card was found”, “the address found”, “the address lost”, etc., which will not be elaborated here.

[0082] It can be seen that this can facilitate the user to select appropriate second text information, prevent the second text information determined by the user from being inaccurate, and improve the user experience.

[0083] The second feature data may be determined according to the second text region data, and the feature dimension of the second feature data is the same as the feature dimension of the first feature data.

[0084] The second text feature of the second text area may be determined according to the second spatial position information and the second text information, so that the second text feature includes the position information, size information, etc. of the second text area.

[0085] Specifically, the second text information can be processed with a unified sequence length, the second text information can be mapped to a unified sequence length, and the text with insufficient length can be padded. At the same time, the bounding box of the second text area is introduced into the spatial position information of the second text area through normalization processing and merged with the second text information. Finally, the second text information containing the spatial position information of the second text area and the unified sequence length is input into a pre-trained text feature extraction model, and each second text feature is obtained according to the output of the pre-trained text feature extraction model.

[0086] In a possible embodiment, the second image information may be input into a pre-trained image feature extraction model, and the second image feature may be obtained according to an output of the pre-trained image feature extraction model.

[0087] In a possible embodiment, the second image information can also be linearly mapped to obtain the second image features. Specifically, the second image information can be scaled to a uniform size and then divided into blocks of a fixed size. The second image features are obtained by flattening and serializing the blocks through linear mapping. The second image information here is the image information of the first image, which can reduce computational overhead.

[0088] It can be seen that in this way, the multimodal features of the second text region can be obtained, which facilitates the subsequent determination of the matching situation with each first text region.

[0089] In order to extract text information that complies with the information extraction instruction, the embodiment of the present application introduces the concept of matching between each first text area and the second text area. If the first text area matches the second text area, it means that the text information in the first text area complies with the information extraction instruction. That is, if the second text information is regarded as a problem, if the first text area matches the second text area, it means that the first text information in the first text area can be regarded as a solution to the above problem.

[0090] The first feature data and the second feature data may be fused to obtain fused feature data. For example, the second text region and the first text region are regarded as equal region nodes, and together constitute a new graph structure.

[0091] Among them, the fused feature data can be processed by the self-attention mechanism to obtain the first self-attention feature of each first text area. Specifically, it can be processed by the Transformer module. The Transformer module has a multi-layer structure, and each layer of the structure is mainly composed of multi-head self-attention and position-wise fully connected feedforward neural network. After the Transformer structure calculation, a better feature vector of the first text area (including image features, semantic features and information) is obtained. After multi-layer calculation, a 768-dimensional multimodal feature is generated for each first text area, and the 768-dimensional multimodal feature is the above-mentioned first self-attention feature.

[0092] Among them, the association between the second text information and the first text information can be strengthened through the cross-attention mechanism, the second text information and the first text information of each first text area can be combined to obtain the fourth text information, the text semantic features of the fourth text information are determined, and the text semantic features and the first self-attention features of each first text area are processed through the cross-attention mechanism to obtain the global attention features of each first text area. Specifically, all the second text information and the first text information of each first text area can be combined to obtain the fourth text information ((prompt_1||prompt_2||prompt_3||……||prompt_n)||text), and the text semantic features of the fourth text information can be extracted through the pre-trained semantic feature extraction model, and then the text semantic features and each first self-attention feature are cross-attention calculated, and the result of the cross-attention calculation is called the global attention feature. The global attention feature integrates the features of multiple dimensions of image, semantics, and space, and the features between different text areas are cross-calculated. The global feature of each text area integrates the information features of the entire image, and the correlation between the first text area and the second text area is strengthened by the cross-attention calculation.

[0093] Among them, the confidence of each first text area and the second text area can be determined according to the global attention feature of each first text area. Specifically, determining whether to match is a binary classification problem, that is, if the confidence between a first text area and a second text area is higher than a preset confidence threshold, then the first text information of the first text area is the answer to the second text information of the second text area. Each second text area can match with multiple first text areas, which means that there are multiple answers to the question. The global attention feature of each first text area can be input into the association classification module, and the matching of each first text area and the second text area is determined according to the output of the association classification module.

[0094] It can be understood that each global attention feature actually contains the semantic information, position information, image information of the first text area, the semantic information, position information, image information of the second text area, and the global feature information, so that the feature fusion between different text area nodes can be completed in the process of calculation iteration. Let each first text area be node i, and the first feature data of each first text area be n_i, that is, n_i not only has the features of the current node i, but also contains the global text area features, so that the feature results of each node can represent the layout information of the whole picture.

[0095] The matching situation includes matching or mismatching.

[0096] The first text region with a confidence level higher than a preset confidence threshold may be determined as a third text region, and the third text information of each third text region may be extracted. The third text information of each third text region may be stored in a structured format.

[0097] It can be seen that, through the above information extraction method and related devices, first, the first text information, first spatial position information and first image information of each first text area in the first image are determined; secondly, the first feature data of each first text area is determined according to the first text information, the first spatial position information and the first image information of each first text area; then, the second text information, the second spatial position information and the second image information of the second text area are determined according to the information extraction instruction; then, the second feature data of the second text area is determined according to the second text information, the second spatial position information and the second image information; then, each first feature data and the second feature data are matched to determine the confidence of each first text area and the second text area; finally, the first text area with a confidence higher than a preset confidence threshold is determined as a third text area, and the third text information of each third text area is extracted. The required information can be extracted for any category of images without re-customizing the model, which greatly reduces the development cost while ensuring the accuracy of the extracted information.

[0098] In combination with the above content, an information extraction method according to an embodiment of the present application is introduced as an example below.

[0099] See also Figure 4 , Figure 4 A flowchart of an information extraction method provided in an embodiment of the present application specifically includes the following steps:

[0100] Step 401: determine first text information, first spatial position information, and first image information of each first text area in a first image.

[0101] Among them, the first image can be recognized by OCR technology to obtain the first spatial position information of each first text area, and then, based on the first spatial position information, the image information of the first image is divided into image information of each first text area, and then, the image information of each first text area is recognized to obtain the first text information of each first text area.

[0102] It can be seen that this can provide data support for the subsequent determination of the first characteristic data.

[0103] Step 402: Determine first feature data of each first text region according to the first text information, the first spatial position information, and the first image information of each first text region.

[0104] Among them, the first feature data includes the first text feature of each first text area and the first image feature of each first text area. The first text feature of each first text area can be determined based on the first spatial position information of each first text area and the first text information of each first text area; the first image information of each first text area is linearly mapped to determine the first image feature of each first text area.

[0105] It can be seen that accurate multimodal features can be determined in this way, providing data reference for the subsequent matching of each first text region with the second text region.

[0106] Step 403: Determine second text information, second spatial position information, and second image information of the second text area according to the information extraction instruction.

[0107] Among them, the information extraction instruction is identified to obtain the second text information; the image information of the first image is used as the second image information; and the second spatial position information is determined according to the second image information.

[0108] Similarly, this can provide data support for the subsequent determination of the second characteristic data.

[0109] Step 404: determine second feature data of the second text area according to the second text information, the second spatial position information, and the second image information.

[0110] The feature dimension of the second feature data is the same as the feature dimension of the first feature data. The second feature data includes a second text feature of the second text area and a second image feature of the second text area. The second text feature of the second text area can be determined based on the second spatial position information and the second text information; the second image information is linearly mapped to determine the second image feature of the second text area.

[0111] It can be seen that accurate multimodal features can be determined in this way, providing data reference for the subsequent matching of each first text region with the second text region.

[0112] Step 405: Match each first feature data with the second feature data to determine the confidence level of each first text region and the second text region.

[0113] Among them, the first feature data and the second feature data can be fused to obtain fused feature data; the fused feature data can be processed by a self-attention mechanism to obtain a first self-attention feature of each first text area; the second text information and the first text information of each first text area are combined to obtain fourth text information; the text semantic features of the fourth text information are determined; the text semantic features of the fourth text information and the first self-attention features of each first text area are processed by a cross-attention mechanism to obtain a global attention feature of each first text area; and the confidence of each first text area and the second text area is determined according to the global attention feature of each first text area.

[0114] Step 406: determine the first text regions whose confidence levels are higher than a preset confidence threshold as third text regions, and extract third text information from each third text region.

[0115] The third text information of each third text area is stored in a structured format.

[0116] It can be seen that, through the above information extraction method and related devices, first, the first text information, first spatial position information and first image information of each first text area in the first image are determined; secondly, the first feature data of each first text area is determined according to the first text information, the first spatial position information and the first image information of each first text area; then, the second text information, the second spatial position information and the second image information of the second text area are determined according to the information extraction instruction; then, the second feature data of the second text area is determined according to the second text information, the second spatial position information and the second image information; then, each first feature data and the second feature data are matched to determine the confidence of each first text area and the second text area; finally, the first text area with a confidence higher than a preset confidence threshold is determined as a third text area, and the third text information of each third text area is extracted. The required information can be extracted for any category of images without re-customizing the model, which greatly reduces the development cost while ensuring the accuracy of the extracted information.

[0117] For the parts not described in detail above, please refer to the detailed description of the above specification and will not be repeated here.

[0118] In combination with the above content, another information extraction method of an embodiment of the present application is introduced as an example below.

[0119] See also Figure 5 , Figure 5A flowchart of another information extraction method provided in an embodiment of the present application specifically includes the following steps:

[0120] Step 501: determine first text region data of each first text region in a first image.

[0121] Step 502: Determine first feature data according to the first text region data of each first text region.

[0122] Step 503: Identify the first image, and determine the type of the first image and text information data of the first image.

[0123] Step 504: determine a first extraction field corresponding to the type of the first image.

[0124] For example, the first extraction fields corresponding to the "ID card" type include "name", "gender", "birth", "address", and "citizen ID number", and the first extraction fields are fields inherent to this type.

[0125] Step 505: Determine fifth text information corresponding to each first extracted field in the text information data.

[0126] That is, the specific name corresponding to the "Name" field and the specific gender corresponding to the "Gender" field are not repeated here.

[0127] Step 506: Determine sixth text information other than the fifth text information in the text information data.

[0128] For example, the ID card is a lost ID card. After being picked up by someone, the words "picked up at address xx" are written on it with an erasable pen. At this time, "picked up at address xx" is the sixth text information.

[0129] Step 507: determine candidate prompt texts according to the semantics of the sixth text information, and display the candidate prompt texts.

[0130] For example, the candidate prompt texts may be “where the ID card was found”, “the address found”, “the address lost”, etc., which will not be elaborated here.

[0131] It can be seen that this can make it easier for users to select appropriate prompt text, prevent the prompt text determined by the user from being inaccurate, and improve the user experience.

[0132] Step 508: Determine the second text region data according to the information extraction instruction.

[0133] Step 509: determine second feature data according to the second text region data.

[0134] Step 510: Match each first feature data with the second feature data to determine the confidence level of each first text region and the second text region.

[0135] Step 511: determine the first text regions whose confidence levels are higher than a preset confidence threshold as third text regions, and extract third text information from each third text region.

[0136] It can be seen that, through the above information extraction method and related devices, first, the first text information, first spatial position information and first image information of each first text area in the first image are determined; secondly, the first feature data of each first text area is determined according to the first text information, the first spatial position information and the first image information of each first text area; then, the second text information, the second spatial position information and the second image information of the second text area are determined according to the information extraction instruction; then, the second feature data of the second text area is determined according to the second text information, the second spatial position information and the second image information; then, each first feature data and the second feature data are matched to determine the confidence of each first text area and the second text area; finally, the first text area with a confidence higher than a preset confidence threshold is determined as a third text area, and the third text information of each third text area is extracted. The required information can be extracted for any category of images without re-customizing the model, which greatly reduces the development cost while ensuring the accuracy of the extracted information.

[0137] For the parts not described in detail above, please refer to the description in the above manual and will not be repeated here.

[0138] IV. Exemplary description of an electronic device

[0139] Combine the following Figure 6 An electronic device in an embodiment of the present application is described. Figure 6 6 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The electronic device 600 includes a processor 601 , a memory 602 , and a communication bus 603 for connecting the processor 601 and the memory 602 .

[0140] In some possible implementations, the memory 602 includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or portable read-only memory (CD-ROM), and the memory 602 is used to store the program code executed by the electronic device 600 and the transmitted data.

[0141] In some possible implementations, the electronic device 600 further includes a communication interface for receiving and sending data.

[0142] In some possible implementations, the processor 601 may be one or more central processing units (CPUs). When the processor 601 is a central processing unit (CPU), the central processing unit (CPU) may be a single-core central processing unit (CPU) or a multi-core central processing unit (CPU).

[0143] In some possible implementations, the processor 601 may be a baseband chip, a chip, a central processing unit (CPU), a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof.

[0144] In a specific implementation, the processor 601 in the electronic device 600 is used to execute the program instructions 621 stored in the memory 602 to perform the following operations:

[0145] determining first text region data for each first text region in the first image;

[0146] Determine first text information, first spatial position information, and first image information of each first text area in the first image;

[0147] Determine first feature data of each first text region according to the first text information, the first spatial position information and the first image information of each first text region;

[0148] Determine second text information, second spatial position information, and second image information of a second text area according to the information extraction instruction;

[0149] determining second feature data of the second text area according to the second text information, the second spatial position information and the second image information;

[0150] Matching each first feature data with the second feature data to determine the confidence level of each first text region and the second text region;

[0151] The first text regions whose confidence level is higher than a preset confidence threshold are determined as third text regions, and third text information of each third text region is extracted.

[0152] It can be seen that, through the above information extraction method and related devices, first, the first text information, first spatial position information and first image information of each first text area in the first image are determined; secondly, the first feature data of each first text area is determined according to the first text information, the first spatial position information and the first image information of each first text area; then, the second text information, the second spatial position information and the second image information of the second text area are determined according to the information extraction instruction; then, the second feature data of the second text area is determined according to the second text information, the second spatial position information and the second image information; then, each first feature data and the second feature data are matched to determine the confidence of each first text area and the second text area; finally, the first text area with a confidence higher than a preset confidence threshold is determined as a third text area, and the third text information of each third text area is extracted. The required information can be extracted for any category of images without re-customizing the model, which greatly reduces the development cost while ensuring the accuracy of the extracted information.

[0153] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above, and the electronic device 600 can be used to execute the above method embodiment of the present application, which will not be described in detail.

[0154] 5. Exemplary description of an information extraction device

[0155] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0156] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0157] In the case of dividing each functional module into corresponding functional modules, Figure 7 This is a block diagram of the functional units of an information extraction device provided in an embodiment of the present application. The information extraction device 700 includes:

[0158] A first area unit 710, configured to determine first text information, first spatial position information, and first image information of each first text area in the first image;

[0159] A first feature unit 720, configured to determine first feature data of each first text region according to the first text information, the first spatial position information, and the first image information of each first text region;

[0160] The second area unit 730 is used to determine the second text information, the second spatial position information and the second image information of the second text area according to the information extraction instruction;

[0161] A second feature unit 740, configured to determine second feature data of the second text area according to the second text information, the second spatial position information and the second image information;

[0162] An associated classification unit 750, configured to match each first feature data with the second feature data, and determine the confidence level of each first text region and the second text region;

[0163] The extraction unit 760 is configured to determine the first text regions whose confidence levels are higher than a preset confidence threshold as third text regions, and extract third text information from each third text region.

[0164] It can be seen that, through the above information extraction method and related devices, first, the first text information, first spatial position information and first image information of each first text area in the first image are determined; secondly, the first feature data of each first text area is determined according to the first text information, the first spatial position information and the first image information of each first text area; then, the second text information, the second spatial position information and the second image information of the second text area are determined according to the information extraction instruction; then, the second feature data of the second text area is determined according to the second text information, the second spatial position information and the second image information; then, each first feature data and the second feature data are matched to determine the confidence of each first text area and the second text area; finally, the first text area with a confidence higher than a preset confidence threshold is determined as a third text area, and the third text information of each third text area is extracted. The required information can be extracted for any category of images without re-customizing the model, which greatly reduces the development cost while ensuring the accuracy of the extracted information.

[0165] It should be noted that the specific implementation of each operation can adopt the corresponding description of the method embodiment shown above, and the information extraction device 700 can be used to execute the above method embodiment of the present application, which will not be repeated here.

[0166] 6. Other examples

[0167] An embodiment of the present application also provides a chip, including a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiment.

[0168] An embodiment of the present application also provides a chip module, including a transceiver component and a chip, the chip including a processor, a memory and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps described in the above method embodiment.

[0169] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiment, and the above computer includes an electronic device.

[0170] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0171] It should be noted that, for the above-mentioned various embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions, steps, modules or units involved are not necessarily required by the embodiments of the present application.

[0172] In the above embodiments, the embodiments of the present application have different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0173] The steps of the method or algorithm described in the embodiments of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, mobile hard disks, read-only compact disks (CD-ROMs) or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device or a management device. Of course, the processor and the storage medium can also be present in a terminal device or a management device as discrete components.

[0174] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0175] The modules / units included in the devices and products described in the above embodiments may be software modules / units or hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for the devices and products applied to or integrated in the chip, the modules / units included therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, which run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits; for the devices and products applied to or integrated in the chip module, the modules / units included therein may all be implemented in the form of hardware such as circuits, and different modules / units may be located in the same component (such as a chip, circuit module, etc.) or in different components of the chip module, or at least some of the modules / units may be implemented in the form of software programs. The software programs run on the processor integrated inside the chip, and the remaining (if any) modules / units may be implemented in the form of hardware such as circuits. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0176] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above description is only the specific implementation method of the embodiments of the present application and is not intended to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. An information extraction method, characterized in that: The method comprises: Determine first text information, first spatial position information, and first image information of each first text area in the first image; Determine first feature data of each first text region according to the first text information, the first spatial position information and the first image information of each first text region; Determine second text information, second spatial position information, and second image information of a second text area according to the information extraction instruction; determining second feature data of the second text area according to the second text information, the second spatial position information and the second image information; Matching each first feature data with the second feature data to determine the confidence level of each first text region and the second text region; The first text regions whose confidence level is higher than a preset confidence threshold are determined as third text regions, and third text information of each third text region is extracted.

2. The method according to claim 1, characterized in that The determining of the first text information, the first spatial position information and the first image information of each first text area in the first image includes: Recognize the first image to obtain the first spatial position information of each first text area; Dividing the image information of the first image into image information of each first text area according to the first spatial position information; The image information of each first text region is identified to obtain the first text information of each first text region.

3. The method according to claim 2, characterized in that The determining the first feature data of each first text region according to the first text information, the first spatial position information and the first image information of each first text region includes: Determining a first text feature of each first text region according to the first spatial position information of each first text region and the first text information of each first text region; The first image information of each first text region is linearly mapped to determine a first image feature of each first text region.

4. The method according to claim 2, characterized in that: The determining the second text information, the second spatial position information and the second image information of the second text area according to the information extraction instruction includes: identifying the information extraction instruction to obtain the second text information; using the image information of the first image as the second image information; The second spatial position information is determined according to the second image information.

5. The method according to claim 4, characterized in that The determining the second feature data of the second text area according to the second text information, the second spatial position information and the second image information includes: determining a second text feature of the second text area according to the second spatial position information and the second text information; The second image information is linearly mapped to determine a second image feature of the second text area.

6. The method according to claim 4, characterized in that The matching of each first feature data with the second feature data to determine the confidence level of each first text region and the second text region includes: Fusing each of the first feature data and the second feature data to obtain fused feature data; Processing the fused feature data through a self-attention mechanism to obtain a first self-attention feature of each first text region; merging the second text information and the first text information of each first text area into fourth text information; Determining text semantic features of the fourth text information; Processing the text semantic features of the fourth text information and the first self-attention features of each first text region through a cross-attention mechanism to obtain a global attention feature of each first text region; The confidence of each of the first text regions and the second text region is determined according to the global attention feature of each of the first text regions.

7. The method according to claim 1, characterized in that Before determining the second text information, the second spatial position information and the second image information of the second text area according to the information extraction instruction, the method further includes: Identify the first image, and obtain the type of the first image and text information data of the first image; Determine a first extraction field corresponding to the type of the first image; Determine fifth text information corresponding to each first extracted field in the text information data; determining sixth text information other than the first text information in the text information data; Determine candidate prompt texts according to the semantics of the sixth text information, and display the candidate prompt texts.

8. An information extraction device, characterized in that: The device comprises: A first area unit, used to determine first text information, first spatial position information and first image information of each first text area in the first image; A first feature unit, configured to determine first feature data of each first text region according to the first text information, the first spatial position information and the first image information of each first text region; A second area unit, used to determine second text information, second spatial position information and second image information of a second text area according to the information extraction instruction; A second feature unit, configured to determine second feature data of the second text area according to the second text information, the second spatial position information and the second image information; an associating classification unit, used for matching each first feature data with the second feature data, and determining the confidence level of each first text region and the second text region; The extraction unit is used to determine the first text regions whose confidence is higher than a preset confidence threshold as third text regions, and extract third text information from each third text region.

9. An electronic device, characterized in that: include: A processor, a memory, and one or more programs; the one or more programs are stored in the memory and are configured to be executed by the processor, the programs comprising instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute the method according to any one of claims 1-7.

Images