Text recognition method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a text recognition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-recognized image; inputting the to-be-recognized image into a visual network of the text recognition model, and extracting visual features of the to-be-recognized image; inputting the visual features into a language network of a text recognition model, and extracting semantic features based on the visual features through the language network; inputting the visual features and the semantic features into a fusion network of a text recognition model, and performing fusion processing on the visual features and the semantic features through a plurality of fusion units in the fusion network to obtain a fusion result; and generating a text recognition result for the to-be-recognized image based on the fusion result. By adopting the method, the text contained in the image can be accurately identified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a text recognition method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] Text recognition technology aims to accurately convert the text content in an image into an editable digital format. Due to the growing demand for digitization, text recognition technology is widely used in various industries.

[0003] Currently, when performing text recognition, the semantics are obtained from a pre-trained language model, and the text context in the image is used to correct recognition errors. However, this method consumes a lot of computing resources and time, which can cause system delays, especially in real-time application scenarios. Moreover, when the text in the image involves specific industry terminology or professional terms, a general pre-trained language model cannot provide accurate context correction, and a professional model needs to be trained separately, further increasing the complexity and cost of the technology.

[0004] Therefore, there is a problem in the prior art that text recognition of an image is not accurate. SUMMARY

[0005] Therefore, there is a problem in the prior art that text recognition of an image is not accurate.

[0006] A text recognition method, the method comprising:

[0007] obtaining an image to be recognized;

[0008] inputting the image to be recognized into a visual network of a text recognition model to extract visual features of the image to be recognized;

[0009] inputting the visual features into a language network of the text recognition model to extract semantic features based on the visual features through the language network;

[0010] inputting the visual features and the semantic features into a fusion network of the text recognition model, performing fusion processing on the visual features and the semantic features through multiple fusion units in the fusion network to obtain a fusion result;

[0011] generating a text recognition result for the image to be recognized based on the fusion result.

[0012] In one embodiment, inputting the image to be recognized into the visual network of the text recognition model to extract the visual features of the image to be recognized comprises:

[0013] obtaining a vector representation sequence of the image to be recognized;

[0014] The vector representation sequence is input to an encoder of the vision network, the vector representation sequence is encoded by the encoder, and an encoded sequence is obtained as the visual feature.

[0015] In one of the embodiments, the vector representation sequence of the image to be recognized is obtained, including:

[0016] The image to be recognized is divided into a plurality of image blocks, and the image block feature of any image block is mapped to the feature vector of the image block.

[0017] The learnable class embedding vector and the position embedding vector for the image to be recognized are obtained, and the vector representation sequence of the image to be recognized is generated based on the learnable class embedding vector, the position embedding vector, and the feature vector corresponding to each image block.

[0018] In one of the embodiments, the visual feature is input to a language network of the text recognition model, and the semantic feature is extracted by the language network based on the visual feature, including:

[0019] The character probability feature is determined according to the visual feature;

[0020] The character mask matrix is determined according to the character prediction order;

[0021] The character probability feature and the character mask matrix are input to the language network to obtain the semantic feature.

[0022] In one of the embodiments, the fusion network includes a first fusion unit, a second fusion unit, and a third fusion unit, the visual feature and the semantic feature are input to the fusion network of the text recognition model, and the visual feature and the semantic feature are fused by a plurality of fusion units in the fusion network to obtain a fusion result, including:

[0023] The visual feature and the semantic feature are input to the first fusion unit to obtain a first fusion result;

[0024] The visual feature, the semantic feature, and the first fusion result are input to the second fusion unit to obtain a second fusion result;

[0025] The visual feature, the semantic feature, and the second fusion result are input to the third fusion unit to obtain the fusion result.

[0026] A training method of a text recognition model, the method further includes:

[0027] Obtaining a training sample image;

[0028] input the training sample image into a visual network of the text recognition model to be trained, extract visual features of the training sample image through the visual network, and output a first character prediction result for the training sample image based on the visual features;

[0029] input the visual features of the training sample image into a language network of the text recognition model to be trained, extract semantic features based on the visual features of the training sample image through the language network, and output a second character prediction result for the training sample image based on the semantic features;

[0030] input the visual features and the semantic features of the training sample image into a fusion network of the text recognition model to be trained, obtain fusion results output by each fusion unit in the fusion network, and generate a text recognition result corresponding to each fusion unit based on the fusion results output by each fusion unit;

[0031] determine visual network loss information according to the character label corresponding to the training sample image and the first character prediction result corresponding to the training sample image, determine language network loss information according to the character label corresponding to the training sample image and the second character prediction result corresponding to the training sample image, and determine fusion network loss information according to the character label corresponding to the training sample image and the text recognition result corresponding to each fusion unit;

[0032] determine total loss information according to the visual network loss information, the language network loss information, and the fusion network loss information;

[0033] train the text recognition model to be trained according to the total loss information.

[0034] A text recognition device, the device comprising:

[0035] an image acquisition module configured to acquire a to-be-recognized image;

[0036] a visual feature extraction module configured to input the to-be-recognized image into a visual network of a text recognition model, and extract visual features of the to-be-recognized image;

[0037] a semantic feature extraction module configured to input the visual features into a language network of the text recognition model, and extract semantic features based on the visual features through the language network;

[0038] a feature fusion module configured to input the visual features and the semantic features into a fusion network of the text recognition model, and perform fusion processing on the visual features and the semantic features through a plurality of fusion units in the fusion network to obtain fusion results;

[0039] a text recognition module configured to generate a text recognition result for the to-be-recognized image based on the fusion results.

[0040] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0041] A computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method when executed by a processor.

[0042] A computer program product comprises a computer program, and the computer program implements the steps of the method when executed by a processor.

[0043] The above text recognition method, device, computer device, storage medium and computer program product, by acquiring the to-be-recognized image; inputting the to-be-recognized image into a visual network of a text recognition model, extracting visual features of the to-be-recognized image; inputting the visual features into a language network of the text recognition model, extracting semantic features based on the visual features through the language network; inputting the visual features and the semantic features into a fusion network of the text recognition model, performing fusion processing on the visual features and the semantic features through multiple fusion units in the fusion network, obtaining a fusion result; generating a text recognition result for the to-be-recognized image based on the fusion result; in this way, the modal features generated by the visual network and the language network are fused, joint representation learning of vision and language is completed, interaction between visual flow and semantic flow in the same information space is realized, and the fusion result obtained based on the interaction can accurately determine the text recognition result. Compared with correcting the text context in the image based on a pre-trained language model, the text in the image can be more efficiently and accurately recognized. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 An application environment diagram of a text recognition method in an embodiment;

[0046] Figure 2 A flowchart of a text recognition method in an embodiment;

[0047] Figure 3 A data processing flowchart of a text recognition method in an embodiment;

[0048] Figure 4 A schematic diagram of an encoder in an embodiment;

[0049] Figure 5 a schematic diagram of a fusion network in one embodiment;

[0050] Figure 6 a schematic diagram of a training method of a text recognition model in one embodiment;

[0051] Figure 7 a schematic diagram of a text recognition method in another embodiment;

[0052] Figure 8 a structural block diagram of a text recognition device in one embodiment;

[0053] Figure 9 a structural block diagram of a training device of a text recognition model in one embodiment;

[0054] Figure 10 an internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0056] The text recognition method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The server 104 obtains an image to be recognized; the server 104 inputs the image to be recognized into a visual network of a text recognition model, and extracts visual features of the image to be recognized; the server 104 inputs the visual features into a language network of the text recognition model, and extracts semantic features based on the visual features through the language network; the server 104 inputs the visual features and the semantic features into a fusion network of the text recognition model, and processes the visual features and the semantic features through a plurality of fusion units in the fusion network to obtain a fusion result; and the server 104 generates a text recognition result for the image to be recognized based on the fusion result. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0057] In one exemplary embodiment, as shown in Figure 2 A text recognition method is provided, which is applied to the server 104 in Figure 1 for example, and includes the following steps S202 to S210. Among them:

[0058] Step S202, obtaining an image to be recognized.

[0059] Among them, the image to be recognized can be an image containing text information.

[0060] Optionally, the server obtains the image to be recognized.

[0061] Step S204, inputting the image to be recognized into a visual network of a text recognition model to extract visual features of the image to be recognized.

[0062] Among them, the text recognition model can be a model mainly including a visual network (Vision Model, VM), a language network (Semantic Model, SM) and a fusion network. Figure 3 A data processing flowchart of the text recognition method is provided, which can be combined with Figure 3 the data processing flowchart to understand the steps S202-S210 of the embodiment.

[0063] Among them, the visual network can adopt a ViT model (Vision Transformer) to capture key visual features from the image.

[0064] Among them, the language network can adopt an SRM model (Semantic Reasoning Model) to process text data to extract semantic information.

[0065] Among them, the fusion network is a multi-modal feature fusion network, which adopts an adaptive selection matrix and a gated mechanism feature fusion strategy to fuse the features of the two modalities in the feature space to form a unified feature representation. Not only can it retain the unique information of each modality, but also can promote the interaction and complementarity of cross-modal information, so as to realize the joint representation learning of vision and language.

[0066] Among them, the visual features can be features with visual information extracted from the spatial structure by the Vision Transformer of the visual network, which can be represented as .

[0067] Optionally, the server inputs the image to be recognized into the visual network of the text recognition model to extract the visual features of the image to be recognized by the visual network.

[0068] In step S206, the visual feature is input to a language network of the text recognition model, and semantic features are extracted based on the visual feature by the language network.

[0069] The semantic features can be features with semantic information output by the language network, and can be represented as .

[0070] Optionally, the server inputs the visual feature extracted by the visual network to the language network of the text recognition model, and extracts semantic features based on the visual feature by the language network.

[0071] In step S208, the visual feature and the semantic feature are input to a fusion network of the text recognition model, and the visual feature and the semantic feature are fused by a plurality of fusion units in the fusion network to obtain a fusion result.

[0072] The fusion result can be a result obtained by aligning and enhancing fusion of the visual feature and the semantic feature, and compared with the visual feature or the semantic feature of a single mode, the fusion result has more rich representation information.

[0073] Optionally, the server inputs the visual feature of the visual space and the semantic feature with semantic information to the fusion network of the text recognition model, and aligns and enhances fusion of the visual feature of the visual space and the semantic feature with semantic information by a plurality of fusion units in the fusion network to obtain a fusion result.

[0074] In step S210, a text recognition result for the to-be-recognized image is generated based on the fusion result.

[0075] The text recognition result can be text finally recognized in the to-be-recognized image.

[0076] Optionally, the server generates a text recognition result for the to-be-recognized image based on the fusion result.

[0077] In the text recognition method, the to-be-recognized image is obtained; the to-be-recognized image is input into a visual network of a text recognition model, and visual features of the to-be-recognized image are extracted; the visual features are input into a language network of the text recognition model, and semantic features are extracted based on the visual features by the language network; the visual features and the semantic features are input into a fusion network of the text recognition model, and the visual features and the semantic features are fused by a plurality of fusion units in the fusion network to obtain a fusion result; and a text recognition result for the to-be-recognized image is generated based on the fusion result. In this way, the modal features generated by the visual network and the language network are fused, joint representation learning of vision and language is completed, interaction between visual flow and semantic flow in the same information space is realized, and the fusion result obtained based on the interaction can accurately determine the text recognition result. Compared with correcting the text context in the image based on the pre-trained language model, the text in the image can be more efficiently and accurately recognized.

[0078] In an example embodiment, the to-be-recognized image is input into the visual network of the text recognition model to extract the visual features of the to-be-recognized image, including: obtaining a vector representation sequence of the to-be-recognized image; inputting the vector representation sequence into an encoder of the visual network, and performing encoding processing on the vector representation sequence by the encoder to obtain an encoded sequence, and taking the encoded sequence as the visual features.

[0079] The vector representation sequence of the to-be-recognized image can be represented as F0, and F0 should be input into the encoder of the visual network.

[0080] Optionally, the server can obtain a vector representation sequence F0 of the to-be-recognized image, input the vector representation sequence F0 into the encoder of the visual network, perform encoding processing on the vector representation sequence F0 by the encoder to obtain an encoded feature , and take the encoded feature as the visual features.

[0081] In this application, considering the influence of the correlation of the positions of the to-be-recognized image on the semantic correlation in the process of generating the visual features, the VIT structure is used to model the long-range dependency relationship in the visual network. The visual network includes a Transformer encoder and a decoder, and only the Transformer encoder is described in this embodiment. For the description of the decoder, please refer to the following embodiments.

[0082] The Transformer encoder is stacked by a plurality of encoders, and the structure of each encoder can refer to Figure 4 . Figure 4 In the Transformer encoder, F0 represents the visual features of the to-be-recognized image, and the acquisition method can refer to the next embodiment, The output of one encoder will be the input of the next encoder. In practical applications, when F0 is input to the encoder: first, layer normalization (LN) processing is needed, which is different from batch normalization (BN). BN normalizes each channel of a batch of data, while LN normalizes a specified dimension of a single data regardless of the batch; then, the vector after the layer normalization processing is input to the multi-head attention layer, which enhances the features of each position through the multi-head attention mechanism and encodes the appearance information of the text. Using multiple heads instead of a single head allows the model to jointly focus on information from different representation subspaces at different positions; then, the information output by the multi-head attention layer is input to the DropPath layer, which corresponds to a pruning method of randomly dropping (or retaining) some layers. Unlike traditional Dropout, which randomly drops neurons in each training iteration, DropPath randomly drops entire layers with a certain probability, effectively reducing network learning parameters, promoting gradient propagation updates, and accelerating network convergence; finally, the output of the DropPath layer is input to the multi-layer perceptron (MLP) after residual connection and LN layer, which is composed of two linear transformation layers and an activation function GELU, as shown in FIG. 3. MLP performs nonlinear transformation and mapping on the input features, which helps the visual network learn more complex feature representations. The output of the multi-head attention layer MHA can be represented as: Figure 4 , , , L is the number of encoders; and the output of the multi-layer perception MLP layer is: , As the input of the next encoder, the encoded features obtained through a linear layer are the encoded features output by the encoder as the final output.

[0083] In this embodiment, the vector representation sequence of the to-be-identified image is obtained; the vector representation sequence is input to the encoder of the visual network, the vector representation sequence is encoded by the encoder, and an encoded sequence is obtained, which is used as the visual feature. In this way, the visual feature of the to-be-identified image in the visual space can be accurately extracted, which is conducive to accurately extracting the semantic feature subsequently.

[0084] In an exemplary embodiment, obtaining the vector representation sequence of the to-be-identified image includes: segmenting the to-be-identified image into a plurality of image blocks, and mapping an image block feature of any image block to a feature vector of the any image block; obtaining a learnable class embedding vector and a position embedding vector for the to-be-identified image, and generating the vector representation sequence of the to-be-identified image based on the learnable class embedding vector, the position embedding vector, and the feature vectors corresponding to the image blocks.

[0085] Optionally, the image to be recognized is segmented into a plurality of two-dimensional sub-image blocks , the size of the image to be recognized is , 3 channels, and the size of the sub-image block is , a two-dimensional feature sequence with a length of N is obtained, and then each sub-image block is mapped into a one-dimensional vector by a linear projection layer, and the size of the one-dimensional vector is , then a learnable class embedding vector with the same size D is inserted in front of it, and the two have the same size and can be spliced together, and a unique position encoding with the same dimension is superimposed in each embedding, and finally the vector F0 is obtained, which is the vector representation sequence of the image to be recognized. F0 will be used as the input of the encoder.

[0086] The input F0 of the encoder can be represented as , wherein is an embedding, is a learnable embedding, is a position embedding.

[0087] In this embodiment, the image to be recognized is segmented into a plurality of image blocks, the image block features of any image block are mapped into feature vectors of the image block, the learnable class embedding vector and the position embedding vector for the image to be recognized are obtained, and the vector representation sequence of the image to be recognized is generated based on the learnable class embedding vector, the position embedding vector and the feature vectors corresponding to the image blocks. In this way, the local detail features of the image can be captured, and the model can not only focus on the content of the image, but also consider the position information of the image blocks and the overall class semantics by combining the learnable class embedding vector and the position embedding vector, so that the visual features of the image can be more comprehensively and accurately represented, and the understanding and distinguishing ability of the image can be enhanced.

[0088] In an example embodiment, the visual features are input into a language network of the text recognition model, and semantic features are extracted from the visual features based on the language network, including: determining character probability features according to the visual features; determining a character mask matrix according to a character prediction order; inputting the character probability features and the character mask matrix into the language network to obtain the semantic features.

[0089] The character prediction order refers to the order in which the language model predicts characters one by one according to certain specific rules or logic in a text sequence generation task, which can be from left to right, from right to left, bidirectional prediction, etc.

[0090] The character mask matrix can be represented as Mmask, and the size of Mmask is LxL, .

[0091] ​​Optionally, the server linearizes and normalizes the visual features to obtain character probability features F P , and determines a character mask matrix Mmask according to a character prediction order, inputs the character probability features F P and the character mask matrix Mmask into a multi-head self-attention network of the language network, and finally outputs semantic features.

[0092] In the present application, the scene character recognition problem is considered from the perspective of context modeling, and the initial character embedding sequence output by the visual model is used as the input of the language model, which itself has language representation corresponding to the text image, and such cross-modal information has little redundancy and can better complement and enhance other modal information in the subsequent fusion process. Based on the mode of text prediction, the SRM is added to model the correlation of characters in the text sequence, a character mask matrix Mmask of LxL size is set according to the character prediction order, and then the visual features ( are linearized and normalized to obtain character probability features F , which are used in the intermediate form in the decoder processing to serve the generation of character prediction . P , F P and Mmask are used as the input of the multi-head self-attention to obtain features , which are input into a multi-layer network to obtain semantic features . 、 The expression of F and Mmask is , where is a transformation matrix obtained by training, and d is the dimension of the multi-head self-attention. The expression of the output feature of the s-th attention head is , where is the output feature of the s-th attention head, Q, are the query, key and value vectors in the self-attention mechanism, respectively, is a scaling factor used to stabilize training.

[0093] In actual application, when the language network predicts the i-th character, i is equal to j, and the mask matrix Mmask at (i, j) is negative infinity at this time. After the softmax function, the finally calculated weight coefficient is 0, that is, the information of the character itself is ignored, and the prediction result is completely obtained from the information of other characters, so as to avoid the interference of the character information on the prediction result and improve the prediction ability of the model. By sequentially predicting the characters at each position, the character features F S with semantic information can be obtained.

[0094] In this embodiment, the character probability feature is determined according to the visual feature, the character mask matrix is determined according to the character prediction order, and the character probability feature and the character mask matrix are input into the language network to obtain the semantic feature. In this way, the prediction of each character is completely obtained from the information of other characters, which is beneficial to avoid the interference of the information of the character itself on the prediction result, and is beneficial to improve the prediction ability of the model.

[0095] In an exemplary embodiment, the fusion network includes a first fusion unit, a second fusion unit, and a third fusion unit. The fusion network inputs the visual feature and the semantic feature into the text recognition model, and performs fusion processing on the visual feature and the semantic feature through multiple fusion units in the fusion network to obtain a fusion result. The fusion network includes: inputting the visual feature and the semantic feature into the first fusion unit to obtain a first fusion result; inputting the visual feature, the semantic feature, and the first fusion result into the second fusion unit to obtain a second fusion result; and inputting the visual feature, the semantic feature, and the second fusion result into the third fusion unit to obtain the fusion result.

[0096] The fusion network constructs the modality information screening mechanism in a more fine-grained and more flexible manner by stacking multiple fusion units. The input of the fusion network is the visual feature and the semantic feature aligned at each time step.

[0097] Optionally, in the model inference stage, the server inputs the visual feature and the semantic feature into the first fusion unit to obtain a first fusion result, inputs the visual feature, the semantic feature, and the first fusion result into the second fusion unit to obtain a second fusion result, and finally inputs the visual feature, the semantic feature, and the second fusion result into the third fusion unit to obtain the fusion result.

[0098] Figure 5 A schematic diagram of the fusion network is provided. The fusion network includes three fusion units. In the fusion units, a hard gating mechanism is used to select the retention mode of the information. A gating function G(·) is used to select a one-way path, where The selection matrix is adaptive. It is updated during the training process. Its main function is to give the current retention distribution of different modality vectors, and dynamically determine which modality (visual or semantic) feature information needs to be retained, and which can be inhibited or discarded. A zero matrix of the same dimension can also be selected. This selection is mainly used in the initial stage of feature fusion, and a good prediction result has been obtained. The zero matrix can reduce the subsequent redundant operation, and for simple recognition tasks, it can improve the calculation speed of the system. Figure 5 Only two selection methods are shown in the fusion unit. In fact, more complex operation modes can be used in the fusion unit to expand the representation ability to cope with larger and more complex task requirements. The output of the fusion unit can be represented as:

[0099] ,

[0100] ,

[0101] wherein, is the standardized input before fusion; , , and , , are learnable weight matrices respectively; is the intermediate feature vector after multi-layer linear transformation inside the fusion unit, used for gating mechanism to filter information, One implementation of It should be noted that is not the input of the initial stage of the fusion unit, the input of the initial stage of the fusion unit is the spliced result of the visual feature and the semantic feature after the first layer of weight matrix transformation ( , ), and is the intermediate result after more weight matrix transformation on this basis, used for subsequent gating calculation. The difference between the two lies in the processing stage, is the product of deep calculation inside the fusion unit; the output of the fusion unit is finally sent to the next stage for the same operation, which will be used as the input of the next fusion unit or time step, realizing chain processing, t represents the time step (such as the character prediction order when decoding), and i represents the fusion unit number; is a zero matrix, used for hard shielding part of the features (directly set to zero), suitable for simple tasks or initial fusion stage to reduce the amount of calculation; is a dynamic gating weight matrix, which adaptively learns the retention ratio of multi-modal features.

[0102] In the model training, each fusion unit outputs the fusion result for prediction, denoted as y t, and the system can learn the fusion degree between the features, that is, the gating weight (the gating weight can be regarded as a quantitative indicator of the fusion degree, if the visual feature weight is high, the system relies more on the image appearance, and if the semantic feature weight is high, the system relies more on the language context, and in the subsequent model inference stage, the feature fusion can be realized directly according to the recorded fusion degree between the features). According to the complexity of the task, the gating mechanism is used to adjust the calculation amount of the system, saving the computing power and time. In addition, in order to prevent the information from being lost in the operation process in the initial stage of model training, a residual connection is added between the visual feature and the output of the fusion module. In the model inference stage, the output of the last fusion unit is taken as the final output, and in the model training stage, the output of each fusion unit corresponds to a text recognition result. In this way, the model training stage realizes deep supervision through unit-by-unit prediction, and the model inference stage realizes the balance between efficiency and accuracy only by using the output result of the last unit.

[0103] In the embodiment, the visual feature and the semantic feature are input into the first fusion unit to obtain the first fusion result; the visual feature, the semantic feature and the first fusion result are input into the second fusion unit to obtain the second fusion result; and the visual feature, the semantic feature and the second fusion result are input into the third fusion unit to obtain the fusion result. In this way, the layer-by-layer fusion of the features is realized, the information in the visual and semantic features can be more fully utilized, and useful information can be preserved and integrated as much as possible in each fusion process, reducing the information loss caused by direct discarding or simple merging.

[0104] In an exemplary embodiment, as shown in Figure 6 , a training method of a text recognition model is provided, which is applied to the server 104 in Figure 1 for example, and includes the following steps S602 to S614. Among them:

[0105] Step S602, obtaining a training sample image.

[0106] The training sample image can be a sample containing text information for training the text recognition model.

[0107] Optionally, the server obtains a training sample image, and the training sample image has a corresponding character label.

[0108] Step S604, inputting the training sample image into the visual network of the text recognition model to be trained, extracting the visual feature of the training sample image through the visual network, and outputting the first character prediction result for the training sample image based on the visual feature.

[0109] The visual feature of the training sample image refers to a feature with visual information coded by an encoder of the visual network.

[0110] The first character prediction result of the training sample image refers to a character prediction probability distribution output by the visual model at the t-th time step (or character position), which can be understood as a probability that the current character is a certain category (such as a letter or a number) according to the visual network.

[0111] Optionally, the server inputs the training sample image into the visual network of the text recognition model to be trained, extracts the visual feature of the training sample image through the encoder of the visual network, and decodes the visual feature of the training sample image through the decoder of the visual network to output the first character prediction result for the training sample image.

[0112] The task of the decoder of the visual network of the present application is to generate a character prediction sequence according to the tokens. In order to make full use of the rich information of each token, a token learning function is used in the decoder to optimize the selection of token pixels, so that the prediction of each character can come from the corresponding token. Specifically, a learnable tokenization function is set , which maps the input to a token vector The function learns to multiply the weight map of the input condition (the weight map of the input condition is an attention distribution dynamically generated by the decoder, which is used to determine which local area of the input image (i.e., the token) should be focused on when predicting the current character. By learning to automatically assign weights, the higher the weight, the more important the corresponding area is to the prediction of the current character) and The calculation process can be represented as: , e represents element-wise multiplication, represents a single or series of convolutional layers plus an activation function, The function represents global pooling, represents a broadcast function. This attention mechanism is in the form of spatial self-attention of elements, and the spatial positions in are adaptively selected to construct N such functions to realize the combination of information. This method makes the token no longer a fixed partition of the input tensor, but a set of adaptively changing spatial selections. Different tokens are allowed to model their spatio-temporal relationships and internal connections. The S tokens generated after learning can be represented as: , S represents the number of adaptively generated tokens, i.e., the number of dynamically selected spatial regions. The S tokens are S key regions (which can be regarded as “softly partitioned” image blocks) dynamically filtered from the input features by the model for subsequent decoding. Finally, a series of linear projection layers form a classification head to generate character predictions , , T represents the length of the character sequence, i.e., the total number of characters to be predicted, such as the number of letters of a word, C represents the total number of character classes, which can include the total number of alphabet sizes and special symbols, the subscript S indicates the Sth adaptively generated token, and the superscript i indicates the i th step of prediction of the decoder.

[0113] In step S606, the visual features of the training sample image are input to the language network of the text recognition model to be trained, the semantic features are extracted based on the visual features of the training sample image through the language network, and the second character prediction result for the training sample image is output based on the semantic features.

[0114] The semantic features of the training sample image refer to the features with semantic information output by the language network.

[0115] The second character prediction result of the training sample image refers to the character prediction probability distribution output by the language network at the t th time step, which can be understood as a prediction result optimized based on context semantic information.

[0116] Optionally, the server inputs the visual features of the training sample image to the language network of the text recognition model to be trained, linearly and normally processes the visual features of the training sample image through the language network to obtain character probability features, inputs the character probability features and the character mask matrix to the multi-head sub-attention network of the language network, finally extracts the semantic features, and outputs the second character prediction result for the training sample image based on the semantic features.

[0117] In step S608, the visual features and the semantic features of the training sample image are input to the fusion network of the text recognition model to be trained, the fusion results output by each fusion unit in the fusion network are obtained, and based on the fusion results output by each fusion unit, the text recognition result corresponding to each fusion unit is generated.

[0118] The fusion result output by each fusion unit can be a result obtained after feature fusion based on the respective operation mode of each fusion unit.

[0119] Optionally, the server inputs the visual features and the semantic features of the training sample image to the fusion network of the text recognition model to be trained, when the fusion network includes a first fusion unit, a second fusion unit, and a third fusion unit, obtains a first fusion result output by the first fusion unit, a second fusion result output by the second fusion unit, and a third fusion result output by the third fusion unit, predicts a first text recognition result based on the first fusion result, predicts a second text recognition result based on the second fusion result, and predicts a third text recognition result based on the third fusion result.

[0120] Step S610, according to the character label corresponding to the training sample image and the first character prediction result corresponding to the training sample image, determine the visual network loss information, and according to the character label corresponding to the training sample image and the second character prediction result corresponding to the training sample image, determine the language network loss information, and according to the character label corresponding to the training sample image and the text recognition result corresponding to each fusion unit, determine the fusion network loss information, determine the fusion network loss information.

[0121] Wherein, the character label can be expressed as , which represents the real character sequence (GroundTruth) corresponding to the image to be recognized, that is, the correct text content that the text recognition model needs to predict.

[0122] Wherein, the expression of the visual network loss information is:

[0123]

[0124] The loss of the visual network is represented by is the character prediction probability distribution output by the visual model at the t-th time step (or character position), which can be understood as the probability that the visual network considers the current character to be a certain class (such as letters or numbers). , T is the maximum length of the word, which is set to 15 in the experiment.

[0125] Wherein, the expression of the language network loss information is:

[0126] ,

[0127] The loss of the language network is represented by is the character prediction probability distribution output by the language network at the t-th time step, which can be understood as the prediction result optimized based on the context semantic information. The above , it is characterized that the visual model and the semantic model can adopt the same cross-entropy loss function, and the optimization goals of the two are to minimize the probability difference between the prediction and the label through "information matching" to respectively improve the feature modeling ability of each modal.

[0128] Optionally, the server determines visual network loss information according to the character label corresponding to the training sample image and the first character prediction result corresponding to the training sample image, determines language network loss information according to the character label corresponding to the training sample image and the second character prediction result corresponding to the training sample image, determines first fusion unit loss information of the first fusion unit according to the character label corresponding to the training sample image and the first text recognition result, determines second fusion unit loss information of the second fusion unit according to the character label corresponding to the training sample image and the second text recognition result, determines third fusion unit loss information of the third fusion unit according to the character label corresponding to the training sample image and the third text recognition result, and determines fusion network loss information according to the first fusion unit loss information, the second fusion unit loss information and the third fusion unit loss information.

[0129] In actual training, for the fusion network, the three fusion units all give text recognition results. In order to reduce the calculation amount of the entire model while maintaining a good text recognition accuracy, the present application adds a penalty term to the cross-entropy loss function to suppress the adaptive selection matrix Under the drive of the recognition loss, the path with larger calculation amount is selected to obtain more accurate prediction results through as few feature dimensions as possible. The loss function expression of the fusion network is:

[0130] ,

[0131] Among them, is the loss function of the fusion network; , , is the cross-entropy loss of the three fusion units, , , respectively, the loss weight of the fusion unit; is the penalty coefficient of the regularization term, which is the L2 norm of the model parameter calculation amount; represents the learnable parameters of the fusion network, specifically including the gating matrix (the adaptive selection matrix), the weight matrix , , and , , (for feature transformation) and other parameters in the fusion unit.

[0132] Step S612, determining total loss information according to the visual network loss information, the language network loss information and the fusion network loss information.

[0133] Optionally, the server determines total loss information according to the visual network loss information, the language network loss information, and the fusion network loss information.

[0134] In actual training, the loss function of the text recognition model can be represented as:

[0135] ,

[0136] wherein, is the loss function of the text recognition model, , , are loss weights of the visual network, the language network, and the fusion network, respectively.

[0137] Step S614: training the text recognition model to be trained according to the total loss information.

[0138] Optionally, the server trains the text recognition model to be trained according to the total loss information until the model converges.

[0139] The training method of the text recognition model, by obtaining a training sample image; inputting the training sample image into a visual network of the text recognition model to be trained, extracting visual features of the training sample image through the visual network, and outputting a first character prediction result for the training sample image based on the visual features; inputting the visual features of the training sample image into a language network of the text recognition model to be trained, extracting semantic features based on the visual features of the training sample image through the language network, and outputting a second character prediction result for the training sample image based on the semantic features; inputting the visual features and the semantic features of the training sample image into a fusion network of the text recognition model to be trained, obtaining fusion results output by each fusion unit in the fusion network, generating a text recognition result corresponding to each fusion unit based on the fusion results output by each fusion unit; determining visual network loss information according to the character label corresponding to the training sample image and the first character prediction result corresponding to the training sample image, determining language network loss information according to the character label corresponding to the training sample image and the second character prediction result corresponding to the training sample image, and determining fusion network loss information according to the character label corresponding to the training sample image and the text recognition result corresponding to each fusion unit; determining total loss information according to the visual network loss information, the language network loss information and the fusion network loss information; training the text recognition model to be trained according to the total loss information; in this way, the modal features output by the visual network and the language network can be used for preliminary prediction, which helps the visual network to learn more accurate vector representation of the spatial position relationship and texture features of the text in the image, and helps the language model to learn language vector relationship conforming to the distribution of the training data, and the deep supervision in the training stage is realized through the element-by-element prediction of each fusion unit. Overall, a text recognition model that can accurately recognize image text can be trained.

[0140] The multimodal fusion process of the method of the present application first cuts and blocks the input image, reshapes and flattens it into non-overlapping small image blocks, then linearly projects the size and adds a learnable embedding to obtain a token sequence, which is input to the encoder of the visual network. The visual network extracts visual features from the spatial structure through the encoder and decodes to predict the character embedding of the visual space. Then, the decoder of the visual network enables adaptive tokenization, dynamically selects tokens according to the input conditions and predicts the corresponding characters. Then, the character embedding predicted by the visual model feature is input to the semantic network constructed by the superimposed multi-layer variant Transformer to generate semantic features. Then, the visual features and semantic features are sent to the multimodal fusion module to realize the self-interaction of the visual stream and the semantic stream in the same space information. Finally, the fused feature sequence is calculated through a fully connected layer and a SoftMax operator to obtain the final prediction result. In this way, the modal features generated by the visual network and the language network are fused, and the joint representation learning of vision and language is completed. The text recognition model mainly includes three main fine-grained components: visual network, semantic network and visual-semantic fusion network. Unlike the way of separately enhancing visual features or semantic features, it can align and enhance each modal information from the interaction mode between different modal streams and perform non-local attention. Finally, it is globally fused.

[0141] In another embodiment, as shown in Figure 7 , a text recognition method is provided. The method is applied to the server 104 in Figure 1 for example, and includes the following steps:

[0142] Step S702, obtaining an image to be recognized.

[0143] Step S704, inputting the image to be recognized into a visual network of a text recognition model to extract visual features of the image to be recognized.

[0144] Step S706, inputting the visual features into a language network of the text recognition model to extract semantic features based on the visual features through the language network.

[0145] Step S708, inputting the visual features and the semantic features into a first fusion unit to obtain a first fusion result.

[0146] Step S710, inputting the visual features, the semantic features and the first fusion result into a second fusion unit to obtain a second fusion result.

[0147] Step S712, inputting the visual features, the semantic features and the second fusion result into a third fusion unit to obtain a fusion result.

[0148] Step S714, generating a text recognition result for the image to be recognized based on the fusion result.

[0149] It should be noted that the specific definitions of the above steps can refer to the specific definitions of the text recognition method described above.

[0150] It should be understood that, although the steps in the flowcharts involved in the embodiments described above are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowcharts involved in the embodiments described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiments of the present application also provide a text recognition device for implementing the text recognition method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific definitions in one or more text recognition device embodiments provided below can refer to the definitions of the text recognition method described above, and will not be repeated here.

[0152] In an exemplary embodiment, as shown in Figure 8 A text recognition device is provided, comprising: an image acquisition module 802, a visual feature extraction module 804, a semantic feature extraction module 806, a feature fusion module 808, and a text recognition module 810, wherein:

[0153] The image acquisition module 802 is configured to acquire an image to be recognized.

[0154] The visual feature extraction module 804 is configured to input the image to be recognized into a visual network of a text recognition model, and extract visual features of the image to be recognized.

[0155] The semantic feature extraction module 806 is configured to input the visual features into a language network of the text recognition model, and extract semantic features based on the visual features through the language network.

[0156] The feature fusion module 808 is configured to input the visual features and the semantic features into a fusion network of the text recognition model, and perform fusion processing on the visual features and the semantic features through multiple fusion units in the fusion network to obtain a fusion result.

[0157] The text recognition module 810 is configured to generate a text recognition result for the image to be recognized based on the fusion result.

[0158] In one of the embodiments, the visual feature extraction module 804 is specifically configured to obtain a sequence of vector representations of the image to be recognized; input the sequence of vector representations to an encoder of the visual network, perform encoding processing on the sequence of vector representations by the encoder, and obtain an encoded sequence as the visual feature.

[0159] In one of the embodiments, the visual feature extraction module 804 is specifically configured to divide the image to be recognized into a plurality of image blocks, map an image block feature of any image block to a feature vector of the any image block; obtain a learnable class embedding vector and a position embedding vector for the image to be recognized, and generate a sequence of vector representations of the image to be recognized based on the learnable class embedding vector, the position embedding vector, and the feature vectors corresponding to the image blocks.

[0160] In one of the embodiments, the semantic feature extraction module 806 is specifically configured to determine a character probability feature according to the visual feature; determine a character mask matrix according to a character prediction order; and input the character probability feature and the character mask matrix to the language network to obtain the semantic feature.

[0161] In one of the embodiments, the fusion network includes a first fusion unit, a second fusion unit, and a third fusion unit, and the feature fusion module 808 is configured to input the visual feature and the semantic feature to the first fusion unit to obtain a first fusion result; input the visual feature, the semantic feature, and the first fusion result to the second fusion unit to obtain a second fusion result; and input the visual feature, the semantic feature, and the second fusion result to the third fusion unit to obtain a fusion result.

[0162] Based on the same inventive concept, the embodiments of the present application also provide a training device of a text recognition model for implementing the training method of the text recognition model. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more text recognition model training device embodiments provided below can refer to the limitations of the text recognition model training method in the above text, which will not be repeated here.

[0163] In one exemplary embodiment, as shown in Figure 9 a training device of a text recognition model is provided, which includes a sample acquisition module 902, a first output module 904, a second output module 906, a third output module 908, a first determination module 910, a second determination module 912, and a model training module 914, wherein:

[0164] The sample acquisition module 902 is configured to acquire a training sample image.

[0165] The first output module 904 is configured to input the training sample image into a visual network of the text recognition model to be trained, extract visual features of the training sample image through the visual network, and output a first character prediction result for the training sample image based on the visual features.

[0166] The second output module 906 is configured to input the visual features of the training sample image into a language network of the text recognition model to be trained, extract semantic features based on the visual features of the training sample image through the language network, and output a second character prediction result for the training sample image based on the semantic features.

[0167] The third output module 908 is configured to input the visual features and the semantic features of the training sample image into a fusion network of the text recognition model to be trained, obtain fusion results output by each fusion unit in the fusion network, and generate a text recognition result corresponding to each fusion unit based on the fusion results output by each fusion unit.

[0168] The first determination module 910 is configured to determine visual network loss information according to a character label corresponding to the training sample image and the first character prediction result corresponding to the training sample image, determine language network loss information according to the character label corresponding to the training sample image and the second character prediction result corresponding to the training sample image, and determine fusion network loss information according to the character label corresponding to the training sample image and the text recognition result corresponding to each fusion unit.

[0169] The second determination module 912 is configured to determine total loss information according to the visual network loss information, the language network loss information, and the fusion network loss information.

[0170] The model training module 914 is configured to train the text recognition model to be trained according to the total loss information.

[0171] The above-described various modules in the text recognition device and the training device of the text recognition model can be all or partially implemented by software, hardware, and a combination thereof. The above-described various modules can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in the computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the above-described various modules.

[0172] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 6. Figure 10As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text recognition data and training data of a text recognition model. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to implement a text recognition method and a training method of a text recognition model.

[0173] Those skilled in the art can understand that, Figure 10 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0174] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the text recognition method and the training method of the text recognition model described above. The steps of the text recognition method and the training method of the text recognition model here can be the steps in the text recognition method and the training method of the text recognition model of each of the embodiments described above.

[0175] In one embodiment, a computer readable storage medium is provided, storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the text recognition method and the training method of the text recognition model described above. The steps of the text recognition method and the training method of the text recognition model here can be the steps in the text recognition method and the training method of the text recognition model of each of the embodiments described above.

[0176] In one embodiment, a computer program product is provided, including a computer program, the computer program being executed by the processor to make the processor execute the steps of the text recognition method and the training method of the text recognition model described above. The steps of the text recognition method and the training method of the text recognition model here can be the steps in the text recognition method and the training method of the text recognition model of each of the embodiments described above.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0178] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0179] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A text recognition method, characterized by, The method comprises: obtaining an image to be recognized; inputting the image to be recognized into a visual network of a text recognition model to extract visual features of the image to be recognized; inputting the visual features into a language network of the text recognition model to extract semantic features based on the visual features by the language network; inputting the visual features and the semantic features into a fusion network of the text recognition model, and performing fusion processing on the visual features and the semantic features by a plurality of fusion units in the fusion network to obtain a fusion result; generating a text recognition result for the image to be recognized based on the fusion result.

2. The method of claim 1, wherein, The inputting of the image to be recognized into the visual network of the text recognition model to extract the visual features of the image to be recognized comprises: obtaining a vector representation sequence of the image to be recognized; inputting the vector representation sequence into an encoder of the visual network to perform encoding processing on the vector representation sequence by the encoder to obtain an encoded sequence, and taking the encoded sequence as the visual features.

3. The method of claim 1, wherein, The obtaining of the vector representation sequence of the image to be recognized comprises: segmenting the image to be recognized into a plurality of image blocks, and mapping an image block feature of any image block to a feature vector of the image block; obtaining a learnable class embedding vector and a position embedding vector for the image to be recognized, and generating a vector representation sequence of the image to be recognized based on the learnable class embedding vector, the position embedding vector, and the feature vector corresponding to each image block.

4. The method of claim 1, wherein, The inputting of the visual features into the language network of the text recognition model to extract semantic features based on the visual features by the language network comprises: determining character probability features according to the visual features; determining a character mask matrix according to a character prediction order; inputting the character probability features and the character mask matrix into the language network to obtain the semantic features.

5. The method of claim 1, wherein, The fusion network comprises a first fusion unit, a second fusion unit, and a third fusion unit, and the inputting of the visual features and the semantic features into the fusion network of the text recognition model, and the fusion processing on the visual features and the semantic features by a plurality of fusion units in the fusion network to obtain a fusion result comprises: inputting the visual features and the semantic features into the first fusion unit to obtain a first fusion result; inputting the visual features, the semantic features, and the first fusion result into the second fusion unit to obtain a second fusion result; inputting the visual features, the semantic features, and the second fusion result into the third fusion unit to obtain the fusion result. 6.A method for training a text recognition model, comprising: The method further comprises: obtaining a training sample image; inputting the training sample image into a visual network of a text recognition model to be trained to extract visual features of the training sample image by the visual network, and outputting a first character prediction result for the training sample image based on the visual features; input the visual features of the training sample image to a language network of the text recognition model to be trained, extract semantic features based on the visual features of the training sample image through the language network, and output a second character prediction result for the training sample image based on the semantic features; input the visual features and the semantic features of the training sample image to a fusion network of the text recognition model to be trained, obtain fusion results output by each fusion unit in the fusion network, and generate a text recognition result corresponding to each fusion unit based on the fusion results output by each fusion unit; determine visual network loss information according to the character label corresponding to the training sample image and the first character prediction result corresponding to the training sample image, determine language network loss information according to the character label corresponding to the training sample image and the second character prediction result corresponding to the training sample image, and determine fusion network loss information according to the character label corresponding to the training sample image and the text recognition result corresponding to each fusion unit; determine total loss information according to the visual network loss information, the language network loss information, and the fusion network loss information; train the text recognition model to be trained according to the total loss information.

7. A text recognition apparatus characterized by comprising: The apparatus comprises: an image acquisition module configured to acquire a to-be-recognized image; a visual feature extraction module configured to input the to-be-recognized image to a visual network of a text recognition model, and extract visual features of the to-be-recognized image; a semantic feature extraction module configured to input the visual features to a language network of the text recognition model, and extract semantic features based on the visual features through the language network; a feature fusion module configured to input the visual features and the semantic features to a fusion network of the text recognition model, and perform fusion processing on the visual features and the semantic features through a plurality of fusion units in the fusion network to obtain fusion results; a text recognition module configured to generate a text recognition result for the to-be-recognized image based on the fusion results.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images