Image recognition method and device, training method, electronic device and storage medium

Abstract

An image recognition method and device, a training method, an electronic device and a storage medium. The image recognition method comprises: obtaining an image to be recognized, the image to be recognized includes a target text; determining the text content of the target text based on knowledge information and image information of the image to be recognized. When performing text recognition, the image recognition method not only considers the image information of the image to be recognized, but also considers the associated knowledge information. Combining these two aspects of information for text recognition can effectively improve the accuracy of text recognition.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image recognition method and apparatus, a training method, an electronic device, and a storage medium. Background Art

[0002] Optical Character Recognition (OCR) is a commonly used text recognition technology. OCR refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text. For example, optical character recognition technology can use optical methods to convert the text in a paper document into a black and white dot matrix image file for printed characters, and use recognition technology to convert the text in the image into a text format for further editing and processing by word processing software. In short, OCR technology can recognize the text in an image and return it in the form of text. Summary of the invention

[0003] At least one embodiment of the present disclosure provides an image recognition method, comprising: acquiring an image to be recognized, wherein the image to be recognized includes a target text; and determining text content of the target text based on knowledge information and image information of the image to be recognized.

[0004] For example, in a method provided in an embodiment of the present disclosure, the text content of the target text is determined based on the knowledge information and the image information of the image to be identified, including: extracting features of the image to be identified to obtain target features; obtaining knowledge representation data based on the knowledge information; fusing the target features and the knowledge representation data to obtain a fusion result; and determining the text content of the target text based on the fusion result.

[0005] For example, in the method provided in one embodiment of the present disclosure, the target feature and the knowledge representation data are fused to obtain the fusion result, including: obtaining a state feature based on the target feature through a recurrent neural network, and splicing the state feature and the knowledge representation data to obtain the fusion result.

[0006] For example, in the method provided in one embodiment of the present disclosure, the state feature and the multiple knowledge representation data are spliced ​​to obtain the fusion result, including: based on the state feature and the multiple knowledge representation data, obtaining the splicing feature of each moment in multiple moments, wherein the splicing feature is the fusion result.

[0007] For example, in the method provided by an embodiment of the present disclosure, the state feature is obtained based on the target feature through the recurrent neural network, including: using a bidirectional recurrent neural network to determine a moment as the current moment among multiple moments; based on the target feature, obtaining the hidden state of the current moment, wherein the hidden state of the current moment includes the forward hidden state of the current moment and the backward hidden state of the current moment; wherein the hidden state of each moment among the multiple moments is the state feature.

[0008] For example, in a method provided by an embodiment of the present disclosure, based on the state features and the knowledge representation data, a splicing feature of each of the multiple moments is obtained, including: based on the hidden state of the current moment and the multiple knowledge representation data, determining the fusion feature of the current moment; splicing the hidden state of the current moment and the fusion feature of the current moment to obtain the splicing feature of the current moment.

[0009] For example, in a method provided in an embodiment of the present disclosure, based on the hidden state at the current moment and the knowledge representation data, determining the fusion features at the current moment includes: based on the hidden state at the current moment, determining M knowledge representation data having a matching association with the hidden state at the current moment from a plurality of knowledge representation data included in the knowledge representation data; based on the M knowledge representation data and the hidden state at the current moment, determining the fusion features at the current moment; M is an integer greater than 0.

[0010] For example, in a method provided in an embodiment of the present disclosure, based on the hidden state at the current moment, the M knowledge representation data having a matching association with the hidden state at the current moment are determined from the multiple knowledge representation data, including: calculating a similarity parameter between each knowledge representation data in the multiple knowledge representation data and the hidden state at the current moment; and determining the M knowledge representation data based on the similarity parameter.

[0011] For example, in the method provided in an embodiment of the present disclosure, the expression of the similarity parameter is as follows: Wherein, ej represents the jth knowledge representation data among the plurality of knowledge representation data, t represents the current time, and the superscript T represents transposition. represents the forward hidden state at the current moment, represents the backward hidden state at the current moment, W represents the parameter matrix, sim(j) represents the similarity parameter between the j-th knowledge representation data and the hidden state at the current moment, t is an integer greater than 0, and j is an integer greater than 0.

[0012] For example, in a method provided in an embodiment of the present disclosure, based on the similarity parameter, the M knowledge representation data are determined, including: taking the M knowledge representation data with the largest similarity parameter as the M knowledge representation data having a matching association with the hidden state at the current moment, or taking the knowledge representation data with a similarity parameter greater than a preset threshold as the M knowledge representation data having a matching association with the hidden state at the current moment.

[0013] For example, in the method provided in an embodiment of the present disclosure, the expression of the fusion feature at the current moment is as follows: Among them, t represents the current moment, Et represents the fusion feature of the current moment, rj represents the j-th knowledge representation data among the M knowledge representation data, sim(j) represents the similarity parameter between the j-th knowledge representation data among the M knowledge representation data and the hidden state at the current moment, sim(k) represents the similarity parameter between the k-th knowledge representation data among the M knowledge representation data and the hidden state at the current moment, and t, j, and k are integers greater than 0.

[0014] For example, in a method provided in an embodiment of the present disclosure, the text content of the target text is determined based on the fusion result, including: based on the splicing feature, outputting a prediction result corresponding to each of the multiple moments, and determining the text content of the target text based on the prediction result.

[0015] For example, in a method provided in an embodiment of the present disclosure, feature extraction is performed on the image to be identified to obtain the target feature, including: using a convolutional neural network to extract feature of the image to be identified to obtain the target feature.

[0016] For example, the method provided by an embodiment of the present disclosure further includes: performing text area detection on the image to be recognized to obtain a text area image including the target text.

[0017] For example, in a method provided in an embodiment of the present disclosure, the text content of the target text is determined based on the knowledge information and the image information of the image to be identified, including: determining the text content of the target text based on the knowledge information and the image information of the text area image.

[0018] For example, the method provided by an embodiment of the present disclosure further includes: outputting the text content of the target text.

[0019] For example, in a method provided in an embodiment of the present disclosure, the image to be identified includes an image of a drug to be identified.

[0020] For example, in a method provided by an embodiment of the present disclosure, the plurality of knowledge representation data comes from a data knowledge base, and the data knowledge base includes a drug knowledge base.

[0021] At least one embodiment of the present disclosure also provides a training method for a first neural network, wherein the first neural network includes a recurrent neural network and a convolutional neural network, and the first neural network is used to: obtain an image to be recognized, wherein the image to be recognized includes a target text; determine the text content of the target text based on knowledge information and image information of the image to be recognized; the training method includes: inputting training data into the first neural network to obtain output data output from the first neural network; and using a back-propagation algorithm to correct parameters of the recurrent neural network and parameters of the convolutional neural network based on a loss function.

[0022] At least one embodiment of the present disclosure further provides an image recognition device, comprising: an acquisition module, configured to acquire an image to be recognized, wherein the image to be recognized includes a target text; and a determination module, configured to determine the text content of the target text based on knowledge information and image information of the image to be recognized.

[0023] At least one embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory, comprising one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include modules for implementing the image recognition method described in any embodiment of the present disclosure.

[0024] At least one embodiment of the present disclosure further provides a non-transitory readable storage medium for storing computer-readable instructions. When the computer-readable instructions are executed by a computer, the image recognition method described in any embodiment of the present disclosure can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0026] Figure 1 A schematic diagram of the structure of a convolutional recurrent neural network provided for at least one embodiment of the present disclosure;

[0027] Figure 2A A flowchart of an image recognition method provided for at least one embodiment of the present disclosure;

[0028] Figure 2B A schematic diagram corresponding to step S102 in an image recognition method provided by at least one embodiment of the present disclosure;

[0029] Figure 2C A schematic diagram corresponding to step S112 in an image recognition method provided by at least one embodiment of the present disclosure;

[0030] Figure 3 An example diagram of an image to be recognized provided by at least one embodiment of the present disclosure;

[0031] Figure 4 A schematic diagram of an image recognition method provided for at least one embodiment of the present disclosure;

[0032] Figure 5 A schematic diagram corresponding to step S131 in an image recognition method provided by at least one embodiment of the present disclosure;

[0033] Fig. 6A According to at least one embodiment of the present disclosure Figure 2C Schematic diagram of the process of step S131;

[0034] Figure 6B According to at least one embodiment of the present disclosure Figure 2C Schematic diagram of the process of step S132; Figure 6C According to at least one embodiment of the present disclosure Figure 6B Schematic diagram of the process of step S13;

[0035] Fig.6D According to at least one embodiment of the present disclosure Figure 6C Schematic diagram of the process of step S21;

[0036] Figure 7 A schematic diagram of an example of using the image recognition method provided by at least one embodiment of the present disclosure for drug name recognition;

[0037] Figure 8 A schematic block diagram of an image recognition device provided for at least one embodiment of the present disclosure;

[0038] Fig. 9 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0039] Fig.10 A schematic block diagram of another image recognition device provided for at least one embodiment of the present disclosure; and

[0040] Fig.11 A schematic diagram of a non-transitory readable storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0042] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] OCR technology can be applied to a variety of scenarios to provide text detection and recognition services. For example, identification of common cards and documents such as ID cards and bank cards, identification of financial bills, medical bills, etc. For example, when visiting a medical institution, a lot of paper documents are usually generated, such as test reports, fee receipts, etc. For example, when making insurance claims or transferring to another hospital, it is usually necessary to convert paper documents into electronic versions. Through OCR technology, the content of paper documents can be converted into electronic files. Compared with manual entry, the use of OCR technology can save time and improve work efficiency.

[0044] For example, in one example, the name of the medicine can be identified from the image of the medicine box taken by the user, which is convenient for the user to input and query the medicine information. For example, in an example scenario, an elderly user took three kinds of medicines from the hospital. There is a medicine query system that can query the usage and dosage of the medicines and the incompatibility between the medicines. At this time, the elderly user needs to query by entering the names of the three medicines. However, it is relatively difficult for elderly users to enter the names of medicines, but it is relatively easy to take pictures with a mobile phone. Therefore, the name of the medicine can be recognized by an image recognition method, and the name of the medicine recognized from the image can be used as input, which is convenient for the elderly user to use.

[0045] For example, in one example, a neural network model can be used to implement image-based text recognition. Figure 1 The convolutional recurrent neural network shown in the figure is used for text recognition. For example, the convolutional recurrent neural network is a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN) and is used to recognize sequential objects in images. Although the convolutional recurrent neural network can be composed of different types of network architectures (such as CNN and RNN), they can be jointly trained through a loss function. For example, the network structure of the convolutional recurrent neural network can be composed of three parts, including a convolutional layer, a recurrent layer, and a transcription layer from bottom to top. For example, the convolutional layer at the bottom of the convolutional recurrent neural network can use any convolutional neural network (CNN), such as Figure 1 The bottom CNN shown is used to extract features from the input image to obtain a convolutional feature map. For example, in one example, the size of the input image (32, 100, 3) is converted into a convolutional feature matrix (convolutional feature map) of size (1, 25, 512) after passing through the CNN, where the size is expressed in the form of (height, width, channel). In order to input the features into the recurrent layer, it is necessary to extract a sequence of feature vectors from the feature map output by the above CNN, and each feature vector is generated from left to right by column on the feature map. For example, in the above example, each feature vector extracted from the convolutional feature matrix of size (1, 25, 512) contains 512-dimensional features, and these feature vectors constitute a feature sequence. The feature sequence can be used as the input of the recurrent layer, and each feature vector is used as the input of the RNN in a time step (time step) (also referred to as a moment in this article). In the above example of the convolutional feature matrix of size (1, 25, 512), there are a total of 25 moments (time steps). For example, the recurrent layer of the convolutional recurrent neural network can adopt any recurrent neural network (RNN), such as a bidirectional recurrent neural network (Bi-directionalRNN, BiRNN), a bidirectional long short-term memory (Bi-directional Long Short Term Memory, BiLSTM) network, etc. Figure 1The RNN shown is used to predict the feature sequence, learn each feature vector in the feature sequence, and then output the prediction result (for example, the probability distribution of the label). For example, there is an input feature vector in a time step, and a probability distribution of all labels is output for each input feature vector. For example, the label category of each column output can be obtained through the argmax function (for example, the function is used to select the label with the largest probability value), forming a preliminary label sequence. For example, the label can be a character or a text, which can be set according to actual needs. For example, in one example, the preliminary label sequence output by the RNN is "-Lalami-Mifuvudine-Tablets", where "-" indicates that there is no output at the corresponding moment. For example, the transcription layer at the top of the convolutional recurrent neural network is used to convert the prediction result label output by the recurrent layer into the final label sequence, thereby realizing image-based text recognition. For example, the transcription layer can use algorithms such as Connectionist Temporal Classification (CTC) to find the label sequence with the highest probability combination based on the prediction results. For example, the preliminary label sequence "-Lalami-Mifuvudine-Tablets" is converted to "Lamivudine Tablets". For example, as Figure 1 As shown, based on the input image at the bottom of the figure, it can be recognized that the text content in the input image is "Lamivudine Tablets".

[0046] It should be noted that the embodiments of the present disclosure do not limit the specific network structure of the above-mentioned convolutional recurrent neural network. You can refer to other relevant documents and adjust it according to actual conditions.

[0047] In order to improve the accuracy of text recognition, at least one embodiment of the present disclosure provides an image recognition method, including: obtaining an image to be recognized, the image to be recognized including a target text; determining the text content of the target text based on knowledge information and image information of the image to be recognized. At least one embodiment of the present disclosure also provides an image recognition device, an electronic device, and a non-transient readable storage medium corresponding to the above-mentioned image recognition method.

[0048] The image recognition method provided by at least one embodiment of the present disclosure not only considers the image information of the image to be recognized when performing text recognition operations, but also considers the knowledge information from the associated data knowledge base. Combining these two aspects of information for text recognition can improve the accuracy of text recognition.

[0049] In conjunction with the accompanying drawings, the image recognition method provided according to at least one embodiment of the present disclosure is non-restrictively described below through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present disclosure.

[0050] Figure 2A A flowchart of an image recognition method provided by at least one embodiment of the present disclosure, Figure 2B A schematic diagram corresponding to step S102 in an image recognition method provided in at least one embodiment of the present disclosure. Figure 2C A schematic diagram corresponding to step S112 in an image recognition method provided by at least one embodiment of the present disclosure is provided. Figure 3 An example diagram of an image to be recognized provided for at least one embodiment of the present disclosure.

[0051] For example, in at least one embodiment of the present disclosure, the image recognition method 100 can be applied to scenarios such as drug name recognition. Figure 2A As shown, the image recognition method 100 may include the following operations:

[0052] Step S101: obtaining an image to be recognized, where the image to be recognized includes a target text;

[0053] Step S102: Determine the text content of the target text based on the knowledge information and the image information of the image to be recognized.

[0054] It should be noted that, in the embodiments of the present disclosure, steps S101-S102 can be executed sequentially or in other adjusted orders, and some or all of the operations in steps S101-S102 can also be executed in parallel. The embodiments of the present disclosure do not limit the execution order of each step, and can be adjusted according to actual conditions. For example, in the examples of the present disclosure, steps S101-S102 can be executed in a separate server (e.g., a cloud server, etc.) or on a local terminal, and the embodiments of the present disclosure do not limit this. For example, in some examples, the image recognition method 100 provided by implementing at least one embodiment of the present disclosure can selectively execute some of the steps in steps S101-S102, and can also execute some additional steps other than steps S101-S102, and the embodiments of the present disclosure do not specifically limit this.

[0055] For example, in at least one embodiment of the present disclosure, the image to be identified may be an image of a drug to be identified, for example, Figure 3 The medicine box picture shown may also be an image of a bank card, an image of a certificate (e.g., an ID card, a driver's license, a visa, etc.), etc., and the embodiments of the present disclosure do not limit this, and this can be set according to actual conditions. For example, the image to be identified may be an image taken by a sensor such as a camera, an image downloaded from the Internet, or a local image stored in a terminal or server, etc., and the embodiments of the present disclosure do not limit this.

[0056] For example, in the embodiment of the present disclosure, the image to be recognized may be an initial image (such as Figure 3 The medicine box photo shown in FIG. 1 may also be a text region image obtained through a text region detection operation (such as Figure 3 The black rectangular frame image shown in FIG. 1 is not limited to this in the embodiments of the present disclosure. For example, the image to be recognized may be an image containing only a text area (such as Figure 3 For example, when the image to be recognized includes multiple text area images, the image recognition method in some embodiments of the present disclosure processes each of the multiple text area images in turn to output the text contents of multiple target texts.

[0057] For example, in at least one embodiment of the present disclosure, the image to be recognized may include one or more text area images. For example, a text area image refers to an image of the area where the target text appears in the image to be recognized (for example, a text box that only covers the length of the text). Usually for text recognition in complex scenes, it is necessary to first locate the location of the text, that is, to perform text area detection and obtain a text area image, so that the text recognition operation can be performed more efficiently and accurately later. For example, in an embodiment of the present disclosure, obtaining the image to be recognized may include obtaining a text area image in the image to be recognized, so as to perform subsequent text recognition operations based on the text area image, and the embodiments of the present disclosure do not limit this. For example, in one example, the image to be recognized itself is a text area image. For example, in another example, the image to be recognized includes multiple text area images. In this case, the text recognition operation can be performed on the multiple text area images in sequence, and the embodiments of the present disclosure do not limit this.

[0058] For example, in one embodiment, text area detection may be performed on the image to be recognized to obtain a text area image including the target text. For example, in one example, determining the text content of the target text based on the knowledge information and the image information of the image to be recognized may include: determining the text content of the target text based on the knowledge information and the image information of the text area image.

[0059] For example, in one example, a connectionist text proposal network (CTPN) can be used to implement text region detection for an image to be recognized. For example, CTPN includes steps such as detecting small-scale text boxes, cyclically connecting text boxes, and text line edge refinement. It should be noted that in the embodiments of the present disclosure, there is no limitation on the method of text region detection.

[0060] For example, Figure 3The image of the area within the black rectangular box shown in the figure can be used as a text area image. For example, the distance between each side of the text box and the nearest text is within a certain threshold range, for example, the threshold is 2 to 5 mm. For example, when the text box is a rectangle, the distance between the upper and lower sides of the rectangle and the upper and lower edges of a line of text does not exceed a certain threshold, and the distance between the left and right sides of the rectangle and the two side edges of a line of text does not exceed a certain threshold. For example, the threshold can be adjusted as needed, that is, the size of the text detection area can be limited and adjusted according to the method used. It should be noted that the shape of the text area image is not limited to Figure 3 The rectangle shown in the figure may also be other shapes such as a stepped shape.

[0061] For example, in at least one embodiment of the present disclosure, the target text refers to a specific text in the image to be recognized. For example, in one example, when the image to be recognized is Figure 3 When the image of the medicine box is shown, the target text may include the drug name "Lamivudine Tablets", "Lamivudine Tablets", the specification "14 tablets / box", "100 mg", and "National Medicine Standard: H2009XXXX", etc. For example, in another example, when the image to be identified is an image of an ID card, the target text may be the name, gender, date of birth, address, citizen ID number, etc. of the ID card holder. The embodiments of the present disclosure do not limit the specific content of the target text, and it can be set according to actual conditions, such as according to a specific application scenario.

[0062] For example, in at least one embodiment of the present disclosure, knowledge information refers to knowledge information associated with the target text, which comes from a data knowledge base associated with the target text. For example, when the target text is the name of a drug in a picture of a medicine box to be identified, the knowledge information may be drug information from a drug knowledge base. When the image recognition method is applied to other scenarios, the knowledge information may be knowledge information of a corresponding type, and the data knowledge base may be a data knowledge base of a corresponding type. The data knowledge base may be established before executing the image recognition method, and may also be optimized and improved during and after executing the image recognition method.

[0063] Therefore, the image recognition method 100 provided in at least one embodiment of the present disclosure not only considers the image information of the image to be recognized, but also considers the associated knowledge information. Combining these two aspects of information for text recognition can effectively improve the accuracy of text recognition.

[0064] like Figure 2B As shown, for example, in one embodiment of the present disclosure, for the above step S102: determining the text content of the target text based on the knowledge information and the image information of the image to be recognized, the following steps S110-S113 may be included:

[0065] Step S110: extracting features from the image to be identified to obtain target features;

[0066] Step S111: obtaining knowledge representation data based on knowledge information;

[0067] Step S112: fusing the target feature and the knowledge representation data to obtain a fusion result;

[0068] Step S113: Determine the text content of the target text based on the fusion result.

[0069] It should be noted that, in the embodiments of the present disclosure, steps S110 to S113 may be executed sequentially or in other adjusted orders, and some or all of the operations in steps S110 to S113 may also be executed in parallel. The embodiments of the present disclosure do not limit the execution order of the various steps, and may be adjusted according to actual conditions. For example, in some examples, the image recognition method 100 provided by at least one embodiment of the present disclosure may selectively execute some steps of steps S110 to S113, or may execute some additional steps in addition to steps S110 to S113, and the embodiments of the present disclosure do not specifically limit this.

[0070] For example, in an embodiment of the present disclosure, for step S110, a convolutional neural network may be used to extract features of the image to be identified to obtain target features, and the embodiment of the present disclosure is not limited to this.

[0071] For example, in an embodiment of the present disclosure, for step S111, the knowledge representation data may be a vector representation of the corresponding knowledge information. For example, the knowledge representation data may include one or more knowledge representation data, and the embodiment of the present disclosure does not limit this. For example, the knowledge representation data may be a vector representation of drug information from a drug knowledge base. For example, in an embodiment of the present disclosure, various methods may be used to represent structured knowledge information in a data knowledge base in the form of vectors, thereby obtaining multiple knowledge representation data. For example, in one example, the knowledge representation data includes a vector representation of the drug name for subsequent splicing operations. For example, the drug knowledge base includes the names of multiple drugs, and the names of multiple drugs are converted into knowledge representation data in vector form respectively. For example, in one example, based on the knowledge information in the data knowledge base, the TransE algorithm may be used to obtain knowledge representation data in vector form, or other algorithms may be used, and the embodiment of the present disclosure does not specifically limit this.

[0072] For example, in at least one embodiment of the present disclosure, for step S112, the fusion result is based on the fusion or splicing of the image information itself (e.g., target features) of the image to be identified and the knowledge information (e.g., multiple knowledge representation data) of the data knowledge base. For example, in at least one embodiment of the present disclosure, the fusion result can be embodied by splicing features, and the fusion result can also be embodied in other forms, and the embodiments of the present disclosure do not specifically limit this.

[0073] It should be noted that in the embodiments of the present disclosure, the target features (for example, output by a convolutional neural network) can be directly spliced ​​with multiple knowledge representation data to obtain a fusion result; or the target features can be converted into state features (for example, the state features include the hidden states at each moment in the recurrent neural network, etc.) through a recurrent neural network, and then the state features can be spliced ​​with multiple knowledge representation data to obtain a fusion result. The embodiments of the present disclosure are not limited to this.

[0074] For example, in at least one embodiment of the present disclosure, for step S112, Figure 2C As shown, fusing the target feature and the knowledge representation data to obtain a fusion result may include the following steps S131-S132:

[0075] Step 131: Obtaining state features based on target features through a recurrent neural network;

[0076] Step 132: Concatenate the state features and the knowledge representation data to obtain a fusion result.

[0077] For example, in at least one embodiment of the present disclosure, the recurrent neural network may be a unidirectional recurrent neural network, a bidirectional recurrent neural network, a bidirectional long short-term memory network recurrent neural network, etc., and the embodiments of the present disclosure are not limited to this.

[0078] For example, in at least one embodiment of the present disclosure, there are multiple pieces of knowledge representation data. For step S132, the state feature and the multiple pieces of knowledge representation data are concatenated to obtain a fusion result, including:

[0079] Based on the state features and multiple knowledge representation data, a splicing feature of each of the multiple moments is obtained, and the splicing feature is a fusion result.

[0080] Figure 4 A schematic diagram of an image recognition method provided by an embodiment of the present disclosure, Figure 5 In an image recognition method provided in an embodiment of the present disclosure, Figure 2B Schematic diagram of some steps.

[0081] Combine the following Figure 4 and Figure 5, the image recognition method 100 provided by the embodiment of the present disclosure is described in detail. For example, in one example, when the image recognition method 100 is applied to a scenario of identifying drug names, knowledge information from a drug knowledge base can be combined in the text recognition process, such as Figure 4 shown.

[0082] For example, the image recognition method 100 can be used for a first neural network, and the first neural network can include a recurrent neural network (RNN) and a convolutional neural network (CNN). For example, the recurrent neural network (RNN) can be a unidirectional recurrent neural network, a bidirectional recurrent neural network, a bidirectional long short-term memory network recurrent neural network (BiLSTM RNN), etc., and the embodiments of the present disclosure are not limited to this. It should be noted that the first neural network can also include other structures, and the first neural network can also include fewer structures. For example, the first neural network can only include a convolutional neural network CNN, etc., and the embodiments of the present disclosure are not limited to this.

[0083] For example, in one example, for step S110, any convolutional neural network (for example, lightweight network ShuffleNet, MobileNet, etc.) can be used to extract features of the image to be recognized (or the text area image of the image to be recognized) to obtain target features.

[0084] It should be noted that the embodiments of the present disclosure do not limit the selection of convolutional neural networks, which can be determined according to actual needs.

[0085] For example, step S110 may be performed in a convolutional layer of the first neural network. For example, the first neural network may include a Figure 1 The convolutional recurrent neural network shown, combined with Figure 1 As described above, convolutional neural network (CNN) can extract features of the image to be identified and obtain target features, which will not be described here.

[0086] Fig. 6A According to at least one embodiment of the present disclosure Figure 2C Schematic diagram of the process of step S131. Figure 6B According to at least one embodiment of the present disclosure Figure 2C Schematic diagram of the process of step S132.

[0087] For example, in at least one embodiment of the present disclosure, Fig. 6A As shown, for step S131, the state feature is obtained based on the target feature through a recurrent neural network, including the following operations:

[0088] Step S11: using a bidirectional recurrent neural network to determine a moment from multiple moments as the current moment;

[0089] Step S12: Based on the target feature, the hidden state at the current moment is obtained, where the hidden state at the current moment includes the forward hidden state at the current moment and the backward hidden state at the current moment; the hidden state at each moment in the multiple moments is the state feature.

[0090] For example, in at least one embodiment of the present disclosure, Figure 6B As shown, for step S132, based on the state feature and the multiple knowledge representation data, the splicing feature of each moment in the multiple moments is obtained, including the following operations:

[0091] Step S13: determining the fusion feature at the current moment based on the hidden state at the current moment and the plurality of knowledge representation data;

[0092] Step S14: concatenate the hidden state at the current moment with the fusion feature at the current moment to obtain the concatenated feature at the current moment.

[0093] For example, in one example, the recurrent layer can use any recurrent neural network. For example, in one example, the recurrent layer can use a unidirectional recurrent neural network, a bidirectional recurrent neural network, a bidirectional long short-term memory network recurrent neural network (BiLSTM RNN), etc., and the embodiments of the present disclosure are not limited to this. Figure 5 As shown, in one example, the recurrent layer adopts a bidirectional long short-term memory network recurrent neural network, including a forward layer and a reverse layer, and the forward layer and the reverse layer each include a plurality of memory units (such as Figure 5 ). For example, Figure 5 As shown, the forward layer includes multiple memory units connected by right arrows, and the reverse layer includes multiple memory units connected by left arrows. Each memory unit corresponds to a moment and an input feature vector (for example, X1,...Xn shown in the figure). For example, the hidden state at the current moment is related to the hidden state at the previous moment and the hidden state at the next moment. These hidden states can be used to memorize the temporal logic of the text. For example, in LSTM, each memory unit includes a cell state, a forget gate, an input gate, and an output gate. These gate structures allow memory information to pass selectively, and are used to remove or add information to the cell state.

[0094] For example, in one example, a moment is determined as the current moment, the current moment is t, and t is an integer greater than 0. Based on the target feature (such as Figure 5 The feature sequence X1,…Xn shown in the figure can obtain the input Xt corresponding to the current time t. For example, the current time t receives an input Xt and generates the hidden state Ht at the current time t, that is, the state feature. The hidden state Ht includes the forward hidden state at the current time t and the backward hidden state at the current time t For example, in one example, the hidden state Ht at the current time t is the forward hidden state and the backward hidden state at the current time t splicing, that is For example, in one example, the forward hidden state at time t is The dimension is set to 256, and the backward hidden state at the current time t The dimension of is set to 256, then the dimension of the hidden state Ht at the current time t is 512.

[0095] For example, in one example, Figure 5 As shown, in order to integrate the knowledge information of the data knowledge base, the knowledge information in the data knowledge base is converted into a knowledge representation vector with the same dimension as the forward hidden state and the backward hidden state at the current moment, that is, the knowledge representation data. For example, multiple knowledge representation data are as follows Figure 5 For example, the forward hidden state at the current time t is The dimension is set to 256, and the backward hidden state at the current time t When the dimension of is set to 256, the dimension of the knowledge representation data is 256. For example, Figure 5 As shown, the solid black circle represents the image information of the image to be recognized (i.e. ) and the knowledge information (e1, e2,…, eN) represented by the knowledge representation data.

[0096] For example, Figure 5 As shown, based on the hidden state at the current time t and multiple knowledge representation data (e1, e2, …, eN), determine the fusion feature (Et) at the current time t, and then convert the hidden state of the current time t And the fusion feature (Et) at the current time t is spliced ​​to obtain the splicing feature ( Et).

[0097] Figure 6C According to at least one embodiment of the present disclosure Figure 6B For example, in at least one embodiment of the present disclosure, for step S13, based on the hidden state at the current moment and the plurality of knowledge representation data, determining the fusion feature at the current moment may include the following operations:

[0098] Step S21: based on the hidden state at the current moment, determining M pieces of knowledge representation data having matching association with the hidden state at the current moment from a plurality of pieces of knowledge representation data;

[0099] Step S22: Determine the fusion feature at the current moment based on the M knowledge representation data and the hidden state at the current moment.

[0100] For example, M is an integer greater than 0, and M is less than or equal to the total number N of the plurality of knowledge representation data (e1, e2 . . . eN) included in the data knowledge base. For example, in one example, M=5, N=50.

[0101] Fig.6D According to at least one embodiment of the present disclosure Figure 6C For example, in at least one embodiment of the present disclosure, Fig.6D As shown, for step S21, based on the hidden state at the current moment, M pieces of knowledge representation data having matching associations with the hidden state at the current moment are determined from a plurality of pieces of knowledge representation data, including the following operations:

[0102] Step S31: Calculate the similarity parameter between each of the plurality of knowledge representation data and the hidden state at the current time t;

[0103] Step S32: Based on the similarity parameter, determine M knowledge representation data.

[0104] For example, in one example, the similarity parameter sim can be calculated by the following expression (1):

[0105]

[0106] Wherein, ej represents the jth knowledge representation data among multiple knowledge representation data (e1, e2, …, eN), t represents the current time, and the superscript T represents transposition. represents the forward hidden state at the current time t, represents the backward hidden state at the current time t, W represents the parameter matrix, sim(j) represents the jth knowledge representation data in multiple knowledge representation data (e1, e2, …, eN) and the hidden state at the current time t Similarity parameter of , where t is an integer greater than 0 and j is an integer greater than 0.

[0107] For example, in one example, the forward hidden state at time t is is a 256×1 matrix, the backward hidden state at the current time t is a 256×1 matrix, and the knowledge representation data ej is a 256×1 matrix, then is a 1×512 matrix, and the parameter matrix W is a 512×256 matrix. The parameter matrix W can be obtained through training based on the training data set.

[0108] For example, in one example, the M pieces of knowledge representation data with the largest similarity parameters calculated according to expression (1) are taken as the M pieces of knowledge representation data (r1, r2, ..., rM) having matching association with the hidden state Ht at the current time t.

[0109] For example, in another example, the knowledge representation data whose similarity parameters calculated according to expression (1) are greater than the preset threshold are used as M knowledge representation data (r1, r2, ..., rM) having a matching association with the hidden state Ht at the current time t. For example, the preset threshold refers to the threshold of the similarity parameter. For example, in one example, the preset threshold can range from 2 to 7. For example, in one example, when the preset threshold is 4, if 3 similarity parameters calculated according to expression (1) are greater than the preset threshold 4, then M = 3, and if 7 similarity parameters calculated according to expression (1) are greater than the preset threshold 4, then M = 7. The embodiments of the present disclosure do not impose specific restrictions on this.

[0110] For example, in at least one embodiment of the present disclosure, since LSTM is adopted, the hidden state at the current moment t aggregates the state information of the previous moment, and the matching relevance of the knowledge representation data with a high similarity to the hidden state of the previous moment will also be high. For example, in one example, assume that the drug name is four characters "Metformin Tablets". It should be noted that in this example, it is assumed that in an ideal state, the first character is output at the first moment, the second character is output at the second moment, the third character is output at the third moment, and the fourth character is output at the fourth moment. However, in the embodiments of the present disclosure, the moment and the output character are not necessarily in a one-to-one correspondence. It is possible that the same character is output at multiple moments (for example, the first character "Er" is output at multiple moments), or there may be no character at some moments. The embodiments of the present disclosure do not limit this. For example, assume that the character recognized at the current moment is "Jia" (for example, the second moment is the current moment). At this time, the hidden state information at the current moment includes the hidden state information of the previous moments, that is, the information of the two characters "Er Jia". Then the similarity to the knowledge representation data "Metformin Tablets" from the drug knowledge base is generally greater than that of the knowledge representation data "Prednisone Tablets", and for example, also greater than that of the knowledge representation data "Pangolin Tablets". Then at the next moment, assuming that the third character is to be recognized at the next moment, the probability of the character "Shuang" is greater than that of other characters. For example, in one example, through expression (1), the similarity parameter between the knowledge representation data "Metformin Tablets" and the hidden state at the current moment can be calculated as 9.88; the similarity parameter between the knowledge representation data "Pangolin Tablets" and the hidden state at the current moment is 6.15; the similarity parameter between the knowledge representation data "Prednisone Tablets" and the hidden state at the current moment is 2.15. Because according to the similarity parameters between multiple knowledge representation data from the drug knowledge base and the hidden state at the current moment, the similarity of "Metformin Tablets" is the largest, which means that the recognized drug is likely to be "Metformin Tablets". Therefore, the probability of the third character being "Shuang" is relatively high. For example, since in the drug knowledge base, there may be multiple drug names with the same prefix. For example, the first four characters of "Metformin Tablets" and "Metformin Enteric-coated Tablets" are the same. Then in the first four states, it may be difficult to determine which one it is. Therefore, in the embodiments of the present disclosure, when determining M knowledge representation data (r1, r2,..., rM) with matching relevance to the hidden state Ht at the current moment t from multiple knowledge representation data (e1, e2... en), M can be selected as an integer greater than 4. For example, M = 5, M = 10, etc. Thus, similar drug names (for example, drug names with a large similarity parameter) can be selected as much as possible for subsequent calculation of the fusion feature. In the embodiments of the present disclosure, the value of M can be determined according to the actual situation.

[0111] It should be noted that, in the embodiments of the present disclosure, the above-mentioned matching correlation may refer to a certain similarity between two feature vectors, or may refer to other pre-set conditions, which can be set according to actual needs, and the embodiments of the present disclosure do not limit this.

[0112] For example, in at least one embodiment of the present disclosure, based on the determined M knowledge representation data and the hidden state at the current moment, the fusion feature at the current moment is determined.

[0113] For example, in one example, the fusion feature Et at the current time t is determined by the following expression (2):

[0114]

[0115] Wherein, t represents the current moment, Et represents the fusion feature at the current moment, rj represents the j-th knowledge representation data among M knowledge representation data, sim(j) represents the similarity parameter between the j-th knowledge representation data among M knowledge representation data and the hidden state at the current moment, sim(k) represents the similarity parameter between the k-th knowledge representation data among M knowledge representation data and the hidden state at the current moment, where t, j, and k are integers greater than 0.

[0116] For example, in one example, the hidden state at the current time t is The fusion feature (Et) at the current time t calculated by the above expression (2) is concatenated to obtain the concatenated feature ( Et). For example, in one example, a concat function may be used to perform a concatenation operation between the hidden state at the current moment and the fused feature at the current moment, or other concatenation methods may be used, which is not limited in the embodiments of the present disclosure. For example, the concatenated feature is a fusion result, that is, the concatenated feature not only includes information from the target feature (i.e., a feature sequence extracted based on the input image) (e.g., embodied in the hidden state Ht), but also includes information from the knowledge representation data (i.e., based on a data knowledge base associated with the target text) (e.g., embodied in the fused feature Et).

[0117] For example, in one example, based on the above method, the splicing features of each moment in multiple moments are obtained ( Et), for example, the prediction result corresponding to each moment can be obtained through the fully connected layer and the Softmax layer (for example, the classification network) included in the first neural network, for example, a character sequence (character 1, ..., character n), etc., such as Figure 5 shown.

[0118] For example, in one example, based on the prediction results (such as Figure 5For example, in one example, the prediction result output by the recurrent layer in the first neural network, such as the above-mentioned label sequence, is input into the transcription layer in the first neural network. For example, the transcription layer adopts the CTC algorithm, and finally the transcription layer outputs the recognized text content, such as the text "lamivudine tablets". Figure 4 shown.

[0119] For example, at least one embodiment of the present disclosure further provides a training method for a first neural network, wherein the first neural network includes a recurrent neural network and a convolutional neural network. The first neural network can be used to: obtain an image to be identified, wherein the image to be identified includes a target text; and determine the text content of the target text based on knowledge information and image information of the image to be identified. For example, in one example, the training method includes: inputting training data into the first neural network to obtain output data output from the first neural network, for example, in one example, the training data includes a sample image, and the output data includes sample text content. For example, the training method also includes using a back-propagation algorithm to modify the parameters of the bidirectional recurrent neural network and the parameters of the convolutional neural network based on a loss function.

[0120] For example, in one example, Figure 4 The convolutional neural network, the bidirectional recurrent neural network and the parameter matrix W in the first neural network shown need to be learned through training data. For example, in one example, assuming that xi is a sample image and yi is the corresponding sample text content, for example, when xi is Figure 4 The corresponding yi of the input image shown is "lamivudine tablets". Then the training data set D is {(xi,yi),i=1,…,Q}, where Q is the number of training data. For example, in one example, the above parameters can be adjusted and optimized by back propagation through the loss function. For example, in one example, the formula of the loss function is:

[0121] Loss=-∑ (xi,yi)∈D P(yi|xi),

[0122] Among them, P is the conditional probability, and the goal of this loss function is to minimize the true conditional probability.

[0123] It should be noted that in the embodiments of the present disclosure, the loss function may adopt any type of function and is not limited to the above-mentioned loss function. This may be determined according to actual needs, and the embodiments of the present disclosure do not impose any restrictions on this.

[0124] For example, in the embodiments of the present disclosure, various methods may be used to obtain the image to be identified, such as photographing it through a sensor such as a camera, downloading it from a network such as the Internet, reading it from the local memory of a terminal or server, etc. The embodiments of the present disclosure are not limited to this.

[0125] For example, in at least one embodiment of the present disclosure, a connectionist text proposal network (CTPN) can be used to implement text region detection of the image to be recognized. CTPN is a fully convolutional network that takes images of any size as input and can directly locate text lines in the convolutional layer. It should be noted that in the embodiments of the present disclosure, other text detection methods (for example, R-CNN (regions with CNN)), Fast R-CNN, region proposal network (RPN), etc.) can also be used to implement text region detection of the image to be recognized, thereby obtaining a text region image including the target text. The embodiments of the present disclosure do not limit the text detection method, as long as the text detection function can be implemented and a text region image including the target text can be obtained.

[0126] For example, in at least one embodiment of the present disclosure, the image recognition method 100 may further include: outputting the text content of the target text. For example, the text content may be displayed on a screen for a user to view, or the text content may be transmitted to a separately provided unit or module to implement further processing based on the text content. The text content may be output in any manner, and the embodiments of the present disclosure do not limit this.

[0127] Figure 7 A schematic diagram of an example of using an image recognition method provided by at least one embodiment of the present disclosure for drug name recognition. For example, in combination with the above-mentioned detailed description of the image recognition method 100, in the case of detecting a text area in an image to be recognized, such as Figure 7 The image recognition method for drug name recognition shown includes: receiving a medicine box picture to be recognized; performing text area detection on the medicine box picture to obtain a text area image including the target text; performing text recognition on the text area image to determine the text content of the target text. The specific text recognition process can be referred to the above content and will not be repeated here. Finally, the drug name is output.

[0128] At least one embodiment of the present disclosure also provides an image recognition device, which, when performing image-based text recognition, not only considers the image information itself of the image to be recognized, but also considers the associated knowledge information, which can effectively improve the accuracy of text recognition.

[0129] Figure 8 A schematic block diagram of an image recognition device provided by at least one embodiment of the present disclosure. Figure 8 As shown, the image recognition device 80 includes: an acquisition module 801 and a determination module 802 .

[0130] For example, in one example, the acquisition module 801 is configured to acquire an image to be recognized, the image to be recognized includes target text. For example, the acquisition module 801 can implement step S101, and its specific implementation method can refer to the relevant description of step S101, which will not be repeated here.

[0131] For example, in one example, the determination module 802 is configured to determine the text content of the target text based on the knowledge information and the image information of the image to be recognized. For example, the determination module 802 can implement step S102, and its specific implementation method can refer to the relevant description of step S102, which will not be repeated here.

[0132] It should be noted that these acquisition modules 801 and determination modules 802 may be implemented by software, hardware, firmware or any combination thereof, for example, they may be implemented as an acquisition circuit 801 and a determination circuit 802 respectively, and the embodiments of the present disclosure do not limit their specific implementations.

[0133] It should be understood that the image recognition device 80 provided in the embodiment of the present disclosure can implement the aforementioned image recognition method 100, and can also achieve technical effects similar to the aforementioned image recognition method 100, which will not be elaborated herein.

[0134] It should be noted that in the embodiments of the present disclosure, the image recognition device 80 may include more or less software, hardware, and firmware, and the connection relationship between the various software, hardware, and firmware is not limited and can be determined according to actual needs. The specific configuration of each software, hardware, and firmware is not limited and can be composed of a digital chip, a processor combined with a memory, or other applicable methods.

[0135] Fig. 9 A schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure. Fig. 9 As shown, the electronic device 90 includes a processor 910 and a memory 920. The memory 920 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 910 is used to run non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 910, one or more steps in the image recognition method described above can be executed. The memory 920 and the processor 910 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0136] For example, the processor 910 may be a central processing unit (CPU), a digital signal processor (DSP), or other forms of processing units with data processing capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 910 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 90 to perform desired functions.

[0137] For example, the memory 920 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 910 may run one or more computer program modules to implement various functions of the electronic device 90. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.

[0138] It should be noted that, in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 90 can refer to the above description of the image recognition method 100, which will not be repeated here.

[0139] Fig.10 A schematic block diagram of another image recognition device provided by at least one embodiment of the present disclosure. The image recognition device 400 is suitable for implementing the image recognition method provided by the embodiment of the present disclosure. The image recognition device 400 may be a user terminal, etc. It should be noted that: Fig.10 The image recognition device 400 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Fig.10As shown, the image recognition device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 410, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 420 or a program loaded from a storage device 480 to a random access memory (RAM) 430. In the RAM 430, various programs and data required for the operation of the image recognition device 400 are also stored. The processing device 410, the ROM 420, and the RAM 430 are connected to each other via a bus 440. An input / output (I / O) interface 450 is also connected to the bus 440.

[0141] Typically, the following devices may be connected to the I / O interface 450: an input device 460 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 470 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 480 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 490. The communication device 490 may allow the image recognition device 400 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig.10 The image recognition device 400 is shown to have various devices, but it should be understood that it is not required to implement or have all the devices shown, and the image recognition device 400 may alternatively implement or have more or fewer devices.

[0142] For example, the image recognition method provided by the embodiment of the present disclosure can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned image recognition method 100. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 490, or installed from the storage device 480, or installed from the ROM 420. When the computer program is executed by the processing device 410, the image recognition method 100 provided by the embodiment of the present disclosure can be executed.

[0143] At least one embodiment of the present disclosure further provides a non-transient readable storage medium for storing non-transient computer-readable instructions, which can implement the image recognition method described in any embodiment of the present disclosure when the non-transient computer-readable instructions are executed by a computer. By using the storage medium, when performing image-based text recognition, the image information of the image to be recognized and the associated knowledge information can be comprehensively considered, thereby effectively improving the accuracy of text recognition.

[0144] Fig.11 A schematic diagram of a non-transitory readable storage medium provided for at least one embodiment of the present disclosure. Fig.11As shown, the storage medium 500 is used to store non-transitory computer-readable instructions 510. For example, when the non-transitory computer-readable instructions 510 are executed by a computer, one or more steps in the image recognition method 100 described above can be performed.

[0145] For example, the storage medium 500 may be applied to the above-mentioned image recognition devices 90 and 400. Fig. 9 The memory 920 in the electronic device 90 shown. For example, the relevant description of the storage medium 500 can refer to Fig. 9 The corresponding description of the memory 920 in the electronic device 90 is shown and will not be repeated here.

[0146] There are a few points to note:

[0147] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to the general design.

[0148] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0149] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An image recognition method, comprising: Acquire an image to be recognized, wherein the image to be recognized includes target text; Determining the text content of the target text based on the knowledge information and the image information of the image to be recognized; Wherein, determining the text content of the target text based on the knowledge information and the image information of the image to be identified includes: extracting features of the image to be identified to obtain target features, obtaining knowledge representation data based on the knowledge information, fusing the target features and the knowledge representation data to obtain a fusion result, and determining the text content of the target text based on the fusion result; Among them, fusing the target feature and the knowledge representation data to obtain the fusion result includes: obtaining a state feature based on the target feature through a recurrent neural network, and splicing the state feature and the knowledge representation data to obtain the fusion result.

2. The image recognition method according to claim 1, wherein: The state feature and the knowledge representation data are combined to obtain the fusion result, including: Based on the state feature and the knowledge representation data, a splicing feature of each moment in a plurality of moments is obtained, wherein the splicing feature is the fusion result.

3. The image recognition method according to claim 1 or 2, wherein: Obtaining the state feature based on the target feature through the recurrent neural network includes: A bidirectional recurrent neural network is used to determine a moment among multiple moments as the current moment; Based on the target feature, obtaining the hidden state at the current moment, wherein the hidden state at the current moment includes the forward hidden state at the current moment and the backward hidden state at the current moment; The hidden state at each of the multiple moments is the state feature.

4. The image recognition method according to claim 3, wherein: Based on the state feature and the knowledge representation data, a splicing feature of each of the multiple moments is obtained, including: Determining a fusion feature at the current moment based on the hidden state at the current moment and the knowledge representation data; The hidden state at the current moment and the fusion feature at the current moment are spliced ​​to obtain the splicing feature at the current moment.

5. The image recognition method according to claim 4, wherein: Determining the fusion feature at the current moment based on the hidden state at the current moment and the knowledge representation data includes: Based on the hidden state at the current moment, determining M pieces of knowledge representation data having a matching association with the hidden state at the current moment from a plurality of pieces of knowledge representation data included in the knowledge representation data; Determine the fusion feature at the current moment based on the M knowledge representation data and the hidden state at the current moment; Wherein, M is an integer greater than 0.

6. The image recognition method according to claim 5, wherein: Based on the hidden state at the current moment, determining the M pieces of knowledge representation data having a matching association with the hidden state at the current moment from the plurality of knowledge representation data, comprising: Calculating a similarity parameter between each of the plurality of knowledge representation data and the hidden state at the current moment; Based on the similarity parameter, the M knowledge representation data are determined.

7. The image recognition method according to claim 6, wherein: The expression of the similarity parameter is as follows: Wherein, ej represents the jth knowledge representation data among the plurality of knowledge representation data, t represents the current time, and the superscript T represents transposition. represents the forward hidden state at the current moment, represents the backward hidden state at the current moment, W represents the parameter matrix, sim(j) represents the similarity parameter between the j-th knowledge representation data and the hidden state at the current moment, t is an integer greater than 0, and j is an integer greater than 0.

8. The image recognition method according to claim 6 or 7, wherein: Based on the similarity parameter, determining the M knowledge representation data includes: The M pieces of knowledge representation data with the largest similarity parameters are used as the M pieces of knowledge representation data having matching association with the hidden state at the current moment, or The knowledge representation data with similarity parameters greater than a preset threshold are taken as the M knowledge representation data having matching association with the hidden state at the current moment.

9. The image recognition method according to claim 6, wherein: The expression of the fusion feature at the current moment is as follows: Among them, t represents the current moment, Et represents the fusion feature of the current moment, ej represents the j-th knowledge representation data among the M knowledge representation data, sim(j) represents the similarity parameter between the j-th knowledge representation data among the M knowledge representation data and the hidden state at the current moment, sim(k) represents the similarity parameter between the k-th knowledge representation data among the M knowledge representation data and the hidden state at the current moment, wherein t, j, and k are integers greater than 0.

10. The image recognition method according to claim 2, wherein: Based on the fusion result, determining the text content of the target text includes: Based on the splicing features, output a prediction result corresponding to each of the multiple moments; Based on the prediction result, the text content of the target text is determined.

11. The image recognition method according to claim 1, wherein: Extracting features from the image to be identified to obtain the target features includes: A convolutional neural network is used to extract features of the image to be identified to obtain the target features.

12. The image recognition method according to claim 1, further comprising: Perform text region detection on the image to be recognized to obtain a text region image including the target text.

13. The image recognition method according to claim 12, wherein: Determining the text content of the target text based on the knowledge information and the image information of the image to be recognized includes: Based on the knowledge information and the image information of the text region image, the text content of the target text is determined.

14. The image recognition method according to claim 1, further comprising: Output the text content of the target text.

15. The image recognition method according to claim 1, wherein: The image to be identified includes an image of a medicine to be identified.

16. The image recognition method according to claim 1, wherein: The knowledge information comes from a data knowledge base, and the data knowledge base includes a drug knowledge base.

17. A method for training a first neural network for image recognition, wherein the first neural network comprises a recurrent neural network and a convolutional neural network, The first neural network is used to: Get the image to be recognized, where: The image to be recognized includes target text; Determining the text content of the target text based on the knowledge information and the image information of the image to be recognized; Wherein, determining the text content of the target text based on the knowledge information and the image information of the image to be identified includes: extracting features of the image to be identified to obtain target features, obtaining knowledge representation data based on the knowledge information, fusing the target features and the knowledge representation data to obtain a fusion result, and determining the text content of the target text based on the fusion result; The target feature and the knowledge representation data are fused to obtain the fusion result, including: obtaining a state feature based on the target feature through a recurrent neural network, and splicing the state feature and the knowledge representation data to obtain the fusion result; The training method comprises: Inputting training data into the first neural network to obtain output data outputted from the first neural network; Using a back propagation algorithm, based on a loss function, parameters of the recurrent neural network and parameters of the convolutional neural network are modified.

18. An image recognition device, comprising: An acquisition module is configured to acquire an image to be recognized, wherein the image to be recognized includes a target text; A determination module, configured to determine the text content of the target text based on the knowledge information and the image information of the image to be recognized; Wherein, determining the text content of the target text based on the knowledge information and the image information of the image to be identified includes: extracting features of the image to be identified to obtain target features, obtaining knowledge representation data based on the knowledge information, fusing the target features and the knowledge representation data to obtain a fusion result, and determining the text content of the target text based on the fusion result; Among them, fusing the target feature and the knowledge representation data to obtain the fusion result includes: obtaining a state feature based on the target feature through a recurrent neural network, and splicing the state feature and the knowledge representation data to obtain the fusion result.

19. An electronic device comprising: processor; a memory including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include modules for implementing the image recognition method described in any one of claims 1-16.

20. A non-transitory readable storage medium for storing computer-readable instructions, which, when executed by a computer, can implement the image recognition method described in any one of claims 1 to 16.

Images