A text recognition method, apparatus, device and medium

Abstract

Embodiments of the present disclosure relate to a text recognition method, device, equipment and medium, wherein the method comprises: determining a target shooting mode based on a to-be-recognized instrument type to shoot the to-be-recognized instrument, obtaining a to-be-recognized instrument image, performing multi-scale feature extraction on the to-be-recognized instrument image to obtain a plurality of feature maps, and performing recognition on each feature map to obtain anchor box information corresponding to each character; calculating a plurality of character features between any two characters based on the anchor box information corresponding to each character; processing the plurality of character features between any two characters and a preset machine learning model to obtain a candidate string set; and matching the candidate string set and a preset instrument string set to determine instrument information corresponding to the to-be-recognized instrument. The above technical solution does not limit the shape and direction of the text presented on the instrument, and can recognize the text on the instrument, thereby improving the text matching efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a text recognition method and device, equipment and medium. BACKGROUND

[0002] At present, medical devices refer to instruments, equipment, appliances, in vitro diagnostic reagents and calibrators, materials and other similar or related items that are directly or indirectly used for the human body. Medical devices are usually printed with batch number, serial number and material number and other information when they leave the factory, representing different batches, types and materials of devices. However, some small-sized devices such as bone nails and bone plates are often contained in medical devices, and the printed text information is difficult to be directly observed by the human eye and ordinary cameras, and usually needs to be manually recognized by professional camera equipment. When the devices are put in and out of the warehouse, they need to be compared and recorded with the device information in the existing database, and the whole process is tedious, time-consuming and labor-intensive.

[0003] In related technologies, the optical character recognition (OCR) technology is relatively mature, but most methods are based on document images, the text characters are in the forward direction, and the characters are arranged in order from left to right. When facing "wrap-around" characters or characters in any direction, the text cannot be effectively and accurately detected and recognized. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a text recognition method, device, equipment and medium.

[0005] The text recognition method provided by the present disclosure comprises:

[0006] Based on the type of the to-be-recognized device, a target shooting mode is determined to shoot the to-be-recognized device, and a to-be-recognized device image is obtained;

[0007] Multi-scale feature extraction is performed on the to-be-recognized device image to obtain a plurality of feature maps, and each feature map is recognized to obtain anchor box information corresponding to each character;

[0008] Based on the anchor box information corresponding to each character, a plurality of character features between any two characters are calculated, and based on the plurality of character features between any two characters and a preset machine learning model, a candidate string set is obtained;

[0009] Based on the candidate string set and a preset device string set, matching is performed to determine device information corresponding to the to-be-recognized device.

[0010] The embodiment of the present disclosure further provides a text recognition device, the device comprises:

[0011] A photographing module is configured to photograph the instrument to be recognized according to the target photographing mode determined based on the type of the instrument to be recognized, to obtain an instrument image to be recognized.

[0012] An extraction module is configured to perform multi-scale feature extraction on the instrument image to be recognized, to obtain a plurality of feature maps.

[0013] An identification module is configured to identify each of the feature maps, to obtain anchor box information corresponding to each character.

[0014] A calculation module is configured to calculate a plurality of character features between any two characters based on the anchor box information corresponding to each character.

[0015] A processing module is configured to process the plurality of character features between any two characters and a preset machine learning model, to obtain a candidate character string set.

[0016] A matching module is configured to match the candidate character string set and a preset instrument character string set, to determine instrument information corresponding to the instrument to be recognized.

[0017] The embodiment of the present disclosure further provides an electronic device, which comprises a processor, a memory for storing executable instructions of the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the text recognition method provided by the embodiment of the present disclosure.

[0018] The embodiment of the present disclosure further provides a computer readable storage medium, which stores a computer program for executing the text recognition method provided by the embodiment of the present disclosure.

[0019] Compared with the prior art, the technical solution provided by the embodiment of the present disclosure has the following advantages: the text recognition scheme provided by the embodiment of the present disclosure photographs the instrument to be recognized according to the target photographing mode determined based on the type of the instrument to be recognized, to obtain an instrument image to be recognized, performs multi-scale feature extraction on the instrument image to be recognized, to obtain a plurality of feature maps, identifies each of the feature maps, to obtain anchor box information corresponding to each character, calculates a plurality of character features between any two characters based on the anchor box information corresponding to each character, processes the plurality of character features between any two characters and a preset machine learning model, to obtain a candidate character string set, and matches the candidate character string set and a preset instrument character string set, to determine instrument information corresponding to the instrument to be recognized. By using the above technical solution, the shape and direction of the text presented on the instrument are not limited, and the text on the instrument can be recognized, to improve the text matching efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, aspects and advantages of embodiments of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or like reference numerals will be understood to refer to the same or like elements, features and structures. It should be noted that the drawings are in simplified form and not to precise scale, but are intended to conceptually illustrate the ideas underlying the present disclosure.

[0021] Figure 1 A flowchart of a text recognition method provided by an embodiment of the present disclosure;

[0022] Figure 2 A flowchart of another text recognition method provided by an embodiment of the present disclosure;

[0023] Figure 3 A structural diagram of a text recognition apparatus provided by an embodiment of the present disclosure;

[0024] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail by referring to the drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of the present disclosure.

[0026] It should be understood that the various steps in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions will be given throughout the description.

[0028] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0031] In the prior art, the specific type of characters of the seal can be recognized, and the non-seal type of characters cannot be recognized. Or, the collected image is rotated and corrected, which is based on image level correction. When the characters in the image have multiple angle orientations, they cannot be effectively detected and recognized, such as "wrap-around" characters. Or, the text line is detected, and the single character is corrected according to the text line direction. However, when the characters in the image have multiple angle orientations, there is no concept of text line, and thus they cannot be effectively detected and recognized, such as "wrap-around" characters. Therefore, the text recognition method in the prior art has certain limitations on the shape and direction of the characters, i.e., only specific character presentation methods can be recognized, which cannot meet the text recognition needs of medical devices and the like.

[0032] To solve the above problems, the present disclosure provides a text recognition method. The method includes: determining a target shooting mode based on a type of an instrument to be recognized, and shooting the instrument to be recognized to obtain an instrument image to be recognized; performing multi-scale feature extraction on the instrument image to be recognized to obtain a plurality of feature maps; identifying each feature map to obtain anchor box information corresponding to each character; calculating a plurality of character features between any two characters based on the anchor box information corresponding to each character; processing the plurality of character features between any two characters and a preset machine learning model to obtain a candidate string set; and matching the candidate string set and a preset instrument string set to determine instrument information corresponding to the instrument to be recognized. The above technical solution does not limit the shape and direction of the characters on the instrument, and can recognize the characters on the instrument, thereby improving the text matching efficiency and accuracy.

[0033] Specifically, Figure 1 A flowchart of a text recognition method provided by the embodiments of the present disclosure is shown in FIG. 1. The method can be executed by a text recognition device. The device can be implemented by software and / or hardware, and can be integrated in an electronic device. As shown in FIG. 1, the method includes the following steps. Figure 1

[0034] Step 101: determining a target shooting mode based on a type of an instrument to be recognized, and shooting the instrument to be recognized to obtain an instrument image to be recognized.

[0035] ​The to-be-identified instrument generally refers to an instrument, a device, a tool, an in-vitro diagnostic reagent and a calibrator, a material, and other similar or related articles, such as a bone nail, a bone plate, and the like, which are directly or indirectly used for a human body. The to-be-identified instrument type refers to one or more of size, shape, and text position information of the to-be-identified instrument.

[0036] In the embodiments of the present disclosure, a plurality of shooting modes can be set, and a corresponding target shooting mode is determined according to different to-be-identified instrument types to shoot the to-be-identified instrument, so as to obtain a to-be-identified instrument image, thereby further improving the accuracy of subsequent text recognition.

[0037] In the embodiments of the present disclosure, there are many ways to determine a target shooting mode based on the to-be-identified instrument type to shoot the to-be-identified instrument and obtain a to-be-identified instrument image. In some embodiments, a first instrument size, a first instrument shape, and first text position information of the to-be-identified instrument are determined based on the to-be-identified instrument type. When the first instrument size is less than a preset first target size, the first instrument shape meets a preset first shape type, and the first text position information meets a preset first text setting, a shooting device is controlled to shoot the to-be-identified instrument in a target shooting mode with a set fixed time and a fixed moving direction, and a to-be-identified instrument image is obtained.

[0038] In other embodiments, a second instrument size, a second instrument shape, and second text position information of the to-be-identified instrument are determined based on the to-be-identified instrument type. When the second instrument size is greater than a preset second target size, the second instrument shape meets a preset second shape type, and the second text position information meets a preset second text setting, the to-be-identified instrument is placed in a target manner, and the shooting device is controlled to shoot the to-be-identified instrument in a target shooting mode with a set multiple times of acquisition, and a to-be-identified instrument image is obtained.

[0039] The above two ways are only examples of determining a target shooting mode based on the to-be-identified instrument type to shoot the to-be-identified instrument and obtain a to-be-identified instrument image. The embodiments of the present disclosure do not limit the specific implementation manner of determining a target shooting mode based on the to-be-identified instrument type to shoot the to-be-identified instrument and obtain a to-be-identified instrument image.

[0040] In step 102, multi-scale feature extraction is performed on the to-be-identified instrument image to obtain a plurality of feature maps, and each feature map is identified to obtain anchor box information corresponding to each character.

[0041] The anchor box information corresponding to each character includes an anchor box center point horizontal coordinate, an anchor box center point vertical coordinate, a length, a width, a rotation angle, and a character category probability of each character.

[0042] Specifically, after obtaining the to-be-recognized instrument image, multi-scale feature extraction can be performed on the to-be-recognized instrument image to obtain a plurality of feature maps. In the embodiment of the present disclosure, the multi-scale feature extraction can be performed on the to-be-recognized instrument image by using a related neural network to obtain a plurality of feature maps. For example, a feature extraction network composed of a plurality of convolution layers, a batch normalization layer and an activation layer performs multi-scale feature extraction on the to-be-recognized instrument image to obtain a plurality of feature maps. The multi-scale can be selected and set according to the application scenario, for example, three different scales are used for feature extraction to obtain three feature maps, and the embodiment of the present disclosure does not make specific limitations.

[0043] Further, after obtaining the plurality of feature maps and performing recognition on each feature map to obtain anchor box information corresponding to each character, in the embodiment of the present disclosure, each feature map can also be recognized by using a related neural network to obtain anchor box information corresponding to each character. For example, an identification network composed of a plurality of convolution layers, a batch normalization layer and an activation layer performs identification on each feature map to obtain anchor box information corresponding to each character. In one specific implementation, each feature map is identified by using a related neural network layer to obtain each pixel position coordinate on each feature map, generate anchor box coordinates corresponding to the coordinates of the to-be-recognized instrument image, and obtain a character class probability corresponding to each anchor box coordinate. Based on the anchor box coordinates and the character class probability corresponding to each anchor box coordinate, the anchor box information corresponding to each character is obtained.

[0044] In step 103, a plurality of character features between any two characters are calculated based on the anchor box information corresponding to each character, and a candidate string set is obtained by processing the plurality of character features between any two characters and a preset machine learning model.

[0045] The plurality of character features refer to one or more of a character size feature, a character direction feature and a character center distance feature.

[0046] In the embodiment of the present disclosure, there are many ways to calculate the plurality of character features between any two characters based on the anchor box information corresponding to each character. In some implementations, the length coordinate and the width coordinate in the anchor box information corresponding to each character are used to calculate the character size feature between any two characters, the rotation angle in the anchor box information corresponding to each character is used to calculate the character direction feature between any two characters, and the anchor box center point horizontal coordinate and the anchor box center point vertical coordinate in the anchor box information corresponding to each character are used to calculate the character center distance feature between any two characters. The character size feature, the character direction feature and the character center distance feature between any two characters are used as the plurality of character features between any two characters.

[0047] In some embodiments, the anchor box information corresponding to any two characters is input into a preset calculation formula to calculate character size features, character direction features and character center distance features between the two characters as a plurality of character features between the two characters.

[0048] Further, the plurality of character features between the two characters and a preset machine learning model are processed to obtain a candidate character string set; the machine learning model is a pre-trained model, and the plurality of character features between the two characters are input into the machine learning model to obtain the candidate character string set. Specifically, a first character without a successor is taken from all characters, a second character without a predecessor is taken from the remaining characters, the plurality of character features between the first character and the second character are input into the machine learning model to obtain an association result of the first character and the second character, and it is determined whether the second character contains the first character. If the association result is associated and the second character contains the first character, the first character and the second character are arranged in order, the second character is taken as the first character, and the process is repeated until all characters without predecessors are traversed to obtain a candidate character sequence. In the case where the candidate character sequence forms a ring, the two adjacent characters with the maximum character distance are disconnected to obtain the candidate character string set.

[0049] Step 104, matching based on the candidate character string set and a preset instrument character string set to determine the instrument information corresponding to the instrument to be recognized.

[0050] In some embodiments, the instrument character string set corresponding to each instrument is stored in advance. In the embodiments of the present disclosure, the candidate character string set and the instrument character string set are matched to determine the instrument information corresponding to the instrument to be recognized. Specifically, each substring in the candidate character string set is compared with the substrings in the instrument character string set. If each substring in the candidate character string set can be successfully matched, it is determined that the text recognition information of the instrument to be recognized is accurate, and the text information corresponding to the candidate character string set is taken as the instrument information corresponding to the instrument to be recognized.

[0051] The text recognition scheme provided by the embodiments of the present disclosure comprises: determining a target shooting mode based on the type of the to-be-recognized instrument, shooting the to-be-recognized instrument based on the target shooting mode, obtaining a to-be-recognized instrument image, performing multi-scale feature extraction on the to-be-recognized instrument image, obtaining a plurality of feature maps, performing recognition on each feature map, obtaining anchor box information corresponding to each character, calculating a plurality of character features between any two characters based on the anchor box information corresponding to each character, processing the plurality of character features between any two characters and a preset machine learning model, obtaining a candidate string set, and matching the candidate string set and a preset instrument string set to determine instrument information corresponding to the to-be-recognized instrument. By using the technical scheme, the shape and direction of the text presented on the instrument are not limited, and the text on the instrument can be recognized, thereby improving the text matching efficiency and accuracy.

[0052] Figure 2 The flowchart of another text recognition method provided by the embodiments of the present disclosure is based on the above-mentioned embodiments, and the above-mentioned text recognition method is further optimized.

[0053] As Figure 2 shown, the method comprises:

[0054] In step 201, based on the type of the to-be-recognized instrument, the first instrument size, the first instrument shape and the first text position information of the to-be-recognized instrument are determined, and when the first instrument size is less than a preset first target size, the first instrument shape meets a preset first shape type, and the first text position information meets a preset first text setting, a shooting device is controlled to shoot the to-be-recognized instrument according to a target shooting mode of a set fixed time and a fixed moving direction, and a to-be-recognized instrument image is obtained.

[0055] Specifically, image acquisition is mainly used for shooting the to-be-recognized instrument, and for the to-be-recognized instrument with a slightly smaller size, a more fixed shape, and text information only presented on a single side, a shooting template information can be input in advance to control the shooting device to realize shooting with a fixed time and a fixed moving direction, and the image of a single to-be-recognized instrument is collected each time. The first target size, the first shape type and the first text setting can all be selected and set according to the application scenario.

[0056] In step 202, based on the type of the to-be-recognized instrument, the second instrument size, the second instrument shape and the second text position information of the to-be-recognized instrument are determined, and when the second instrument size is greater than a preset second target size, the second instrument shape meets a preset second shape type, and the second text position information meets a preset second text setting, the to-be-recognized instrument is placed according to a target mode, and a shooting device is controlled to shoot the to-be-recognized instrument according to a target shooting mode of a set multiple collection, and a to-be-recognized instrument image is obtained.

[0057] Specifically, for the to-be-recognized instrument with slightly larger size, larger shape change, and text information on multiple surfaces, multiple image collections can be performed under the shooting device in a handheld or flat placement manner. The second target size, the second shape type, and the second text setting can be selected and set according to application scenarios.

[0058] It should be noted that the text printed on the to-be-recognized instrument with slightly smaller size and slightly larger size cannot be directly observed by the human eye. In order to cover as many types of to-be-recognized instruments as possible, more shooting models and corresponding shooting conditions can also be set to further meet the use requirements.

[0059] The execution order of steps 201 and 202 is selected according to application scenarios, for example, different to-be-recognized instruments with different sizes, shapes, and text position information are processed in different ways, and after step 201 or step 202, step 203 can be executed. The specific execution order Figure 2 is only an example.

[0060] Step 203: Multi-scale feature extraction is performed on the to-be-recognized instrument image to obtain multiple feature maps, anchor box coordinates corresponding to the coordinates of the to-be-recognized instrument image are generated based on the coordinates of each pixel position on each feature map, and character class probabilities corresponding to each anchor box coordinate are obtained. Based on the anchor box coordinates and the character class probabilities corresponding to each anchor box coordinate, anchor box information corresponding to each character is obtained.

[0061] Specifically, a deep neural network model is used to recognize multiple characters in the to-be-recognized instrument image. The deep neural network model proposed in the present disclosure is divided into two parts: a feature extraction network and a head network.

[0062] Specifically, the feature extraction network can be composed of multiple convolution layers, batch normalization layers, and activation layers, and its purpose is to extract features in the image. For example, a to-be-recognized instrument image with an input resolution of [H, W, C] is input, where H, W, and C represent the width, height, and channel number of the image, respectively. After passing through the feature extraction network, three feature maps with different scales are extracted, which are represented by a first feature map, a second feature map, and a third feature map, respectively, with shapes of [H / 8, W / 8, C1], [H / 16, W / 16, C2], and [H / 32, W / 32, C3].

[0063] Specifically, the head network is also composed of multiple convolutional layers, batch normalization layers and activation layers, and the purpose is to recognize the characters that may exist in the image. For the above three different scale feature maps, the head network generates anchor box coordinates corresponding to the original image coordinates based on the pixel position coordinates on each feature map, and the anchor box coordinates are represented by [x, y, w, h, a], which respectively represent the horizontal coordinate of the anchor box center point, the vertical coordinate of the anchor box center point, the length, the width and the rotation angle.

[0064] Further, the number of anchor boxes generated for the first, second and third feature maps is N1, N2 and N3 respectively, a total of N anchor boxes, N=N1+N2+N3. Input the above three different scale feature maps, the head network will output a fourth feature map with a shape of [N, M] and a fifth feature map with a shape of [N, 5]. The first dimension N in the fourth feature map corresponds to the number of generated anchor boxes, and the second dimension M represents the class probability of the character to which the anchor box belongs. Among the M categories, the category with the highest probability is taken as the category to which the anchor box belongs, denoted as c. The first dimension N in the fifth feature map corresponds to the number of generated anchor boxes, and the second dimension 5 corresponds to the anchor box coordinates [x, y, w, h, a]. For the N anchor boxes, non-maximum suppression and setting a fixed probability value will be used to filter low confidence and overlapping anchor boxes.

[0065] Therefore, the characters that may exist in the image to be recognized are recognized to form an independent character set U, and each character is represented by [x, y, w, h, a].

[0066] Step 204, based on the length coordinate and the width coordinate in the anchor box information corresponding to each character, the character size feature between any two characters is obtained, and based on the rotation angle in the anchor box information corresponding to each character, the character direction feature between any two characters is obtained.

[0067] Step 205, based on the anchor box center point horizontal coordinate and the anchor box center point vertical coordinate in the anchor box information corresponding to each character, the character center distance feature between any two characters is obtained, and the character size feature, the character direction feature and the character center distance feature between any two characters are taken as multiple character features between any two characters.

[0068] Step 206, the first character without successor is taken out from all characters, and the second character without predecessor is obtained from the remaining all characters, and the multiple character features between the first character and the second character are input into the machine learning model to obtain the association result of the first character and the second character, and it is judged whether the second character contains the predecessor first character.

[0069] Step 207, if the association result is association and the second character contains a preceding first character, arrange in the order of the first character and the second character, and process the second character as the first character until all characters without a preceding are traversed, to obtain a candidate character sequence, in the case of a ring formed by the candidate character sequence, disconnect the two adjacent characters with the largest character distance to obtain a candidate character string set.

[0070] Specifically, the multiple independent characters are associated and combined, which is essentially a step of connecting multiple characters into a character string, a machine learning model containing multiple character features is established to determine whether any two characters are associated, and for multiple character features of any two characters i and j, the following features are considered: character size feature T1, character direction feature T2, and character center distance feature T3, which are specifically calculated as formulas (1)-(3).

[0071]

[0072] T2 = |a i -a j | (2)

[0073]

[0074] Further, the established machine learning model determines whether any two characters are associated, and if a plurality of characters found through the predecessors or successors of the characters form a character sequence, i.e., a character string. The specific determination steps are as follows:

[0075] Step 1, take the first character i without a successor from the independent character set U. Step 2, take the second character j without a predecessor from the remaining independent character set U, where i≠j. Step 3, input the multiple character features of the two characters into the machine learning model to predict whether the character i is associated with the character j, and determine whether the character j contains a predecessor. Step 4, if no, repeat steps 2 and 3 until all characters without a successor are traversed. Step 5, if yes, consider the character i as the predecessor of the character j, i.e., arrange in the order of the character i and the character j. And take j as the first character, repeat steps 2 and 3 until all characters without a predecessor are traversed. Step 6, determine whether the character sequence formed constitutes a ring, if yes, disconnect the two adjacent characters with the largest character distance to form a loop-free sequence. Thus, multiple character strings existing in the instrument image to be recognized are obtained to form a candidate character string set S1.

[0076] Step 208, a first substring is obtained from the candidate string set, and a second substring is obtained from the instrument string set. If the first substring and the second substring are equal, the first substring is marked as a match success, and the substring in the candidate string set is continuously obtained for matching until all substrings in the candidate string set are matched.

[0077] Step 209, the number of substrings in the candidate string set that match successfully is equal to the number of substrings in the candidate string set, and it is determined that the text information corresponding to the instrument string set is the instrument information corresponding to the to-be-identified instrument.

[0078] Specifically, the string existing in the to-be-identified instrument image is matched with the string in the database. It can be understood that the database contains the related information of multiple instruments, and each instrument information contains multiple strings with different meanings. All the strings contained in the medical instrument constitute an instrument string set S2. In the text matching process, a many-to-many matching method is adopted, and the specific steps are as follows:

[0079] Step 1, a substring s1 is taken out from the string set S1. Step 2, a string s2 is taken out from the string set S2. Step 3, whether s1 and s2 are equal is compared. If not equal, return to step 2. If equal, the string is marked as a match success, and return to step 1 for continuous execution. Whether the number of matching successful strings in S1 and the number of S1 strings are equal is judged. If equal, it is considered that the instrument information matches successfully. If not, it is considered that the medical instrument information matches unsuccessfully.

[0080] As an example of a scenario, by pre-entering the shooting template information, shooting in a fixed time and fixed moving direction is realized. In an embodiment, a 10-row and 20-column bone screw box is shot, the shooting time is fixed at 0.5 seconds, and the moving distance of the shooting device is 1 centimeter. The user can also place the to-be-identified instrument on the platform or hold the instrument for shooting. In an embodiment, the shooting times are 2 times. It should be noted that the 2 or more images after shooting are set as the information of the same to-be-identified instrument.

[0081] Further, an input image to be recognized of resolution [H, W, C] is input, where H, W, and C represent the width, height, and channel number of the image, respectively. After passing through the feature extraction network, three feature maps of different scales are extracted, which are denoted as a first feature map, a second feature map, and a third feature map, and have shapes of [H / 8, W / 8, C1], [H / 16, W / 16, C2], and [H / 32, W / 32, C3], respectively. In an embodiment, the resolution of the image to be recognized is [1024, 2048, 3], and the resolutions of the first feature map, the second feature map, and the third feature map are [128, 128, 256], [64, 64, 256], and [32, 32, 256], respectively.

[0082] Further, the number of anchor boxes generated for the first, second, and third feature maps is N1, N2, and N3, respectively, and the total number of anchor boxes is N=N1+N2+N3. In an embodiment, N1 is 49152, N2 is 12288, N3 is 3072, and N is 64512. The above-mentioned three feature maps of different scales are input, and the head network outputs a fourth feature map of shape [N, M] and a fifth feature map of shape [N, 5]. In an embodiment, the class M is 37, including 10 digits (0-9), 26 English characters (a-z), and one background class. The first dimension N in the fifth feature map corresponds to the number of generated anchor boxes, and the second dimension 5 corresponds to the anchor box coordinates [x, y, w, h, a]. For N anchor boxes, non-maximum suppression and a fixed probability value are used to filter low-confidence and overlapping anchor boxes. In an embodiment, the confidence is set to 0.8, and the intersection over union of non-maximum suppression is set to 0.5.

[0083] Finally, multiple independent characters are associated and combined, which is essentially a step of connecting multiple characters into a string. A machine learning model including multiple character features is established to determine whether any two characters are associated. For multiple character features of any two characters i and j, the following features are considered: character size feature T1, character direction feature T2, and character center distance feature T3. Further, the established machine learning model determines whether any two characters are associated. If a plurality of characters found by the predecessor or successor of a character form a character sequence, i.e., a string, and the string present in the image to be recognized is successfully paired with a string in the database, it can be determined that the text information corresponding to the string set of the instrument is the instrument information corresponding to the image to be recognized.

[0084] The text recognition scheme provided by the embodiments of the present disclosure comprises the following steps: determining a first instrument size, a first instrument shape and first text position information of an instrument to be recognized based on a type of the instrument to be recognized; when the first instrument size is smaller than a preset first target size, the first instrument shape satisfies a preset first shape type and the first text position information satisfies a preset first text setting, controlling a shooting device to shoot the instrument to be recognized in a target shooting mode with a set fixed time and a set fixed moving direction, and obtaining an instrument to be recognized image; determining a second instrument size, a second instrument shape and second text position information of the instrument to be recognized based on the type of the instrument to be recognized; when the second instrument size is greater than a preset second target size, the second instrument shape satisfies a preset second shape type and the second text position information satisfies a preset second text setting, placing the instrument to be recognized in a target manner, and controlling the shooting device to shoot the instrument to be recognized in a target shooting mode with a set multiple times of acquisition, and obtaining the instrument to be recognized image; performing multi-scale feature extraction on the instrument to be recognized image, and obtaining a plurality of feature maps; based on each pixel position coordinate on each feature map, generating anchor box coordinates corresponding to coordinates of the instrument to be recognized image, and obtaining character class probability corresponding to each anchor box coordinate; based on the anchor box coordinates and the character class probability corresponding to each anchor box coordinate, obtaining anchor box information corresponding to each character; based on length coordinates and width coordinates in the anchor box information corresponding to each character, calculating character size features between any two characters; based on a rotation angle in the anchor box information corresponding to each character, calculating character direction features between any two characters; based on an anchor box center point horizontal coordinate and an anchor box center point vertical coordinate in the anchor box information corresponding to each character, calculating character center distance features between any two characters; taking the character size features, the character direction features and the character center distance features between any two characters as a plurality of character features between any two characters; taking a first character without a successor from all characters, and obtaining a second character without a predecessor from the remaining all characters; inputting the plurality of character features between the first character and the second character into a machine learning model, obtaining an association result of the first character and the second character, and determining whether the second character contains the predecessor first character; if the association result is association and the second character contains the predecessor first character, arranging the first character and the second character in order, and taking the second character as the first character for processing; until all characters without predecessors are traversed, obtaining a candidate character sequence; in the case that the candidate character sequence constitutes a ring, disconnecting two adjacent characters with a maximum character distance, obtaining a candidate character string set; obtaining a first substring from the candidate character string set, and obtaining a second substring from an instrument character string set; if the first substring and the second substring are equal, marking the first substring as matching successful, and continuing to obtain substrings from the candidate character string set for matching until all substrings in the candidate character string set are matched,The number of successfully matched sub-strings in the candidate string set is equal to the number of sub-strings in the candidate string set, and the text information corresponding to the instrument string set is determined as the instrument information corresponding to the to-be-recognized instrument. By using the technical solution, the shape and direction of the text presented on the instrument are not limited, and the text on the instrument can be recognized, thereby improving the text matching efficiency and accuracy.

[0085] Figure 3 A structural diagram of a text recognition device provided by the embodiment of the present disclosure is shown in FIG. 1. The device can be implemented by software and / or hardware, and can be integrated in an electronic device. As shown in FIG. 1, the device includes: Figure 3 A shooting module 301 is configured to determine a target shooting mode based on a to-be-recognized instrument type, shoot the to-be-recognized instrument, and obtain a to-be-recognized instrument image.

[0086] An extraction module 302 is configured to perform multi-scale feature extraction on the to-be-recognized instrument image, and obtain a plurality of feature maps.

[0087] An identification module 303 is configured to identify each feature map, and obtain anchor box information corresponding to each character.

[0088] A calculation module 304 is configured to calculate a plurality of character features between any two characters based on the anchor box information corresponding to each character.

[0089] A processing module 305 is configured to process the plurality of character features between any two characters and a preset machine learning model, and obtain a candidate string set.

[0090] A matching module 306 is configured to match the candidate string set and a preset instrument string set, and determine instrument information corresponding to the to-be-recognized instrument.

[0091] Optionally, the shooting module 301 is specifically configured to:

[0092] determine a first instrument size, a first instrument shape, and first text position information of the to-be-recognized instrument based on the to-be-recognized instrument type;

[0093] when the first instrument size is smaller than a preset first target size, the first instrument shape satisfies a preset first shape type, and the first text position information satisfies a preset first text setting, control a shooting device to shoot the to-be-recognized instrument according to a target shooting mode with a set fixed time and a set fixed moving direction, and obtain the to-be-recognized instrument image.

[0094] Optionally, the shooting module 301 is specifically configured to:

[0095] determine a second instrument size, a second instrument shape, and second text position information of the to-be-recognized instrument based on the to-be-recognized instrument type;

[0096] determine a second instrument size, a second instrument shape and second text position information of the instrument to be identified based on the instrument type to be identified;

[0097] place the instrument to be identified in a target manner and control a photographing device to photograph the instrument to be identified in a target photographing mode set for multiple times of acquisition, to obtain the instrument to be identified image, when the second instrument size is greater than a preset second target size, the second instrument shape satisfies a preset second shape type, and the second text position information satisfies a preset second text setting.

[0098] Optionally, the identification module 303 is specifically configured to:

[0099] generate anchor box coordinates corresponding to the instrument to be identified image coordinates based on each pixel position coordinate on each feature map, and obtain a character class probability corresponding to each anchor box coordinate;

[0100] obtain anchor box information corresponding to each character based on the anchor box coordinates and the character class probability corresponding to each anchor box coordinate.

[0101] Optionally, the calculation module 304 is specifically configured to:

[0102] calculate a length coordinate and a width coordinate in the anchor box information corresponding to each character to obtain a character size feature between any two characters;

[0103] calculate a rotation angle in the anchor box information corresponding to each character to obtain a character direction feature between any two characters;

[0104] calculate an anchor box center point horizontal coordinate and an anchor box center point vertical coordinate in the anchor box information corresponding to each character to obtain a character center distance feature between any two characters;

[0105] use the character size feature, the character direction feature and the character center distance feature between the any two characters as a plurality of character features between the any two characters.

[0106] Optionally, the processing module 305 is specifically configured to:

[0107] extract a first character without a successor from all characters, and extract a second character without a predecessor from all remaining characters;

[0108] input a plurality of character features between the first character and the second character into the machine learning model to obtain an association result of the first character and the second character, and determine whether the second character contains the first character as a predecessor;

[0109] If the association result is association and the second character contains a predecessor of the first character, arrange in the order of the first character and the second character, and process the second character as the first character until all characters without predecessors are traversed to obtain a candidate character sequence;

[0110] In the case that the candidate character sequence forms a ring, disconnect two adjacent characters of the maximum character distance to obtain the candidate string set.

[0111] Optionally, the matching module 305 is specifically configured to:

[0112] obtain a first substring from the candidate string set and a second substring from the instrument string set;

[0113] If the first substring and the second substring are equal, mark the first substring as matching success, and continue to obtain substrings from the candidate string set for matching until all substrings in the candidate string set are matched.

[0114] If the number of substrings in the candidate string set that match successfully is equal to the number of substrings in the candidate string set, it is determined that the text information corresponding to the instrument string set is the instrument information corresponding to the to-be-recognized instrument.

[0115] The text recognition apparatus provided in the embodiments of the present disclosure can perform the text recognition method provided in any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0116] The embodiments of the present disclosure further provide a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the text recognition method provided in any of the embodiments of the present disclosure.

[0117] Figure 4 A structural schematic diagram of an electronic device provided in the embodiments of the present disclosure is provided. The following specifically refers to Figure 4 which shows a structural schematic diagram of an electronic device 400 suitable for being used to implement the embodiments of the present disclosure. The electronic device 400 in the embodiments of the present disclosure can include but is not limited to a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (for example, a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 4 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0118] As Figure 4As shown, the electronic device 400 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0119] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device 400 to communicate wirelessly or wired with other devices to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that all of the illustrated devices are not required, but instead can be replaced by more or fewer devices.

[0120] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 409, or installed from the storage devices 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-described functions defined in the text recognition method of embodiments of the present disclosure are performed.

[0121] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer the program for use by or in connection with the instruction execution system, apparatus or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0122] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0123] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and is not assembled into the electronic device.

[0124] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: in a playing process of a video, receive an information display triggering operation of a user; obtain at least two target information associated with the video; display first target information of the at least two target information in an information display area of a playing page of the video, wherein a size of the information display area is smaller than a size of the playing page; and receive a first switching triggering operation of the user, and switch the first target information displayed in the information display area to second target information of the at least two target information.

[0125] Computer program code for carrying out operations of the present disclosure can be written in any one or more of a variety of programming languages or combinations of languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0126] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0127] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0128] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0129] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, comprising:

[0131] a processor;

[0132] a memory for storing the processor-executable instructions;

[0133] the processor, configured to read the executable instructions from the memory and execute the instructions to implement any of the text recognition methods provided by the present disclosure.

[0134] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, which stores a computer program for executing any of the text recognition methods provided by the present disclosure.

[0135] The above description is merely exemplary of the present disclosure and the application of the principles thereof and the scope of protection to be afforded the disclosure is not limited to the foregoing merely by the description that is set forth above but only by the claims that follow and that are delineated to most clearly determine the bounds of the inventive subject matter specified by the disclosure.

[0136] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the disclosure. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while several specific implementation details have been discussed, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0137] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A text recognition method, characterized by, The method comprises the following steps: determining a target shooting mode based on the type of the instrument to be identified, shooting the instrument to be identified, and obtaining an instrument image to be identified; performing multi-scale feature extraction on the instrument image to be identified to obtain a plurality of feature maps, and identifying each feature map to obtain anchor box information corresponding to each character; calculating a plurality of character features between any two characters based on the anchor box information corresponding to each character, and processing the plurality of character features between any two characters and a preset machine learning model to obtain a candidate string set; wherein the identification of each feature map to obtain anchor box information corresponding to each character comprises: generating anchor box coordinates corresponding to the coordinates of the instrument image to be identified based on the coordinates of each pixel position on each feature map, and obtaining the character class probability corresponding to each anchor box coordinate; and obtaining the anchor box information corresponding to each character based on the anchor box coordinates and the character class probability corresponding to each anchor box coordinate; wherein the calculation of a plurality of character features between any two characters based on the anchor box information corresponding to each character comprises: calculating character size features between any two characters based on the length coordinates and width coordinates in the anchor box information corresponding to each character; calculating character direction features between any two characters based on the rotation angle in the anchor box information corresponding to each character; calculating character center distance features between any two characters based on the horizontal anchor box center point coordinates and the vertical anchor box center point coordinates in the anchor box information corresponding to each character; and taking the character size features, character direction features, and character center distance features between any two characters as the plurality of character features between any two characters; matching the candidate string set with a preset instrument string set to determine the instrument information corresponding to the instrument to be identified.

2. The text recognition method of claim 1, wherein, The method comprises the following steps: determining a target shooting mode based on the type of the instrument to be identified, shooting the instrument to be identified, and obtaining an instrument image to be identified; determining a first instrument size, a first instrument shape, and first text position information of the instrument to be identified based on the type of the instrument to be identified; 3. The text recognition method of claim 1, wherein, when the first instrument size is smaller than a preset first target size, the first instrument shape meets a preset first shape type, and the first text position information meets a preset first text setting, controlling a shooting device to shoot the instrument to be identified in a target shooting mode with a set fixed time and a set fixed moving direction, and obtaining the instrument image to be identified. The method comprises the following steps: determining a target shooting mode based on the type of the instrument to be identified, shooting the instrument to be identified, and obtaining an instrument image to be identified; determining a second instrument size, a second instrument shape, and second text position information of the instrument to be identified based on the type of the instrument to be identified; In a case where the second instrument size is greater than a preset second target size, the second instrument shape satisfies a preset second shape type, and the second text position information satisfies a preset second text setting, the to-be-identified instrument is placed in a target manner, and a photographing device is controlled to photograph the to-be-identified instrument in a target photographing mode of multiple times of acquisition, to obtain the to-be-identified instrument image.

4. The text recognition method of claim 1, wherein, The processing based on the plurality of character features between the arbitrary two characters and the preset machine learning model obtains a candidate character string set, including: A first character without a successor is taken out from all characters, and a second character without a predecessor is taken out from the remaining all characters; A plurality of character features between the first character and the second character are input into the machine learning model, to obtain an association result of the first character and the second character, and it is judged whether the second character contains the first character as a predecessor; If the association result is association and the second character contains the first character as a predecessor, the first character and the second character are arranged in order, and the second character is processed as a first character until all characters without predecessors are traversed, to obtain a candidate character sequence; In a case where the candidate character sequence constitutes a ring, two adjacent characters of a maximum character distance are disconnected, to obtain the candidate character string set.

5. The text recognition method of claim 1, wherein, The matching based on the candidate character string set and a preset instrument character string set determines instrument information corresponding to the to-be-identified instrument, including: A first substring is taken out from the candidate character string set, and a second substring is taken out from the instrument character string set; If the first substring and the second substring are equal, the first substring is marked as matching successful, and a substring is continuously taken out from the candidate character string set for matching until all substrings in the candidate character string set are matched; In a case where a number of substrings in the candidate character string set that are matched is equal to a number of substrings in the candidate character string set, it is determined that text information corresponding to the instrument character string set is instrument information corresponding to the to-be-identified instrument.

6. A text recognition apparatus characterized by comprising: The method comprises: The photographing module is configured to determine a target photographing mode based on a to-be-identified instrument type, and photograph the to-be-identified instrument to obtain a to-be-identified instrument image; The extraction module is configured to perform multi-scale feature extraction on the to-be-identified instrument image to obtain a plurality of feature maps; The identification module is configured to identify each feature map to obtain anchor box information corresponding to each character; wherein the identification module is specifically configured to generate anchor box coordinates corresponding to to-be-identified instrument image coordinates based on each pixel position coordinate on each feature map, and obtain a character category probability corresponding to each anchor box coordinate; and obtain anchor box information corresponding to each character based on the anchor box coordinates and the character category probability corresponding to each anchor box coordinate. The computing module is configured to: calculate character size features between any two characters based on the length coordinates and the width coordinates in the anchor box information corresponding to each character; calculate character direction features between any two characters based on the rotation angle in the anchor box information corresponding to each character; calculate character center distance features between any two characters based on the anchor box center point horizontal coordinates and the anchor box center point vertical coordinates in the anchor box information corresponding to each character; and take the character size features, the character direction features and the character center distance features between any two characters as the character features between the two characters. The processing module is configured to process the character features between any two characters and a preset machine learning model to obtain a candidate string set. The matching module is configured to match the candidate string set and a preset instrument string set to determine the instrument information corresponding to the instrument to be recognized.

7. An electronic device, comprising: The electronic device comprises: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the text recognition method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the text recognition method in any one of claims 1-5. The storage medium stores a computer program for executing the text recognition method in any one of claims 1-5.

Images