Text recognition method based on wearable device, wearable device and storage medium

Abstract

The embodiment of the invention provides a text recognition method and device based on wearable equipment, the wearable equipment and a storage medium. The method comprises the steps of collecting a to-be-recognized image stream through the wearable device; determining fingertip key points from the to-be-recognized image stream, and determining a recognition starting position and a starting image frame based on the fingertip key points; character-level text recognition is carried out from the initial image frame and the recognition initial position until text recognition is finished at the recognition end position and the end image frame, and a text recognition result is obtained. According to the embodiment of the invention, the to-be-recognized image stream and the fingertip key points are combined, and while the character-level text recognition based on the wearable device is realized, the real-time performance of text recognition is also realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text recognition method based on a wearable device, a wearable device, and a computer-readable storage medium. Background Art

[0002] Optical Character Recognition (OCR) refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then uses character recognition methods to translate the shapes into computer text. With the development of technology, optical character recognition functions can be implemented not only on mobile phones and tablets, but also on wearable devices such as smart glasses and smart helmets.

[0003] In the related art, the optical character recognition function of wearable devices can realize text recognition processing such as text translation. However, due to the limited screen size and limited interaction methods of wearable devices, the text recognition processing based on wearable devices cannot freely select any word, phrase, sentence and paragraph in the entire text on the screen or using tools such as keyboard and mouse like mobile phones and tablets, that is, it is impossible to realize free selection of text. In addition, because the text recognition processing method of wearable devices in the related art can only recognize words or entire paragraphs, the wearer can only recognize the text after the text is determined, and cannot obtain the recognition result while selecting the text, that is, it is impossible to obtain a real-time updated text recognition result. Summary of the invention

[0004] The present application provides a text recognition method based on a wearable device, a wearable device and a computer-readable storage medium, which aims to combine the image stream to be recognized and the fingertip key points, and realize character-level text recognition based on the wearable device while also realizing the real-time performance of text recognition.

[0005] To achieve the above objectives, the present application provides a text recognition method based on a wearable device, the method comprising:

[0006] Collecting an image stream to be identified through the wearable device;

[0007] Determine fingertip key points from the image stream to be recognized, and determine a recognition start position and a start image frame based on the fingertip key points;

[0008] Character-level text recognition is performed starting from the starting image frame and the recognition starting position until text recognition is terminated at the recognition ending position and the ending image frame to obtain a text recognition result.

[0009] In addition, to achieve the above purpose, the present application also provides a text recognition device, including:

[0010] A collection module, the collection module is used to collect the image stream to be identified through the wearable device;

[0011] A determination module, the determination module is used to determine the fingertip key points from the image stream to be identified, and determine the identification starting position and the starting image frame based on the fingertip key points;

[0012] The text recognition module is used to perform character-level text recognition starting from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame to obtain a text recognition result.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a wearable device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of any one of the text recognition methods based on a wearable device provided in the embodiments of the present application when executing the computer program.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of any one of the text recognition methods based on a wearable device provided in the embodiments of the present application.

[0015] The text recognition method based on wearable device, text recognition device, wearable device and computer-readable storage medium disclosed in the embodiment of the present application can collect the image stream to be recognized through the wearable device, determine the fingertip key points from the image stream to be recognized, and determine the recognition start position and the starting image frame based on the fingertip key points. Thus, character-level text recognition can be performed from the starting image frame and the recognition start position until the text recognition ends at the recognition end position and the end image frame to obtain the text recognition result. The present application collects the image stream to be recognized through the image acquisition unit of the wearable device, and determines the recognition start position using the fingertip key points, and then directly starts to recognize the text on the starting image frame, so that the wearer can perform text recognition in real time according to the position of the fingertip key points, thereby meeting the real-time nature of text recognition. Further, through the positioning of the fingertip key points, the recognition start position can be freely selected, and by combining it with character-level text recognition, it is possible to realize the free selection of characters, sentences and paragraphs in the text, providing a more flexible text selection capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a flowchart of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of a process for determining a recognition starting position and a starting image frame provided by an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a first scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a scenario for determining fingertip coordinates provided in an embodiment of the present application;

[0021] Figure 5 This is a flowchart of text recognition provided by an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of a second scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0023] Figure 7 is a schematic diagram of a third scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0024] Figure 8 This is a flowchart of obtaining text recognition results provided by an embodiment of the present application;

[0025] Fig. 9 is a schematic diagram of a fourth scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0026] Fig.10 is a schematic block diagram of a text recognition device provided in an embodiment of the present application;

[0027] Fig.11 It is a schematic block diagram of a wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0030] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0032] See also Figure 1 , Figure 1 FIG. 1 is a flowchart of a text recognition method based on a wearable device provided in an embodiment of the present application. Figure 1 As shown, the text recognition method based on the wearable device includes steps S11 to S13.

[0033] Step S11: collecting the image stream to be identified through the wearable device.

[0034] Among them, the wearable device can be smart glasses, such as AR glasses, VR glasses, etc., which is not limited in this application.

[0035] Furthermore, an image acquisition unit may be installed on the wearable device to acquire the image stream to be identified. The image acquisition unit includes components such as a camera, a sensor, and a lens installed on the wearable device. These components work together to acquire images and information of the surrounding environment.

[0036] The image stream to be identified is a continuous image frame collected by the image acquisition unit. In the image stream to be identified, each image frame is continuous in time and presented in time sequence, thereby forming a continuous image stream.

[0037] It can be understood that the image stream to be identified includes at least text and the hand information of the wearer. Among them, the hand information may include the fingertips of the fingers, and the fingertips of any finger (such as the fingertips of the index finger) may correspond to characters in the text for selecting the characters to be identified. Specifically, the movement of the wearer's fingertips on the text can be captured by the image acquisition unit, thereby determining the starting point and end point of the fingertips on the text, and then determining the character corresponding to the starting point of the fingertip, the character corresponding to the end point of the fingertip, and all characters between the two as the characters to be identified.

[0038] In addition, when the wearer's fingertip does not move on the text for more than a preset time, the image acquisition unit can also directly acquire the image stream of the wearer's fingertip on the text during this period of time to determine the characters to be recognized. Thus, the image stream to be recognized can be used to determine the text to be recognized.

[0039] In an embodiment of the present application, the image stream to be recognized can be acquired by the image acquisition unit of the wearable device to determine the text to be recognized.

[0040] Step S12: determining fingertip key points from the image stream to be recognized, and determining the recognition start position and the start image frame based on the fingertip key points.

[0041] Among them, the fingertip key point is a significant point at a specific position of the fingertip, usually the position of the end of the finger.

[0042] Specifically, the target detection algorithm can be used to detect the image stream to be identified, and the hand area in the image stream to be identified can be identified. Furthermore, the key point detection algorithm can be used to perform key point detection on the hand area to determine the fingertip key points. Among them, the target detection algorithm includes Haar cascade classifier, YOLO (You Only Look Once) model, etc.; the key point detection algorithm includes convolutional neural network, stacked hourglass model, etc., which is not limited in this application.

[0043] It is understandable that the recognition start position and the start image frame are determined based on the fingertip key points, and the recognition start position can be used as the basis for word retrieval, that is, it is determined that the current wearer is in the word retrieval state. In this way, starting from the starting image frame, the movement of the fingertip key points of the finger in the subsequent image frames can be tracked, and the starting point and end point of the fingertip can be determined on the text, so that the text to be recognized can be selected and recognized according to the starting point and end point of the fingertip. It is understandable that the starting position is a position in the starting image frame.

[0044] It should be noted that the present application does not limit the method for determining the identification starting position and the starting image frame based on the fingertip key points. For example, the identification starting position and the starting image frame can be determined based on the correspondence between the fingertip key points and the finger joint points and whether the dwell time of the fingertip key points is greater than a preset duration. This application will be described in detail later.

[0045] In an embodiment of the present application, fingertip key points can be determined from the image stream to be identified, and the identification starting position and starting image frame can be determined based on the fingertip key points, and then based on the identification starting position, it can be determined that the current wearer is in a word-capturing state, so as to track the movement of the fingertip key points of the finger in subsequent image frames starting from the starting image frame, and simultaneously perform text selection and identification.

[0046] Step S13: character-level text recognition is performed starting from the starting image frame and the recognition starting position until the text recognition is terminated at the recognition ending position and the ending image frame to obtain a text recognition result.

[0047] Specifically, character-level text recognition can be performed through an optical character recognition model starting from the starting image frame and the recognition starting position. Furthermore, starting from the starting image frame, the movement of the fingertip key points in the subsequent image frames is tracked, and the starting point and end point of the fingertip can be determined on the text, wherein the starting point of the fingertip corresponds to the starting image frame, and the end point of the fingertip corresponds to the ending image frame. In this way, the text to be recognized can be selected and recognized according to the starting point and end point of the fingertip, until the text recognition is completed at the recognition end position and the ending image frame, and the text recognition result is obtained.

[0048] It should be noted that the present application does not limit the method for determining the end position and the end image frame of the recognition, for example, it can be determined based on the dwell time of the fingertip key point, that is, when the dwell time of the fingertip key point exceeds the preset time, the position corresponding to the current fingertip key point can be determined as the end position of the recognition, and then the image frame corresponding to the end position of the recognition can be determined as the end image frame. Among them, the end image frame is the start image frame or an image frame after the start image frame.

[0049] In addition, it can also be determined based on the wearer's instructions. For example, when the wearer finishes word capture, an end instruction can be sent to the wearable device, and then the wearable device can use the position corresponding to the current fingertip key point as the recognition end position based on the end instruction, and then determine the image frame corresponding to the recognition end position as the end image frame, and end word capture.

[0050] In an embodiment of the present application, starting from the starting image frame and the recognition starting position, word extraction can be performed character by character in continuous image frames, and text recognition can be performed through an optical character recognition model until word extraction and text recognition are completed at the recognition ending position and the ending image frame to obtain a text recognition result. Since the present application uses the fingertip key points to determine the recognition starting position, and then starts recognizing the text directly on the starting image frame, the wearer can perform text recognition in real time according to the position of the fingertip key points, thereby meeting the real-time nature of text recognition. Furthermore, by positioning the fingertip key points, the recognition starting position can be freely selected, and by combining it with character-level text recognition, it is possible to freely select characters, sentences, and paragraphs in the text, providing a more flexible text selection capability.

[0051] Optionally, text recognition is ended until the identification end position and the end image frame are identified, including: when the fingertip key point meets the second preset requirement, the image frame corresponding to the fingertip key point is used as the end image frame, and the position corresponding to the fingertip key point is used as the identification end position, wherein the end image frame is the starting image frame or an image frame after the starting image frame.

[0052] Among them, the second preset requirement may be that the dwell time of the fingertip key point exceeds the preset time, for example, the preset time is 2s, 3s, etc. This application takes the preset time of 2s as an example for explanation.

[0053] Specifically, after determining the recognition start position, if the dwell time of the fingertip key point exceeds 2 seconds, the image frame corresponding to the fingertip key point can be used as the end image frame, and the position corresponding to the fingertip key point can be used as the recognition end position.

[0054] It is understandable that the recognition end position may be the same as the recognition start position, or the recognition end position may be after the recognition start position. Similarly, the end image frame may be the start image frame or an image frame after the start image frame.

[0055] In an embodiment of the present application, when it is determined that the fingertip key point meets the second preset requirement, the recognition end position and the end image frame can be determined based on the fingertip key point, and the text recognition can be ended at the recognition end position and the end image frame.

[0056] Optionally, character-level text recognition is performed starting from a starting image frame and a recognition starting position until text recognition is terminated at a recognition ending position and an ending image frame, including: taking the Laplace variance of the starting image frame as a reference variance; obtaining the Laplace variance of each image frame after the starting image frame; and performing character-level text recognition on image frames whose Laplace variance is greater than the reference variance until text recognition is terminated at a recognition ending position and an ending image frame.

[0057] It is understandable that in the process of collecting the image stream to be recognized by the image acquisition unit, the position and angle of the camera may change, so that some of the collected image frames may be blurred or unstable, and thus text recognition cannot be achieved. Therefore, it is necessary to determine the stability of the image frame based on the time series to prevent image frames with poor stability from affecting the text recognition results.

[0058] Since the wearer's fingertips have not moved when the identification start position is determined, the stability of the starting image frame corresponding to the identification start position is relatively strong, so the Laplace variance can be calculated from the starting image frame and used as the benchmark variance. Among them, the Laplace variance is an indicator used to measure image changes, which describes the degree of difference between different pixels in the image. Therefore, the Laplace variance of each subsequent image frame can be determined according to the time series and compared with the benchmark variance to evaluate the stability of the image frame.

[0059] Optionally, in order to reduce the amount of calculation of the Laplace variance, each image frame of the image stream to be identified can also be converted into a grayscale image. Since a grayscale image has only one channel, it is simpler and more efficient to process than three channels (red, green, and blue) of a color image, and the amount of calculation is lower.

[0060] It can be understood that when the Laplace variance of an image frame is greater than the reference variance, it means that the image frame is relatively stable and text recognition can be performed. Therefore, the image frame whose Laplace variance is greater than the reference variance can be used for character-level text recognition until the text recognition ends at the recognition end position and the end image frame.

[0061] In the embodiment of the present application, the Laplace variance of the starting image frame can be used as a measure of image stability, and the stability of other image frames can be determined by comparing it with the Laplace variance of other image frames. In this way, text recognition can be performed when the stability of other image frames is relatively strong, thereby improving the accuracy and reliability of text recognition results.

[0062] The text recognition method based on a wearable device disclosed in the embodiment of the present application can collect the image stream to be recognized through the image acquisition unit of the wearable device, determine the fingertip key points from the image stream to be recognized, and determine the recognition start position and the starting image frame based on the fingertip key points. Thus, character-level text recognition can be performed from the starting image frame and the recognition start position until the text recognition ends at the recognition end position and the end image frame to obtain the text recognition result. The present application collects the image stream to be recognized through the image acquisition unit of the wearable device, and determines the recognition start position using the fingertip key points, and then directly starts to recognize the text on the starting image frame, so that the wearer can perform text recognition in real time according to the position of the fingertip key points, thereby meeting the real-time nature of text recognition. Further, through the positioning of the fingertip key points, the recognition start position can be freely selected, and by combining it with character-level text recognition, it is possible to realize the free selection of characters, sentences and paragraphs in the text, providing a more flexible text selection capability.

[0063] See also Figures 2 to 4 , Figure 2 It is a schematic diagram of a process for determining a recognition starting position and a starting image frame provided by an embodiment of the present application; Figure 3 is a schematic diagram of a first scenario of a text recognition method based on a wearable device provided in an embodiment of the present application; Figure 4 Schematic diagram of a scene for determining the fingertip coordinates provided by an embodiment of the present application. Figure 2 As shown, steps S121 to S122 can be used to determine the recognition start position and the start image frame based on the fingertip key points.

[0064] Step S121: obtaining fingertip key points in the image stream to be identified, and obtaining finger joint key points related to the fingertip key points.

[0065] Among them, the fingertip key point is a significant point at a specific position of the fingertip, usually the position of the end of the finger.

[0066] The knuckle key point related to the fingertip key point is a significant point of the knuckle of the hand where the fingertip is located, that is, the position of the knuckle.

[0067] Specifically, the target detection algorithm can be used to detect the image stream to be identified, and the hand region in the image stream to be identified can be identified. Furthermore, the key point detection algorithm can be used to detect the key points of the hand region to determine the fingertip key points and the knuckle key points.

[0068] Among them, the target detection algorithm includes Haar cascade classifier, YOLO (You Only Look Once) model, etc.; the key point detection algorithm includes convolutional neural network, stacked hourglass model, etc., which is not limited in this application.

[0069] Step S122: When the fingertip key points and the finger joint key points meet the first preset requirement, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the recognition starting position.

[0070] Optionally, when the fingertip key points and the knuckle key points meet the first preset requirement, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the recognition starting position, including: performing mutual correlation matching on the fingertip key points and the knuckle key points, and when the match is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; segmenting the fingertip area from the starting image frame according to the fingertip key points, and determining the fingertip coordinates from the fingertip area; when the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the recognition starting position.

[0071] like Figure 3 As shown, when the fingertip key points and the finger joint key points are matched with each other, multiple finger joint key points can be connected with the wrist key points to obtain a polygonal closed area. Then, it is judged whether the fingertip key points are in the closed area to determine whether the wearer is in the word-picking state. If the fingertip key points are in the closed area, it means that the match fails and the wearer is not in the word-picking state; if the fingertip key points are not in the closed area, it means that the match is successful and the wearer is in the word-picking state. At this time, when the match is successful, the image frame corresponding to the fingertip key points can be used as the starting image frame.

[0072] Further, such as Figure 4 As shown, a specific area can be selected on the starting image frame image with the fingertip key point as the center to segment the fingertip area. The specific area can be a rectangular or elliptical area of ​​fixed size, etc., to ensure that the fingertip is covered. In the fingertip area, a fixed coordinate system can be established in advance to determine the fingertip coordinates. And it is determined whether the fingertip coordinates exist in all image frames within a preset time, and whether the change of the fingertip coordinate system in the preset time period is less than a preset threshold. In this way, it can be determined whether the fingertip coordinates can be used as the starting position for identification.

[0073] It can be understood that if the image frames within the preset time all have fingertip coordinates, and whether the change amount of the fingertip coordinate system within the preset time period is less than the preset threshold, it means that the wearer's fingertip stays at a fixed position for a preset time, and at this time, the fingertip coordinates can be determined as the recognition starting position.

[0074] It should be noted that the present application does not limit the above-mentioned preset time, for example, the preset time is 2s, 3s, 5s, etc.; further, the present application does not limit the preset threshold, for example, it can be 1px, 3px and 5px, etc.

[0075] In the embodiment of the present application, the information of the fingertip key points and the finger joint key points can be fully utilized, and accurate and reliable identification of the starting position and the determination of the starting image frame can be achieved through mutual correlation matching, image segmentation and determination of the fingertip coordinates.

[0076] See also Figures 5 to 7 , Figure 5 This is a flowchart of text recognition provided by an embodiment of the present application; Figure 6 is a schematic diagram of a second scenario of a text recognition method based on a wearable device provided in an embodiment of the present application; Figure 7 is a schematic diagram of a third scenario of a text recognition method based on a wearable device provided in an embodiment of the present application. Figure 5 As shown, after character-level text recognition is performed starting from the starting image frame and the recognition starting position, text recognition can be performed through steps S131 to S134.

[0077] Step S131: determining a first image region in a starting image frame based on the identification starting position.

[0078] The size of the first image area is smaller than the size of the starting image frame.

[0079] Step S132: determining a second image region in the next image frame based on the identification start position.

[0080] The size of the second image area is smaller than the size of the next image frame.

[0081] Step S133: Determine a transformation matrix based on the first image region and the second image region.

[0082] The first image region is an image region including the fingertip key points in the starting image frame; and the second image region is an image region including the fingertip key points in the next image frame.

[0083] Specifically, the first image area and the second image area of ​​a certain size can be cropped from the starting image frame and the next image frame, respectively. For example, the cropping can be performed with the recognition starting position as the center, so as to obtain the first image area and the second image area including the fingertip key point.

[0084] It should be noted that the present application does not limit the size of the first image area and the second image area, for example, 300px*300px, 300px*450px, 450px*450px, etc.

[0085] Preferably, the first image area and the second image area have the same size, and the center points are both recognition start positions. In this way, the first image area can be directly mapped to the second image area to determine the transformation matrix, thereby improving the efficiency of determining the transformation matrix.

[0086] Optionally, determining a transformation matrix based on the first image area and the second image area includes: obtaining a first target key point of the first image area, and obtaining a second target key point of the second image area; performing feature matching on the first target key point and the second target key point to obtain a matching pair between the first image area and the second image area; and determining the transformation matrix based on the matching pair.

[0087] Among them, the first target key point is the key point corresponding to the first image area; the second target key point is the key point corresponding to the second image area.

[0088] It should be noted that key points are a concept used in the field of computer vision and image processing to represent important features in an image. Key points are also called interest points or corner points, which are usually the edges, corners, texture changes, etc. of an object. Key points can be seen as prominent points in an image, which have high contrast and structure in a local area and are therefore more sensitive to image changes. Key points help represent local features of an image and are used for tasks such as feature matching and object recognition.

[0089] Furthermore, the present application does not limit the types of the first target key point and the second target key point, for example, including corner points, edges, etc. of the image.

[0090] like Figure 6 As shown, the left image is the first image area, and the right image is the second image area. Specifically, target key points (such as corner points) can be extracted from the first image area and the second image area, respectively. Further, feature descriptors of the corner points of the first image area and feature descriptors of the corner points of the second image area can be determined, and feature matching operations can be performed on the two. ， In order to find the most similar matching pair, the transformation matrix can be calculated based on the matching pair by using the least square method, random sampling consistency algorithm, etc.

[0091] It should be noted that the feature descriptor is a vector or matrix that represents the pixel information around the key point, which is used to capture the appearance and structural characteristics of the area around the key point. The feature descriptor provides a way to compare corresponding key points in different images, thereby achieving tasks such as image matching and target tracking. Usually, the feature descriptor extracts information such as the gradient, color histogram, and local binary pattern of the pixels around the key point, and then encodes this information into a vector or matrix, which becomes a data that can compactly represent the key point features.

[0092] Furthermore, the present application does not limit the above-mentioned feature descriptors. For example, the feature descriptors are information such as gradients, color histograms, local binary patterns, etc. corresponding to corner points.

[0093] Optionally, feature matching of feature descriptors of corner points of the first image region and feature descriptors of corner points of the second image region can be achieved by nearest neighbor matching, nearest neighbor matching and thresholding, etc. Specifically, for each feature descriptor of a corner point of the first image region, the feature descriptor of a corner point of the second image region with the closest Euclidean distance can be found by the above method, thereby achieving the most similar matching pair.

[0094] Step S134: using the transformation matrix, determining the position corresponding to the recognition start position in the next image frame, and performing character-level text recognition based on the fingertip key points in the next image frame.

[0095] like Figure 7 As shown in FIG. 1 , after obtaining the transformation matrix, each coordinate point in the current image area can be applied to the next image area, thereby realizing the mapping between the image areas. Thus, the recognition start position in the current image area can be mapped to the next image frame. That is, the position corresponding to the recognition start position is determined in the next image frame, thereby character-level text recognition can be performed based on the fingertip key points in the next image frame.

[0096] In the embodiment of the present application, feature matching and transformation matrix can be used to more accurately determine the mapping relationship between the current image area and the next image area, thereby determining the position corresponding to the recognition start position in the next image frame. In this way, the wearer can perform text recognition in real time according to the position of the key points of the fingertips to meet the real-time nature of text recognition.

[0097] See also Figure 8 and Fig. 9 , Figure 8 This is a flowchart of obtaining text recognition results provided by an embodiment of the present application; Fig. 9 is a schematic diagram of a fourth scenario of a text recognition method based on a wearable device provided in an embodiment of the present application. Figure 8 As shown, the text recognition result can be obtained through steps S135 to S137.

[0098] Step S135: determining the first text line closest to the recognition start position, and determining first character information of the character closest to the recognition start position according to the positions of each character in the first text line.

[0099] Step S136: determining a second text line closest to the recognition end position, and determining second character information of the character closest to the recognition end position according to the positions of each character in the second text line.

[0100] Step S137: determining all target character information from the first character information to the second character information, and taking the target character information as the text recognition result.

[0101] The first text line is the text line that starts recognition, that is, the text line closest to the recognition start position; the second text line is the text line that ends recognition, that is, the text line closest to the recognition end position.

[0102] Specifically, the first text line closest to the recognition start position can be determined, and based on the positions of the characters in the first text line, the character closest to the recognition start position in the first text line, that is, the character to be recognized at the start, can be determined. Furthermore, the character to be recognized at the start can be recognized to obtain the corresponding first character information. Similarly, the character closest to the recognition end position in the second text line, that is, the character to be recognized at the end, can be determined, and the character to be recognized at the end can be recognized to obtain the corresponding second character information.

[0103] It can be understood that after determining the character to be recognized at the start and the character to be recognized at the end, all characters between the two can be recognized to obtain target character information, and then the target character information can be used as the text recognition result.

[0104] In the embodiment of the present application, the character to be started for recognition and the character to be ended for recognition can be located according to the recognition start position and the recognition end position, respectively, and then all characters between the two can be recognized to obtain a text recognition result.

[0105] Optionally, after obtaining the text recognition result, the method includes: projecting the text recognition result onto a display module of the wearable device; and determining the starting point and the end point corresponding to the target text from the text recognition result in response to a control instruction.

[0106] Among them, the display module refers to the component in the wearable device used to display images, text and other information.

[0107] Specifically, after obtaining the text recognition result, the text recognition result can be projected onto the display module of the wearable device. Furthermore, after receiving the control instructions such as gestures, sounds, buttons, etc. from the wearer, the starting point and end point corresponding to the target text can be determined from the text recognition result, that is, the range of text recognition can be specified to obtain the target text.

[0108] Optionally, before obtaining the text recognition result, the recognized character information can be displayed in real time through the display module of the wearable device to provide the wearer with the recognized information in real time.

[0109] In the embodiment of the present application, the text recognition result can be presented on the display module of the wearable device, and the target text can be determined from the text recognition result through the control instruction, so as to provide a more flexible and personalized text browsing experience, so that the wearer can select the target text according to his or her needs.

[0110] See also Fig.10 , Fig.10 It is a schematic block diagram of a text recognition device provided in an embodiment of the present application. The text recognition device can be configured in a server to execute the steps of the aforementioned wearable device-based text recognition method.

[0111] like Fig.10 As shown, the text recognition device 200 includes: a collection module 201 , a determination module 202 , and a text recognition module 203 .

[0112] The acquisition module 201 is used to acquire the image stream to be identified through the wearable device;

[0113] A determination module 202, configured to determine fingertip key points from the image stream to be identified, and determine an identification start position and a start image frame based on the fingertip key points;

[0114] The text recognition module 203 is used to perform character-level text recognition starting from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame to obtain a text recognition result.

[0115] The determination module 202 is also used to obtain the fingertip key points in the image stream to be identified, and to obtain the finger joint key points related to the fingertip key points; when the fingertip key points and the finger joint key points meet the first preset requirements, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the identification starting position.

[0116] The determination module 202 is also used to perform mutual correlation matching on the fingertip key points and the finger joint key points, and when the matching is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; the fingertip area is segmented from the starting image frame according to the fingertip key points, and the fingertip coordinates are determined from the fingertip area; when the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the identification starting position.

[0117] The text recognition module 203 is also used to use the Laplace variance of the starting image frame as a reference variance; obtain the Laplace variance of each image frame after the starting image frame; and perform character-level text recognition on the image frames whose Laplace variance is greater than the reference variance until text recognition is terminated at the recognition end position and the end image frame.

[0118] The text recognition module 203 is also used to use the image frame corresponding to the fingertip key point as the end image frame and the position corresponding to the fingertip key point as the recognition end position when the fingertip key point meets the second preset requirement, wherein the end image frame is the starting image frame or an image frame after the starting image frame.

[0119] The text recognition module 203 is also used to determine a first image area in the starting image frame based on the recognition starting position, wherein the size of the first image area is smaller than the size of the starting image frame; determine a second image area in the next image frame based on the recognition starting position, wherein the size of the second image area is smaller than the size of the next image frame; determine a transformation matrix based on the first image area and the second image area; use the transformation matrix to determine the position corresponding to the recognition starting position in the next image frame, and perform character-level text recognition based on the fingertip key points in the next image frame.

[0120] The text recognition module 203 is also used to obtain the first target key point of the first image area and the second target key point of the second image area; perform feature matching on the first target key point and the second target key point to obtain a matching pair between the first image area and the second image area; and determine the transformation matrix based on the matching pair.

[0121] The text recognition module 203 is also used to determine the first text line closest to the recognition starting position, and determine the first character information of the character closest to the recognition starting position based on the positions of each character in the first text line; determine the second text line closest to the recognition ending position, and determine the second character information of the character closest to the recognition ending position based on the positions of each character in the second text line; determine all target character information from the first character information to the second character information, and use the target character information as the text recognition result.

[0122] The text recognition module 203 is also used to project the text recognition result onto the display module of the wearable device; and in response to the control instruction, determine the starting point and end point corresponding to the target text from the text recognition result.

[0123] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0125] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Fig.11 The wearable device shown is running on the

[0126] See also Fig.11 , Fig.11 is a schematic diagram of a wearable device provided in an embodiment of the present application. The wearable device may be a server.

[0127] like Fig.11 As shown, the wearable device 400 includes a processor 401, a memory 402, an image acquisition module 403 and a network interface connected via a system bus, wherein the memory 402 may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0128] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor 401 can execute any text recognition method based on a wearable device.

[0129] The processor 401 is used to provide computing and control capabilities to support the operation of the entire wearable device 400 .

[0130] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any text recognition method based on the wearable device.

[0131] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the wearable device is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the wearable device 400 to which the present application scheme is applied. The specific wearable device 400 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] It should be understood that the processor 401 may be a central processing unit (CPU), and the processor 401 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0133] Among them, in some embodiments, the processor 401 is used to run a computer program stored in a memory to implement the following steps: acquiring an image stream to be identified through an image acquisition unit of the wearable device; determining fingertip key points from the image stream to be identified, and determining the identification starting position and the starting image frame based on the fingertip key points; performing character-level text recognition from the starting image frame and the identification starting position until text recognition is ended at the identification ending position and the ending image frame to obtain a text recognition result.

[0134] In some embodiments, the processor 401 is also used to obtain the fingertip key points in the image stream to be identified, and to obtain the finger joint key points related to the fingertip key points; when the fingertip key points and the finger joint key points meet the first preset requirements, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the identification starting position.

[0135] In some embodiments, the processor 401 is also used to perform mutual correlation matching on the fingertip key points and the finger joint key points, and when the match is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; the fingertip area is segmented from the starting image frame according to the fingertip key points, and the fingertip coordinates are determined from the fingertip area; when the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the identification starting position.

[0136] In some embodiments, the processor 401 is further used to use the Laplace variance of the starting image frame as a benchmark variance; obtain the Laplace variance of each image frame after the starting image frame; and perform character-level text recognition on the image frames whose Laplace variance is greater than the benchmark variance until text recognition ends at the recognition end position and the end image frame.

[0137] In some embodiments, the processor 401 is also used to use the image frame corresponding to the fingertip key point as the end image frame and the position corresponding to the fingertip key point as the identification end position when the fingertip key point meets the second preset requirement, wherein the end image frame is the start image frame or an image frame after the start image frame.

[0138] In some embodiments, the processor 401 is also used to determine a first image area in the starting image frame based on the recognition starting position, wherein the size of the first image area is smaller than the size of the starting image frame; determine a second image area in the next image frame based on the recognition starting position, wherein the size of the second image area is smaller than the size of the next image frame; determine a transformation matrix based on the first image area and the second image area; use the transformation matrix to determine the position corresponding to the recognition starting position in the next image frame, and perform character-level text recognition based on the fingertip key points in the next image frame.

[0139] In some embodiments, the processor 401 is also used to obtain target key points and feature descriptors in the first image area and the second image area; perform feature matching based on the target key points and the feature descriptors to obtain a matching pair between the first image area and the second image area; and determine the transformation matrix based on the matching pair.

[0140] In some embodiments, the processor 401 is also used to determine the first text line closest to the recognition starting position, and determine the first character information of the character closest to the recognition starting position based on the positions of each character in the first text line; determine the second text line closest to the recognition ending position, and determine the second character information of the character closest to the recognition ending position based on the positions of each character in the second text line; determine all target character information from the first character information to the second character information, and use the target character information as the text recognition result.

[0141] In some embodiments, the processor 401 is further configured to project the text recognition result onto a display module of the wearable device; and in response to a control instruction, determine a starting point and an end point corresponding to the target text from the text recognition result.

[0142] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any one of the text recognition methods based on a wearable device provided in the embodiment of the present application is implemented.

[0143] The computer-readable storage medium may be an internal storage unit of the wearable device described in the foregoing embodiment, such as a hard disk or memory of the wearable device. The computer-readable storage medium may also be an external storage device of the wearable device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the wearable device.

[0144] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A text recognition method based on a wearable device, characterized in that: The method comprises: Collecting a to-be-recognized image stream through the wearable device; Determine fingertip key points from the image stream to be recognized, and determine a recognition start position and a start image frame based on the fingertip key points; Character-level text recognition is performed starting from the starting image frame and the recognition starting position until text recognition is terminated at the recognition ending position and the ending image frame to obtain a text recognition result.

2. The method according to claim 1, characterized in that The step of determining fingertip key points from the image stream to be identified, and determining the identification start position and the start image frame based on the fingertip key points, comprises: Acquire the fingertip key points in the to-be-recognized image stream, and acquire the finger joint key points related to the fingertip key points; When the fingertip key point and the finger joint key point meet the first preset requirement, the image frame corresponding to the fingertip key point is used as the starting image frame, and the position corresponding to the fingertip key point is used as the recognition starting position.

3. The method according to claim 2, characterized in that When the fingertip key point and the finger joint key point meet the first preset requirement, the image frame corresponding to the fingertip key point is used as the starting image frame, and the position corresponding to the fingertip key point is used as the recognition starting position, including: The fingertip key points and the finger joint key points are matched with each other, and when the matching is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; Segmenting a fingertip region from the starting image frame according to the fingertip key points, and determining fingertip coordinates from the fingertip region; When the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the recognition starting position.

4. The method according to claim 2, characterized in that: After taking the image frame corresponding to the fingertip key point as the starting image frame, the method includes: Using the Laplace variance of the starting image frame as a reference variance; The step of performing character-level text recognition from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame comprises: Obtaining the Laplace variance of each image frame after the starting image frame; Character-level text recognition is performed on the image frame whose Laplace variance is greater than the reference variance until text recognition ends at the recognition end position and the end image frame.

5. The method according to claim 2, characterized in that: The step of ending text recognition at the recognition end position and the end image frame includes: When the fingertip key point meets the second preset requirement, the image frame corresponding to the fingertip key point is used as the end image frame, and the position corresponding to the fingertip key point is used as the recognition end position, wherein the end image frame is the start image frame or an image frame after the start image frame.

6. The method according to claim 1, characterized in that After the character-level text recognition is started from the starting image frame and the recognition starting position, the method includes: Determining a first image region in the starting image frame based on the identification starting position, wherein a size of the first image region is smaller than a size of the starting image frame; determining a second image area in a next image frame based on the recognition start position, wherein a size of the second image area is smaller than a size of the next image frame; determining a transformation matrix based on the first image region and the second image region; The transformation matrix is ​​used to determine the position corresponding to the recognition start position in the next image frame, and character-level text recognition is performed based on the fingertip key points in the next image frame.

7. The method according to claim 6, characterized in that The determining of a transformation matrix based on the first image area and the second image area comprises: Acquire a first target key point of the first image area, and acquire a second target key point of the second image area; Performing feature matching on the first target key point and the second target key point to obtain a matching pair between the first image area and the second image area; The transformation matrix is ​​determined based on the matching pairs.

8. The method according to claim 1, characterized in that The character-level text recognition is performed from the starting image frame and the recognition starting position until the text recognition is terminated at the recognition ending position and the ending image frame to obtain the text recognition result, including: Determine a first text line closest to the recognition start position, and determine first character information of the character closest to the recognition start position according to the positions of each character in the first text line; Determine a second text line closest to the recognition end position, and determine second character information of the character closest to the recognition end position according to the positions of each character in the second text line; All target character information from the first character information to the second character information is determined, and the target character information is used as the text recognition result.

9. The method according to claim 1, characterized in that: After obtaining the text recognition result, the following steps are included: Projecting the text recognition result onto a display module of the wearable device; In response to the control instruction, a starting point and an end point corresponding to the target text are determined from the text recognition result.

10. A text recognition device, characterized in that: include: A collection module, the collection module is used to collect the image stream to be identified through the wearable device; A determination module, the determination module is used to determine the fingertip key points from the image stream to be identified, and determine the identification starting position and the starting image frame based on the fingertip key points; The text recognition module is used to perform character-level text recognition starting from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame to obtain a text recognition result.

11. A wearable device, characterized in that: include: An image acquisition module, a memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the steps of the text recognition method based on a wearable device as described in any one of claims 1 to 9 by running the program stored in the memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the text recognition method based on a wearable device as described in any one of claims 1 to 9.

Images