Wearable-device-based text recognition method, wearable device, and storage medium

Abstract

Provided in the embodiments of the present invention are a wearable-device-based text recognition method, a text recognition apparatus, a wearable device, and a storage medium. The method comprises: by means of a wearable device, collecting an image stream to be subjected to recognition; determining a key fingertip point from said image stream, and on the basis of the key fingertip point, determining a recognition start position and a start image frame; and starting character-level text recognition from the start image frame and the recognition start position until the text recognition ends at a recognition end position and an end image frame, so as to obtain a text recognition result. The embodiments of the present application aim to achieve wearable-device-based character-level text recognition in view of an image stream to be subjected to recognition and a key fingertip point, and also achieve real-time text recognition.

Description

Text recognition method based on wearable device, wearable device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 27, 2023, with application number 2023112616969 and invention name “Text recognition method based on wearable device, wearable device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a text recognition method based on a wearable device, a wearable device, and a computer-readable storage medium. Background Art

[0003] Optical character recognition (OCR) is the process by which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shape by detecting patterns of light and dark, and then uses character recognition methods to translate these shapes into computer text. With the advancement of technology, OCR can now be implemented not only on mobile phones and tablets, but also on wearable devices such as smart glasses and smart helmets.

[0004] In the related art, the optical character recognition function of wearable devices can be used to achieve text recognition processing such as text translation. However, due to the limited screen size and limited interaction methods of wearable devices, the text recognition processing based on wearable devices cannot, like mobile phones and tablets, freely select any word, phrase, sentence, paragraph, etc. in the entire text on the screen or using tools such as keyboard and mouse, that is, it is impossible to achieve free selection of text. In addition, because the text recognition processing method of wearable devices in the related art can only recognize words or entire paragraphs, the wearer can only recognize the text after the text is determined, and cannot obtain the recognition result while selecting the text, that is, it is impossible to obtain a real-time updated text recognition result.

[0005] Summary of the Invention

[0006] The present application provides a text recognition method based on a wearable device, a wearable device, and a computer-readable storage medium, which aims to combine the image stream to be recognized and the fingertip key points to realize character-level text recognition based on the wearable device while also realizing real-time text recognition.

[0007] To achieve the above objectives, the present application provides a text recognition method based on a wearable device, the method comprising:

[0008] Collecting an image stream to be identified through the wearable device;

[0009] Determining fingertip key points from the image stream to be recognized, and determining a recognition starting position and a starting image frame based on the fingertip key points;

[0010] Character-level text recognition is performed starting from the starting image frame and the recognition starting position, and ending at the recognition ending position and the ending image frame to obtain a text recognition result.

[0011] In addition, to achieve the above objectives, the present application also provides a text recognition device, comprising:

[0012] An acquisition module, configured to acquire an image stream to be identified through the wearable device;

[0013] a determination module, the determination module being configured to determine fingertip key points from the image stream to be identified, and to determine an identification starting position and a starting image frame based on the fingertip key points;

[0014] The text recognition module is used to perform character-level text recognition starting from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame to obtain a text recognition result.

[0015] In addition, to achieve the above-mentioned purpose, the present application further provides a wearable device, the wearable device comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the following steps when executing the computer program:

[0016] Collecting an image stream to be identified through the wearable device;

[0017] Determining fingertip key points from the image stream to be recognized, and determining a recognition starting position and a starting image frame based on the fingertip key points;

[0018] Character-level text recognition is performed starting from the starting image frame and the recognition starting position, and ending at the recognition ending position and the ending image frame to obtain a text recognition result.

[0019] In addition, to achieve the above-mentioned purpose, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the following steps:

[0020] Collecting an image stream to be identified through the wearable device;

[0021] Determining fingertip key points from the image stream to be recognized, and determining a recognition starting position and a starting image frame based on the fingertip key points;

[0022] Character-level text recognition is performed starting from the starting image frame and the recognition starting position, and ending at the recognition ending position and the ending image frame to obtain a text recognition result.

[0023] The text recognition method, text recognition device, wearable device and computer-readable storage medium based on wearable devices disclosed in the embodiments of the present application can collect the image stream to be recognized through the wearable device, determine the fingertip key points from the image stream to be recognized, and determine the recognition starting position and the starting image frame based on the fingertip key points. Thus, character-level text recognition can be started from the starting image frame and the recognition starting position until the text recognition ends at the recognition end position and the end image frame to obtain the text recognition result. The present application collects the image stream to be recognized through the image acquisition unit of the wearable device, and determines the recognition starting position using the fingertip key points, and then starts to recognize the text directly on the starting image frame, so that the wearer can perform text recognition in real time according to the position of the fingertip key points, thereby meeting the real-time nature of text recognition. Furthermore, through the positioning of the fingertip key points, the recognition starting position can be freely selected, and by combining it with character-level text recognition, it is possible to freely select characters, sentences and paragraphs in the text, providing a more flexible text selection capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] FIG1 is a flow chart of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0026] FIG2 is a schematic diagram of a process for determining a recognition starting position and a starting image frame according to an embodiment of the present application;

[0027] FIG3 is a schematic diagram of a first scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0028] FIG4 is a schematic diagram of a scenario for determining fingertip coordinates provided by an embodiment of the present application;

[0029] FIG5 is a schematic diagram of a flow chart of text recognition provided by an embodiment of the present application;

[0030] FIG6 is a schematic diagram of a second scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0031] FIG7 is a schematic diagram of a third scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0032] FIG8 is a schematic diagram of a process for obtaining text recognition results provided by an embodiment of the present application;

[0033] FIG9 is a schematic diagram of a fourth scenario of a text recognition method based on a wearable device provided in an embodiment of the present application;

[0034] FIG10 is a schematic block diagram of a text recognition device provided in an embodiment of the present application;

[0035] FIG11 is a schematic block diagram of a wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] The flowcharts shown in the accompanying drawings are illustrative only and do not necessarily include all content and operations / steps, nor do they necessarily need to be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual order of execution may vary depending on the actual situation. In addition, although the functional modules are divided in the device schematics, in some cases, the module division may be different from that shown in the device schematics.

[0038] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0039] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0040] Please refer to Figure 1, which is a flowchart of a text recognition method based on a wearable device provided by an embodiment of the present application. As shown in Figure 1, the text recognition method based on a wearable device includes steps S11 to S13.

[0041] Step S11: collecting the image stream to be identified through the wearable device.

[0042] Among them, the wearable device can be smart glasses, such as AR glasses, VR glasses, etc., which is not limited in this application.

[0043] Furthermore, the wearable device can be equipped with an image acquisition unit to capture the image stream to be recognized. The image acquisition unit includes components such as a camera, sensor, and lens installed on the wearable device. These components work together to capture images and information about the surrounding environment.

[0044] The image stream to be identified is a continuous image frame collected by the image collection unit. In the image stream to be identified, each image frame is continuous in time and presented in time sequence, thus forming a continuous image stream.

[0045] It can be understood that the image stream to be recognized includes at least text and the wearer's hand information. Among them, the hand information may include the fingertips of the fingers. The fingertips of any finger (such as the fingertip of the index finger) may correspond to characters in the text, which can be used to select the characters to be recognized. Specifically, the movement of the wearer's fingertips on the text can be captured by the image acquisition unit, thereby determining the starting point and end point of the fingertips on the text, and then determining the character corresponding to the starting point of the fingertip, the character corresponding to the end point of the fingertip, and all characters between the two as the characters to be recognized.

[0046] Furthermore, if the wearer's fingertip remains on the text for a predetermined period of time, the image acquisition unit can directly capture an image stream of the wearer's fingertip on the text during that period of time to determine the character to be recognized. Thus, the image stream to be recognized can be used to determine the text to be recognized.

[0047] In an embodiment of the present application, the image stream to be recognized can be collected by the image acquisition unit of the wearable device to determine the text to be recognized.

[0048] Step S12: determining fingertip key points from the image stream to be recognized, and determining the recognition starting position and starting image frame based on the fingertip key points.

[0049] Among them, the fingertip key point is a significant point at a specific position of the fingertip, usually the position of the finger end.

[0050] Specifically, the target detection algorithm can be used to detect the image stream to be identified, and the hand area in the image stream to be identified can be identified. Furthermore, the key point detection algorithm can be used to perform key point detection on the hand area to determine the fingertip key points. Among them, the target detection algorithm includes Haar cascade classifier, YOLO (You Only Look Once) model, etc.; the key point detection algorithm includes convolutional neural network, stacked hourglass model, etc., which are not limited in this application.

[0051] It is understandable that by determining the recognition starting position and starting image frame based on the fingertip key points, the recognition starting position can be used as the basis for word retrieval, that is, to determine whether the current wearer is in the word retrieval state. In this way, starting from the starting image frame, the movement of the fingertip key points in subsequent image frames can be tracked, and the starting and ending points of the fingertips can be determined on the text, so that the text to be recognized can be selected and recognized based on the starting and ending points of the fingertips. It is understandable that the starting position is a position in the starting image frame.

[0052] It should be noted that this application does not limit the method for determining the recognition starting position and starting image frame based on the fingertip key points. For example, the recognition starting position and starting image frame can be determined based on the correspondence between the fingertip key points and the finger joint joint points and whether the fingertip key points stay for more than a preset time. This application will be described in detail later.

[0053] In an embodiment of the present application, the fingertip key points can be determined from the image stream to be identified, and the identification starting position and starting image frame can be determined based on the fingertip key points. Then, based on the identification starting position, it can be determined that the current wearer is in a word-capturing state, so as to track the movement of the fingertip key points of the finger in subsequent image frames starting from the starting image frame, and simultaneously perform text selection and recognition.

[0054] Step S13: character-level text recognition is performed starting from the starting image frame and the recognition starting position, until the text recognition is completed at the recognition ending position and the ending image frame, to obtain a text recognition result.

[0055] Specifically, character-level text recognition can be performed using an optical character recognition model starting from the starting image frame and the recognition starting position. Furthermore, starting from the starting image frame, the movement of the fingertip key points in subsequent image frames is tracked, and the starting point and end point of the fingertip can be determined on the text, wherein the starting point of the fingertip corresponds to the starting image frame, and the end point of the fingertip corresponds to the ending image frame. In this way, the text to be recognized can be selected and recognized based on the starting point and end point of the fingertip, until the text recognition is completed at the recognition end position and the ending image frame, and the text recognition result is obtained.

[0056] It should be noted that this application does not limit the method for determining the end position and end image frame of recognition. For example, it can be determined based on the dwell time of a fingertip key point. That is, when the dwell time of a fingertip key point exceeds a preset duration, the position corresponding to the current fingertip key point can be determined as the end position of recognition, and the image frame corresponding to the end position of recognition can be determined as the end image frame. The end image frame is the starting image frame or an image frame after the starting image frame.

[0057] In addition, it can also be determined based on the wearer's instructions. For example, when the wearer finishes word capture, an end instruction can be sent to the wearable device, and then the wearable device can use the position corresponding to the current fingertip key point as the recognition end position based on the end instruction, and then determine the image frame corresponding to the recognition end position as the end image frame, and end word capture.

[0058] In an embodiment of the present application, starting from the starting image frame and the recognition starting position, word extraction can be performed character by character in continuous image frames, and text recognition can be performed through the optical character recognition model until the word extraction and text recognition are completed at the recognition end position and the end image frame, and the text recognition result is obtained. Since the present application uses the fingertip key points to determine the recognition starting position, and then starts to recognize the text directly on the starting image frame, the wearer can perform text recognition in real time according to the position of the fingertip key points, thereby meeting the real-time nature of text recognition. Furthermore, by positioning the fingertip key points, the recognition starting position can be freely selected, and by combining it with character-level text recognition, it is possible to freely select characters, sentences, and paragraphs in the text, providing a more flexible text selection capability.

[0059] Optionally, text recognition is ended until the recognition end position and the end image frame are recognized, including: when the fingertip key point meets the second preset requirement, the image frame corresponding to the fingertip key point is used as the end image frame, and the position corresponding to the fingertip key point is used as the recognition end position, wherein the end image frame is the starting image frame or the image frame after the starting image frame.

[0060] Among them, the second preset requirement can be that the stay time of the fingertip key point exceeds the preset time, for example, the preset time is 2s, 3s, etc. This application takes the preset time of 2s as an example for explanation.

[0061] Specifically, after determining the recognition starting position, if the dwell time of the fingertip key point exceeds 2 seconds, the image frame corresponding to the fingertip key point can be used as the end image frame, and the position corresponding to the fingertip key point can be used as the recognition end position.

[0062] It is understandable that the recognition end position can be the same as the recognition start position, or the recognition end position can be after the recognition start position. Similarly, the end image frame is the start image frame or an image frame after the start image frame.

[0063] In an embodiment of the present application, when it is determined that the fingertip key point meets the second preset requirement, the recognition end position and the end image frame can be determined based on the fingertip key point, and text recognition can be ended at the recognition end position and the end image frame.

[0064] Optionally, character-level text recognition is performed starting from the starting image frame and the recognition starting position until text recognition is terminated at the recognition ending position and the end image frame, including: taking the Laplace variance of the starting image frame as a reference variance; obtaining the Laplace variance of each image frame after the starting image frame; performing character-level text recognition on image frames whose Laplace variance is greater than the reference variance, until text recognition is terminated at the recognition ending position and the end image frame.

[0065] Understandably, as the image acquisition unit captures the image stream to be recognized, the camera's position and angle may change, resulting in some of the captured image frames being blurred or unstable, making text recognition impossible. Therefore, it is necessary to determine the stability of the image frames based on the time series to prevent unstable image frames from affecting the text recognition results.

[0066] Because the wearer's fingertip has not moved when the recognition starting position is determined, the starting image frame corresponding to the recognition starting position is relatively stable. Therefore, the Laplace variance of the starting image frame can be calculated and used as the baseline variance. Laplace variance is a metric used to measure image variation, describing the degree of difference between different pixels in the image. Therefore, the Laplace variance of each subsequent image frame can be determined in a time series and compared to the baseline variance to assess the stability of the image frame.

[0067] Optionally, to reduce the computational effort of calculating the Laplace variance, each frame of the image stream to be recognized can be converted into a grayscale image. Because grayscale images have only one channel, they are simpler and more efficient to process than color images, which have three channels (red, green, and blue), and require less computation.

[0068] It can be understood that when the Laplace variance of an image frame is greater than the baseline variance, it indicates that the image frame is relatively stable and can be used for text recognition. Therefore, image frames with a Laplace variance greater than the baseline variance can be used for character-level text recognition until text recognition ends at the recognition end position and the end image frame.

[0069] In this embodiment of the present application, the Laplace variance of the starting image frame can be used as a measure of image stability. By comparing it with the Laplace variances of other image frames, the stability of the other image frames can be determined. This allows text recognition to be performed when the other image frames are more stable, improving the accuracy and reliability of the text recognition results.

[0070] The text recognition method based on a wearable device disclosed in an embodiment of the present application can collect the image stream to be recognized through the image acquisition unit of the wearable device, determine the fingertip key points from the image stream to be recognized, and determine the recognition starting position and the starting image frame based on the fingertip key points. Thus, character-level text recognition can be performed starting from the starting image frame and the recognition starting position until the text recognition ends at the recognition end position and the end image frame to obtain the text recognition result. The present application collects the image stream to be recognized through the image acquisition unit of the wearable device, and determines the recognition starting position using the fingertip key points, and then starts recognizing the text directly on the starting image frame, so that the wearer can perform text recognition in real time according to the position of the fingertip key points, thereby meeting the real-time nature of text recognition. Furthermore, through the positioning of the fingertip key points, the recognition starting position can be freely selected, and by combining it with character-level text recognition, it is possible to freely select characters, sentences, paragraphs, etc. in the text, providing a more flexible text selection capability.

[0071] Please refer to Figures 2 to 4. Figure 2 is a schematic diagram of a process for determining a recognition starting position and a starting image frame according to an embodiment of the present application; Figure 3 is a schematic diagram of a first scenario of a text recognition method based on a wearable device according to an embodiment of the present application; and Figure 4 is a schematic diagram of a scenario for determining fingertip coordinates according to an embodiment of the present application. As shown in Figure 2, steps S121 to S122 can be used to determine the recognition starting position and starting image frame based on fingertip key points.

[0072] Step S121: obtaining fingertip key points in the image stream to be identified, and obtaining finger joint key points related to the fingertip key points.

[0073] Among them, the fingertip key point is a significant point at a specific position of the fingertip, usually the position of the finger end.

[0074] The knuckle key points related to the fingertip key points are the significant points of the knuckles of the hand where the fingertips are located, that is, the positions of the knuckles.

[0075] Specifically, the target detection algorithm can be used to detect the image stream to be identified, and the hand area in the image stream to be identified can be identified. Furthermore, the key point detection algorithm can be used to detect the key points of the hand area to determine the fingertip key points and the knuckle key points.

[0076] Among them, the target detection algorithm includes Haar cascade classifier, YOLO (You Only Look Once) model, etc.; the key point detection algorithm includes convolutional neural network, stacked hourglass model, etc., which are not limited in this application.

[0077] Step S122: When the fingertip key point and the finger joint key point meet the first preset requirement, the image frame corresponding to the fingertip key point is used as the starting image frame, and the position corresponding to the fingertip key point is used as the recognition starting position.

[0078] Optionally, when the fingertip key points and the knuckle key points meet the first preset requirements, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the recognition starting position, including: performing mutual correlation matching on the fingertip key points and the knuckle key points, and using the image frame corresponding to the fingertip key points as the starting image frame when the matching is successful; segmenting the fingertip area from the starting image frame according to the fingertip key points, and determining the fingertip coordinates from the fingertip area; when the fingertip coordinates exist in all image frames within a preset time, using the fingertip coordinates as the recognition starting position.

[0079] As shown in Figure 3, when matching fingertip key points with knuckle key points, multiple knuckle key points can be connected with the wrist key points to form a polygonal closed region. This region is then used to determine whether the fingertip key points are within the closed region to determine whether the wearer is in the word capture state. If the fingertip key points are within the closed region, the match fails and the wearer is not in the word capture state. If the fingertip key points are not within the closed region, the match succeeds and the wearer is in the word capture state. In this case, if the match is successful, the image frame corresponding to the fingertip key points can be used as the starting image frame.

[0080] Furthermore, as shown in FIG4 , a specific area can be selected on the starting image frame with the fingertip key point as the center to segment the fingertip area. The specific area can be a rectangular or elliptical area of ​​a fixed size to ensure that the fingertip is covered. Within the fingertip area, a fixed coordinate system can be pre-established to determine the fingertip coordinates. It is then determined whether the fingertip coordinates exist in all image frames within a preset time, and whether the change in the fingertip coordinate system within the preset time period is less than a preset threshold. In this way, it can be determined whether the fingertip coordinates can be used as the starting position for recognition.

[0081] It is understandable that if the fingertip coordinates are present in all image frames within the preset time period, and the change in the fingertip coordinates within the preset time period is less than a preset threshold, it means that the wearer's fingertip has stayed at a fixed position for the preset time, and the fingertip coordinates can be determined as the recognition starting position.

[0082] It should be noted that the present application does not limit the above-mentioned preset time, for example, the preset time is 2s, 3s, 5s, etc.; further, the present application does not limit the preset threshold, for example, it can be 1px, 3px and 5px, etc.

[0083] In the embodiment of the present application, the information of the fingertip key points and the finger joint key points can be fully utilized, and accurate and reliable identification of the starting position and the determination of the starting image frame can be achieved through mutual correlation matching, image segmentation and determination of the fingertip coordinates.

[0084] Please refer to Figures 5 to 7. Figure 5 is a schematic diagram of a text recognition process according to an embodiment of the present application; Figure 6 is a schematic diagram of a second scenario of a text recognition method based on a wearable device according to an embodiment of the present application; and Figure 7 is a schematic diagram of a third scenario of a text recognition method based on a wearable device according to an embodiment of the present application. As shown in Figure 5, after character-level text recognition is performed starting from the starting image frame and the recognition starting position, text recognition can be performed through steps S131 to S134.

[0085] Step S131: determining a first image area in a starting image frame based on the identification starting position.

[0086] The size of the first image area is smaller than the size of the starting image frame.

[0087] Step S132: determining a second image area in the next image frame based on the recognition starting position.

[0088] The size of the second image area is smaller than the size of the next image frame.

[0089] Step S133: Determine a transformation matrix based on the first image region and the second image region.

[0090] The first image area is an image area including the fingertip key points in the starting image frame; and the second image area is an image area including the fingertip key points in the next image frame.

[0091] Specifically, a first image region and a second image region of a certain size may be cropped from the starting image frame and the next image frame, respectively. For example, cropping may be performed with the recognition starting position as the center, thereby obtaining the first image region and the second image region including the fingertip key points.

[0092] It should be noted that the present application does not limit the size of the first image area and the second image area, for example, 300px*300px, 300px*450px, 450px*450px, etc.

[0093] Optionally, the first image area and the second image area have the same size and both have their center points as the recognition starting position. In this way, the first image area can be directly mapped to the second image area to determine the transformation matrix, thereby improving the efficiency of determining the transformation matrix.

[0094] Optionally, determining a transformation matrix based on the first image area and the second image area includes: obtaining a first target key point of the first image area, and obtaining a second target key point of the second image area; performing feature matching on the first target key point and the second target key point to obtain a matching pair between the first image area and the second image area; and determining the transformation matrix based on the matching pair.

[0095] The first target key point is a key point corresponding to the first image area; the second target key point is a key point corresponding to the second image area.

[0096] It's important to note that keypoints are a concept used in computer vision and image processing to represent important features in an image. Keypoints, also known as points of interest or corners, are typically found at edges, corners, or where texture changes occur. Keypoints can be considered prominent points in an image. They possess high contrast and structure within a local area, making them more sensitive to image changes. Keypoints help represent local features of an image and are used in tasks such as feature matching and object recognition.

[0097] Furthermore, the present application does not limit the types of the first target key point and the second target key point, and they may include, for example, corner points and edges of an image.

[0098] As shown in Figure 6, the image on the left is the first image region, and the image on the right is the second image region. Specifically, target key points (such as corner points) can be extracted from the first image region and the second image region, respectively. Furthermore, feature descriptors for the corner points of the first image region and the corner points of the second image region can be determined, and feature matching operations can be performed on the two to find the most similar matching pairs. Thus, a transformation matrix can be calculated based on the matching pairs using a least squares method, a random sampling consensus algorithm, or the like.

[0099] It's important to note that a feature descriptor is a vector or matrix representation of pixel information surrounding a keypoint, capturing the appearance and structural characteristics of the area surrounding the keypoint. Feature descriptors provide a way to compare corresponding keypoints in different images, enabling tasks like image matching and object tracking. Typically, a feature descriptor extracts information such as gradients, color histograms, and local binary patterns of pixels surrounding a keypoint, encoding this information into a vector or matrix, creating a compact representation of the keypoint's features.

[0100] Furthermore, the present application does not limit the above-mentioned feature descriptors. For example, the feature descriptors may be information such as gradients, color histograms, and local binary patterns corresponding to corner points.

[0101] Optionally, feature matching of the feature descriptors of the corner points of the first image region and the feature descriptors of the corner points of the second image region can be achieved using methods such as nearest neighbor matching, nearest neighbor matching and thresholding. Specifically, for each feature descriptor of the corner points of the first image region, the feature descriptor of the corner point of the second image region with the closest Euclidean distance can be found using the above method, thereby finding the most similar matching pair.

[0102] Step S134: using the transformation matrix, determining the position corresponding to the recognition start position in the next image frame, and performing character-level text recognition based on the fingertip key points in the next image frame.

[0103] As shown in Figure 7, after obtaining the transformation matrix, each coordinate point in the current image region can be applied to the next image region, thereby achieving a mapping between these image regions. This allows the recognition starting position in the current image region to be mapped to the next image frame. In other words, the position corresponding to the recognition starting position is determined in the next image frame, allowing character-level text recognition to be performed based on the fingertip key points in the next image frame.

[0104] In the embodiment of the present application, feature matching and transformation matrices can be used to more accurately determine the mapping relationship between the current image area and the next image area, thereby determining the position corresponding to the recognition starting position in the next image frame. This allows the wearer to perform real-time text recognition based on the position of the key points on the fingertips, ensuring real-time text recognition.

[0105] Please refer to Figures 8 and 9. Figure 8 is a schematic diagram of a process for obtaining text recognition results according to an embodiment of the present application; Figure 9 is a schematic diagram of a fourth scenario of a text recognition method based on a wearable device according to an embodiment of the present application. As shown in Figure 8, the text recognition results can be obtained through steps S135 to S137.

[0106] Step S135: determining the first text line closest to the recognition start position, and determining first character information of the character closest to the recognition start position according to the positions of the characters in the first text line.

[0107] Step S136: determining a second text line closest to the recognition end position, and determining second character information of the character closest to the recognition end position according to the positions of the characters in the second text line.

[0108] Step S137: determining all target character information from the first character information to the second character information, and taking the target character information as the text recognition result.

[0109] The first text line is the text line that starts recognition, that is, the text line closest to the recognition start position; the second text line is the text line that ends recognition, that is, the text line closest to the recognition end position.

[0110] Specifically, the first text line closest to the recognition start position can be determined, and based on the positions of the characters in the first text line, the character closest to the recognition start position in the first text line, i.e., the character to be recognized at the start, can be determined. Furthermore, the character to be recognized at the start can be recognized to obtain the corresponding first character information. Similarly, the character closest to the recognition end position in the second text line, i.e., the character to be recognized at the end, can be determined, and the character to be recognized at the end can be recognized to obtain the corresponding second character information.

[0111] It can be understood that after determining the character to start recognition and the character to end recognition, all characters between the two can be recognized to obtain target character information, and then the target character information can be used as the text recognition result.

[0112] In an embodiment of the present application, the character to be started for recognition and the character to be ended for recognition can be positioned according to the recognition start position and the recognition end position, respectively, and then all characters between the two can be recognized to obtain a text recognition result.

[0113] Optionally, after obtaining the text recognition result, the method includes: projecting the text recognition result onto a display module of the wearable device; and determining the starting point and end point corresponding to the target text from the text recognition result in response to a control instruction.

[0114] Among them, the display module refers to the component in the wearable device used to display images, text and other information.

[0115] Specifically, after obtaining the text recognition results, the text recognition results can be projected onto the display module of the wearable device. Furthermore, after receiving control commands such as gestures, voice, and buttons from the wearer, the starting and ending points corresponding to the target text can be determined from the text recognition results, that is, the range of text recognition is specified to obtain the target text.

[0116] Optionally, before obtaining the text recognition result, the recognized character information can be displayed in real time through the display module of the wearable device to provide the wearer with the recognized information in real time.

[0117] In the embodiment of the present application, the text recognition results can be displayed on the display module of the wearable device, and the target text can be determined from the text recognition results through control instructions. This provides a more flexible and personalized text browsing experience, allowing the wearer to select the target text according to their needs.

[0118] Please refer to Figure 10, which is a schematic block diagram of a text recognition device provided in an embodiment of the present application. The text recognition device can be configured in a server to execute the steps of the aforementioned wearable device-based text recognition method.

[0119] As shown in FIG10 , the text recognition device 200 includes: a collection module 201 , a determination module 202 , and a text recognition module 203 .

[0120] The acquisition module 201 is configured to acquire an image stream to be identified through the wearable device;

[0121] A determination module 202 is configured to determine fingertip key points from the image stream to be recognized, and determine a recognition starting position and a starting image frame based on the fingertip key points;

[0122] The text recognition module 203 is configured to perform character-level text recognition starting from the starting image frame and the recognition starting position, and ending the text recognition at the recognition ending position and the ending image frame, to obtain a text recognition result.

[0123] The determination module 202 is also used to obtain the fingertip key points in the image stream to be identified, and to obtain the finger joint key points related to the fingertip key points; when the fingertip key points and the finger joint key points meet the first preset requirements, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the identification starting position.

[0124] The determination module 202 is also used to perform mutual correlation matching on the fingertip key points and the finger joint key points, and when the matching is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; the fingertip area is segmented from the starting image frame according to the fingertip key points, and the fingertip coordinates are determined from the fingertip area; when the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the recognition starting position.

[0125] The determination module 202 is also used to connect multiple finger joint key points and wrist key points to obtain a polygonal closed area, and determine whether the fingertip key point is within the closed area; if the fingertip key point is not within the closed area, it is determined that the fingertip key point and the finger joint key point are successfully matched, and the image frame corresponding to the fingertip key point is used as the starting image frame.

[0126] The determination module 202 is further configured to use the fingertip coordinates as the identification starting position if the fingertip coordinates exist in all image frames within a preset time and the change in the fingertip coordinate system within the preset time period is less than a preset threshold.

[0127] The text recognition module 203 is further configured to use the Laplace variance of the starting image frame as a reference variance;

[0128] Acquire the Laplace variance of each image frame after the starting image frame; perform character-level text recognition on the image frame whose Laplace variance is greater than the reference variance, until text recognition ends at the recognition end position and the end image frame.

[0129] The text recognition module 203 is also used to use the image frame corresponding to the fingertip key point as the end image frame and the position corresponding to the fingertip key point as the recognition end position when the fingertip key point meets the second preset requirement, wherein the end image frame is the starting image frame or an image frame after the starting image frame.

[0130] The text recognition module 203 is also used to determine a first image area in the starting image frame based on the recognition starting position, wherein the size of the first image area is smaller than the size of the starting image frame; determine a second image area in the next image frame based on the recognition starting position, wherein the size of the second image area is smaller than the size of the next image frame; determine a transformation matrix based on the first image area and the second image area; use the transformation matrix to determine the position corresponding to the recognition starting position in the next image frame, and perform character-level text recognition based on the fingertip key points in the next image frame.

[0131] The text recognition module 203 is also used to obtain the first target key point of the first image area and the second target key point of the second image area; perform feature matching on the first target key point and the second target key point to obtain a matching pair between the first image area and the second image area; and determine the transformation matrix based on the matching pair.

[0132] The text recognition module 203 is also used to determine the first text line closest to the recognition starting position, and determine the first character information of the character closest to the recognition starting position based on the positions of each character in the first text line; determine the second text line closest to the recognition ending position, and determine the second character information of the character closest to the recognition ending position based on the positions of each character in the second text line; determine all target character information from the first character information to the second character information, and use the target character information as the text recognition result.

[0133] The text recognition module 203 is further configured to project the text recognition result onto the display module of the wearable device; and in response to a control instruction, determine the starting point and end point corresponding to the target text from the text recognition result.

[0134] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] The methods and apparatus of the present application can be used in a wide variety of general-purpose or specialized computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above.

[0136] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a wearable device as shown in FIG11 .

[0137] Please refer to Figure 11, which is a schematic diagram of a wearable device provided in an embodiment of the present application. The wearable device can be a server.

[0138] As shown in FIG11 , the wearable device 400 includes a processor 401 , a memory 402 , an image acquisition module 403 , and a network interface connected via a system bus, wherein the memory 402 may include a volatile storage medium, a non-volatile storage medium, and an internal memory.

[0139] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, which, when executed, can enable the processor 401 to perform any one of the wearable device-based text recognition methods.

[0140] The processor 401 is used to provide computing and control capabilities to support the operation of the entire wearable device 400.

[0141] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any text recognition method based on the wearable device.

[0142] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that the structure of the wearable device is merely a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the wearable device 400 to which the present application solution is applied. The specific wearable device 400 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0143] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0144] In some embodiments, the processor 401 is used to run a computer program stored in a memory to implement the following steps: acquiring an image stream to be identified through an image acquisition unit of the wearable device; determining fingertip key points from the image stream to be identified, and determining the identification starting position and the starting image frame based on the fingertip key points; performing character-level text recognition from the starting image frame and the identification starting position until text recognition is completed at the identification end position and the end image frame to obtain a text recognition result.

[0145] In some embodiments, the processor 401 is also used to obtain the fingertip key points in the image stream to be identified, and to obtain the finger joint key points related to the fingertip key points; when the fingertip key points and the finger joint key points meet the first preset requirements, the image frame corresponding to the fingertip key points is used as the starting image frame, and the position corresponding to the fingertip key points is used as the identification starting position.

[0146] In some embodiments, the processor 401 is also used to perform mutual correlation matching on the fingertip key points and the finger joint key points, and when the matching is successful, the image frame corresponding to the fingertip key point is used as the starting image frame; the fingertip area is segmented from the starting image frame according to the fingertip key point, and the fingertip coordinates are determined from the fingertip area; when the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the identification starting position.

[0147] In some embodiments, the processor 401 is also used to connect multiple finger joint key points and wrist key points to obtain a polygonal closed area, and determine whether the fingertip key point is within the closed area; if the fingertip key point is not within the closed area, it is determined that the fingertip key point and the finger joint key point are successfully matched, and the image frame corresponding to the fingertip key point is used as the starting image frame.

[0148] In some embodiments, the processor 401 is further used to use the fingertip coordinates as the identification starting position when the fingertip coordinates exist in the image frames within a preset time and the change in the fingertip coordinate system within the preset time period is less than a preset threshold.

[0149] In some embodiments, the processor 401 is further used to use the Laplace variance of the starting image frame as a benchmark variance; obtain the Laplace variance of each image frame after the starting image frame; and perform character-level text recognition on the image frames whose Laplace variance is greater than the benchmark variance until text recognition is completed at the recognition end position and the end image frame.

[0150] In some embodiments, the processor 401 is also used to use the image frame corresponding to the fingertip key point as the end image frame and the position corresponding to the fingertip key point as the recognition end position when the fingertip key point meets the second preset requirement, wherein the end image frame is the starting image frame or an image frame after the starting image frame.

[0151] In some embodiments, the processor 401 is further used to determine a first image area in the starting image frame based on the recognition starting position, wherein the size of the first image area is smaller than the size of the starting image frame; determine a second image area in the next image frame based on the recognition starting position, wherein the size of the second image area is smaller than the size of the next image frame; determine a transformation matrix based on the first image area and the second image area; use the transformation matrix to determine the position corresponding to the recognition starting position in the next image frame, and perform character-level text recognition based on the fingertip key points in the next image frame.

[0152] In some embodiments, the processor 401 is further used to obtain target key points and feature descriptors in the first image area and the second image area; perform feature matching based on the target key points and the feature descriptors to obtain a matching pair between the first image area and the second image area; and determine the transformation matrix based on the matching pair.

[0153] In some embodiments, the processor 401 is also used to determine the first text line closest to the recognition starting position, and determine the first character information of the character closest to the recognition starting position based on the positions of each character in the first text line; determine the second text line closest to the recognition ending position, and determine the second character information of the character closest to the recognition ending position based on the positions of each character in the second text line; determine all target character information from the first character information to the second character information, and use the target character information as the text recognition result.

[0154] In some embodiments, the processor 401 is further configured to project the text recognition result onto a display module of the wearable device; and in response to a control instruction, determine the starting point and end point corresponding to the target text from the text recognition result.

[0155] The present application also provides a computer-readable storage medium having a computer program stored thereon. The computer program includes program instructions, and when the program instructions are executed, the following steps are implemented:

[0156] Collecting an image stream to be identified through the wearable device;

[0157] Determining fingertip key points from the image stream to be recognized, and determining a recognition starting position and a starting image frame based on the fingertip key points;

[0158] Character-level text recognition is performed starting from the starting image frame and the recognition starting position, and ending at the recognition ending position and the ending image frame to obtain a text recognition result.

[0159] The computer-readable storage medium may be an internal storage unit of the wearable device described in the aforementioned embodiment, such as a hard disk or memory of the wearable device. The computer-readable storage medium may also be an external storage device of the wearable device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the wearable device.

[0160] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0161] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A text recognition method based on a wearable device, wherein: The method comprises: Collecting an image stream to be identified through the wearable device; Determine fingertip key points from the image stream to be recognized, and determine a recognition start position and a start image frame based on the fingertip key points; Character-level text recognition is performed starting from the starting image frame and the recognition starting position until text recognition is terminated at the recognition ending position and the ending image frame to obtain a text recognition result.

2. The method according to claim 1, wherein: The step of determining fingertip key points from the image stream to be recognized, and determining a recognition start position and a start image frame based on the fingertip key points, includes: Acquire the fingertip key points in the to-be-recognized image stream, and acquire finger joint key points related to the fingertip key points; When the fingertip key point and the finger joint key point meet the first preset requirement, the image frame corresponding to the fingertip key point is used as the starting image frame, and the position corresponding to the fingertip key point is used as the recognition starting position.

3. The method according to claim 2, wherein: When the fingertip key point and the finger joint key point meet the first preset requirement, the image frame corresponding to the fingertip key point is used as the starting image frame, and the position corresponding to the fingertip key point is used as the recognition starting position, including: The fingertip key points and the finger joint key points are matched with each other, and when the matching is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; Segmenting a fingertip region from the starting image frame according to the fingertip key points, and determining fingertip coordinates from the fingertip region; When the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the recognition starting position.

4. The method according to claim 3, wherein: The step of matching the fingertip key points with the finger joint key points in a correlated manner, and taking the image frame corresponding to the fingertip key points as the starting image frame when the matching is successful, includes: Connecting a plurality of the finger joint key points and the wrist key points to obtain a polygonal closed area, and determining whether the fingertip key point is within the closed area; If the fingertip key point is not within the closed area, it is determined that the fingertip key point and the finger joint key point are matched successfully, and the image frame corresponding to the fingertip key point is used as the starting image frame.

5. The method according to claim 3, wherein: When the fingertip coordinates exist in all image frames within the preset time, taking the fingertip coordinates as the recognition starting position includes: When the fingertip coordinates exist in all the image frames within a preset time and the change amount of the fingertip coordinates in a preset time period is less than a preset threshold, the fingertip coordinates are used as the identification starting position.

6. The method according to claim 2, wherein: After taking the image frame corresponding to the fingertip key point as the starting image frame, the method includes: Using the Laplace variance of the starting image frame as a reference variance; The step of performing character-level text recognition from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame comprises: Obtaining the Laplace variance of each image frame after the starting image frame; Character-level text recognition is performed on the image frame whose Laplace variance is greater than the reference variance until text recognition ends at the recognition end position and the end image frame.

7. The method according to claim 2, wherein: The step of ending text recognition at the recognition end position and the end image frame includes: When the fingertip key point meets the second preset requirement, the image frame corresponding to the fingertip key point is used as the end image frame, and the position corresponding to the fingertip key point is used as the recognition end position, wherein the end image frame It is the starting image frame or an image frame after the starting image frame.

8. The method according to claim 1, wherein: After the character-level text recognition is started from the starting image frame and the recognition starting position, the method includes: Determining a first image region in the starting image frame based on the identification starting position, wherein a size of the first image region is smaller than a size of the starting image frame; determining a second image area in a next image frame based on the recognition start position, wherein a size of the second image area is smaller than a size of the next image frame; determining a transformation matrix based on the first image region and the second image region; The transformation matrix is ​​used to determine the position corresponding to the recognition start position in the next image frame, and character-level text recognition is performed based on the fingertip key points in the next image frame.

9. The method according to claim 8, wherein: The determining of a transformation matrix based on the first image area and the second image area comprises: Acquire a first target key point of the first image area, and acquire a second target key point of the second image area; Performing feature matching on the first target key point and the second target key point to obtain a matching pair between the first image region and the second image region; The transformation matrix is ​​determined based on the matching pairs.

10. The method according to claim 1, wherein: The character-level text recognition is performed from the starting image frame and the recognition starting position until the text recognition is terminated at the recognition ending position and the ending image frame to obtain the text recognition result, including: Determine a first text line closest to the recognition start position, and determine first character information of the character closest to the recognition start position according to the positions of each character in the first text line; Determine a second text line closest to the recognition end position, and determine second character information of the character closest to the recognition end position according to the positions of each character in the second text line; All target character information from the first character information to the second character information is determined, and the target character information is used as the text recognition result.

11. The method according to claim 1, wherein: After obtaining the text recognition result, the following steps are included: Projecting the text recognition result onto a display module of the wearable device; In response to the control instruction, a starting point and an end point corresponding to the target text are determined from the text recognition result.

12. A text recognition device, wherein: include: A collection module, the collection module is used to collect the image stream to be identified through the wearable device; A determination module, the determination module is used to determine the fingertip key points from the image stream to be identified, and determine the identification starting position and the starting image frame based on the fingertip key points; The text recognition module is used to perform character-level text recognition starting from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame to obtain a text recognition result.

13. A wearable device, wherein: include: An image acquisition module, a memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the following steps by running the program stored in the memory: Collecting an image stream to be identified through the wearable device; Determine fingertip key points from the image stream to be recognized, and determine a recognition start position and a start image frame based on the fingertip key points; Character-level text recognition is performed starting from the starting image frame and the recognition starting position until text recognition is terminated at the recognition ending position and the ending image frame to obtain a text recognition result.

14. The wearable device according to claim 13, wherein: When the processor determines the fingertip key points from the image stream to be recognized, and determines the recognition start position and the start image frame based on the fingertip key points, the processor is used to implement: Acquire the fingertip key points in the to-be-recognized image stream, and acquire finger joint key points related to the fingertip key points; When the fingertip key point and the finger joint key point meet the first preset requirement, the image frame corresponding to the fingertip key point is used as the starting image frame, and the position corresponding to the fingertip key point is used as the recognition starting position.

15. The wearable device according to claim 14, wherein: The processor, when the fingertip key point and the finger joint key point meet the first preset requirement, uses the image frame corresponding to the fingertip key point as the starting image frame, and uses the position corresponding to the fingertip key point as the recognition starting position, and is further used to implement: The fingertip key points and the finger joint key points are matched with each other, and when the matching is successful, the image frame corresponding to the fingertip key points is used as the starting image frame; Segmenting a fingertip region from the starting image frame according to the fingertip key points, and determining fingertip coordinates from the fingertip region; When the fingertip coordinates exist in all image frames within a preset time, the fingertip coordinates are used as the recognition starting position.

16. The wearable device according to claim 14, wherein: After implementing the step of using the image frame corresponding to the fingertip key point as the starting image frame, the processor is further configured to implement: Using the Laplace variance of the starting image frame as a reference variance; The step of performing character-level text recognition from the starting image frame and the recognition starting position until the text recognition ends at the recognition ending position and the ending image frame comprises: Obtaining the Laplace variance of each image frame after the starting image frame; Character-level text recognition is performed on the image frame whose Laplace variance is greater than the reference variance until text recognition ends at the recognition end position and the end image frame.

17. The wearable device according to claim 14, wherein: The processor implements the step of ending text recognition at the recognition end position and the end image frame, and is used to implement: When the fingertip key point meets the second preset requirement, the image frame corresponding to the fingertip key point is used as the end image frame, and the position corresponding to the fingertip key point is used as the recognition end position, wherein the end image frame is the start image frame or an image frame after the start image frame.

18. The wearable device according to claim 13, wherein: After implementing the character-level text recognition from the starting image frame and the recognition starting position, the processor is further used to implement: Determining a first image region in the starting image frame based on the identification starting position, wherein a size of the first image region is smaller than a size of the starting image frame; determining a second image area in a next image frame based on the recognition start position, wherein a size of the second image area is smaller than a size of the next image frame; determining a transformation matrix based on the first image region and the second image region; The transformation matrix is ​​used to determine the position corresponding to the recognition start position in the next image frame, and character-level text recognition is performed based on the fingertip key points in the next image frame.

19. The wearable device according to claim 18, wherein: The processor determines the transformation matrix based on the first image area and the second image area to implement: Acquire a first target key point of the first image area, and acquire a second target key point of the second image area; Performing feature matching on the first target key point and the second target key point to obtain a matching pair between the first image region and the second image region; The transformation matrix is ​​determined based on the matching pairs.

20. A computer-readable storage medium, wherein: The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the following steps: Collecting an image stream to be identified through the wearable device; Determine fingertip key points from the image stream to be recognized, and determine a recognition start position and a start image frame based on the fingertip key points; Character-level text recognition is performed starting from the starting image frame and the recognition starting position until text recognition is terminated at the recognition ending position and the ending image frame to obtain a text recognition result.

Images