Handwritten content recognition method and device, electronic equipment and computer program product
Through intelligent wearable devices and gesture muscle movement models, handwriting content recognition under specific writing equipment and good lighting conditions is achieved, solving the problem of device dependence and lighting sensitivity of the prior art, and improving the accuracy and applicability of the recognition.
Patent Information
- Application Number
- CN202410963155.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
AI Technical Summary
Existing handwriting content recognition technologies rely on specific writing equipment and lighting conditions, and cannot achieve real-time recognition in the absence of equipment or poor lighting.
The smart wearable device worn by the user obtains motion perception data, establishes a three-dimensional coordinate system, uses gesture muscle motion models to extract and identify gesture features, and recognizes writing content in real time without additional input devices.
Accurate identification of handwritten content in various environments, reduce dependence on specific hardware and lighting conditions, and improve user experience and work efficiency.
Smart Images

Figure CN118915910A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of human-computer interaction, and in particular to a handwriting content recognition method, device, electronic device, and computer program product. Background Art
[0002] In recent years, with the rapid development of human-computer interaction, handwriting recognition technology has made significant progress. This technology can convert the user's handwriting into digital text for easy storage or display, greatly facilitating the user's application in learning, work and life.
[0003] In the related technologies, touch screen input technology, digital handwriting tablet capture or image recognition are usually used to identify the user's handwritten content and convert it into a data format that can be understood and parsed by the computer. Touch screen devices track the movement of fingertips or stylus pens through built-in sensors; digital handwriting tablets use electromagnetic induction or pressure sensing technology to record writing movements; image acquisition solutions use cameras and other devices to capture image information of handwritten content on paper, and then apply image processing and pattern recognition algorithms to convert it into digital text.
[0004] However, this handwriting content recognition method relies on specific writing equipment, and image recognition has high requirements for lighting conditions. In the absence of relevant writing equipment or poor lighting conditions, real-time recognition of user writing content cannot be achieved. Summary of the invention
[0005] In view of this, exemplary embodiments of the present disclosure provide a handwritten content recognition method, apparatus, electronic device, and computer program product to solve the problems existing in the related art.
[0006] According to a first aspect of the exemplary embodiments of the present disclosure, a method for recognizing handwritten content is provided, the method comprising:
[0007] When detecting that the user starts writing, obtaining hand movement data of the user in the process of writing the text to be recognized;
[0008] Extracting features to be recognized from the hand motion data, and matching content parameter recognition results of the text to be recognized based on the features to be recognized;
[0009] Based on the content parameter recognition result and the hand movement data, the written content of the text to be recognized is restored.
[0010] According to a second aspect of the exemplary embodiments of the present disclosure, there is provided a handwritten content recognition device, the device comprising:
[0011] A data acquisition module, for acquiring hand movement data of a user in the process of writing the text to be recognized when detecting that the user starts writing;
[0012] A data processing module, used for extracting features to be identified from the hand motion data, and matching content parameter identification results of the text to be identified based on the features to be identified;
[0013] A data processing module is used to restore the written content of the text to be recognized based on the content parameter recognition result and the hand movement data.
[0014] According to a third aspect of the exemplary embodiments of the present disclosure, there is provided an electronic device, including:
[0015] at least one processor;
[0016] A memory for storing at least one processor-executable instruction; wherein the at least one processor is configured to execute the instruction to implement the method described in the exemplary embodiments of the present disclosure.
[0017] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method described in the exemplary embodiments of the present disclosure is implemented.
[0018] At least one of the above technical solutions adopted by the exemplary embodiments of the present disclosure can achieve the following beneficial effects: when it is detected that the user starts writing, the hand motion data of the user writing the text to be recognized is obtained; the features to be recognized of the hand motion data are extracted, and the content parameter recognition results of the text to be recognized are matched based on the features to be recognized; based on the content parameter recognition results and the hand motion data, the written content of the text to be recognized is restored. Among them, by extracting and matching the hand motion features to determine the content parameters of the text to be recognized, and restoring the actual written content of the text to be recognized based on the content parameter recognition results and the hand motion data, it can not only reduce the dependence on specific writing hardware equipment and lighting conditions, but also improve the accuracy of the handwriting content recognition method, so that users can smoothly perform handwriting input and recognition in various environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the exemplary embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0020] Figure 1 A flowchart of a handwritten content recognition method provided by an exemplary embodiment of the present disclosure;
[0021] Figure 2A flowchart of a training method for a gesture muscle movement model provided as an example in the present disclosure;
[0022] Figure 3 A flowchart of a handwritten content recognition method provided as an example in the present disclosure;
[0023] Figure 4 A schematic block diagram of functional modules of a handwriting content recognition device provided as an example in the present disclosure;
[0024] Figure 5 A schematic block diagram of a chip provided as an example in the present disclosure;
[0025] Figure 6 The present invention is an exemplary block diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0028] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0031] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0032] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0033] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0034] In recent years, with the rapid development of human-computer interaction, handwriting recognition technology has made significant progress. This technology can convert the user's handwriting into digital text for easy storage or display, greatly facilitating the user's application in learning, work and life.
[0035] In the related technologies, touch screen input technology, digital handwriting tablet capture or image recognition are usually used to identify the user's handwritten content and convert it into a data format that can be understood and parsed by the computer. Touch screen devices track the movement of fingertips or stylus pens through built-in sensors; digital handwriting tablets use electromagnetic induction or pressure sensing technology to record writing movements; image acquisition solutions use cameras and other devices to capture image information of handwritten content on paper, and then apply image processing and pattern recognition algorithms to convert it into digital text.
[0036] However, this handwriting content recognition method relies on specific writing equipment, and image recognition has high requirements for lighting conditions. In the absence of relevant writing equipment or poor lighting conditions, real-time recognition of user writing content cannot be achieved.
[0037] Therefore, in order to solve the above problems, the exemplary embodiments of the present disclosure provide a method for recognizing handwritten content, which obtains the motion perception data of the user when writing through the smart wearable device worn by the user and establishes a three-dimensional coordinate system, then extracts the user's gesture features through the gesture muscle motion model, and identifies the content parameters corresponding to the user's gesture features to obtain the user's writing content parameters. Then, the hand movements of the user when writing are obtained through the smart wearable device, and the strokes corresponding to the user's gesture movements are identified to obtain the user's writing content in real time.
[0038] The user wears a smart wearable device such as a ring or bracelet, and taps the plane or object on which he is going to write with his finger or pen tip to start writing. The user's writing content is recognized in real time through the smart wearable device and the gesture muscle movement model. After the recognition is completed, the recognition result is sent to the target terminal, such as the PPT (PowerPoint, presentation software) being explained. The user can see the real-time presentation of the writing content in the blank space on the screen.
[0039] The entire process does not require additional input devices and can be completed by simply wearing a smart wearable device. This convenient handwritten content recognition method is suitable for various scenarios and can significantly improve user experience and work efficiency.
[0040] For example, Figure 1 This is a flowchart of a handwritten content recognition method provided by an exemplary embodiment of the present disclosure, such as Figure 1 As shown, the following steps may be specifically included:
[0041] Step S110: Acquire motion sensing data and establish a three-dimensional coordinate system.
[0042] Exemplarily, the initialization process of the smart wearable device (such as a wristband and a finger ring) may include: after startup, it may communicate with the server through wireless technologies such as Bluetooth, and complete the system matching between the smart wearable device and the server. Multiple sensors are built into the smart device, including gyroscopes, accelerometers, electromyography sensors, and magnetometers, which together provide motion perception data when the user performs hand movements.
[0043] Exemplarily, the motion sensing data may include gyroscope data, electromyography sensor data, and magnetometer data.
[0044] The gyroscope data may include: a starting direction, an ending direction, a motion trajectory, a starting attitude angle, and an ending attitude angle.
[0045] The electromyographic sensor data may include: starting action potential, ending action potential, signal amplitude, signal integration, signal density, starting muscle activation time point, ending muscle activation time point, and dynamic electromyogram.
[0046] The magnetometer data may include: a starting magnetic field strength, an ending magnetic field strength, a starting inclination angle, and an ending inclination angle.
[0047] After the smart wearable device is initialized, in order to accurately track and obtain the user's gestures, a three-dimensional coordinate system can be constructed based on any smart wearable device. Among them, since the gyroscope can provide stable data of the current direction of the smart wearable device, the X-axis can be established based on the gyroscope as the origin and the horizontal plane of the gyroscope, the Y-axis can be established in the direction perpendicular to the X-axis, and the Z-axis can be established in the direction perpendicular to both the X-axis and the Y-axis.
[0048] Among them, the X-axis and Y-axis are mainly used to accurately locate and compare the written content, while the Z-axis data is used to monitor the position changes during the writing process in real time, thereby assisting in determining the writing position.
[0049] Based on this, this coordinate system layout not only helps to accurately locate the user's writing movements, but also monitors the vertical changes of the writing position through the Z axis, increasing the ability to capture complex gestures. In addition, the construction of this three-dimensional coordinate system not only enables the system to effectively recognize and analyze handwritten content in three-dimensional space, but also breaks through the traditional two-dimensional limitations, greatly improving the applicability and flexibility of handwritten content recognition methods.
[0050] Step S120: constructing a gesture muscle motion model based on the motion perception data and the distance relationship matrix.
[0051] Exemplarily, a gesture muscle motion model of the target user may be constructed based on the motion perception data of the target user and the distance relationship matrix. The gesture muscle motion model is used to identify content parameters of the actual writing content of the target user.
[0052] Exemplarily, a method of obtaining a distance relationship matrix may include:
[0053] The target user taps different planes or objects with his fingers or a pen. The 2-4 microphones in the smart wearable device capture the sound intensity of the tapping, determine the three-dimensional coordinate data of the tapping point, and calculate the distance from each tapping point to the origin of the three-dimensional coordinate system to generate a preliminary distance relationship matrix.
[0054] In the subsequent use process, more distance relationship data can be collected according to user-specific behaviors, such as time, user ID and other fields, to further improve and adjust the distance relationship matrix, so that the constructed gesture muscle movement model can better fit the movement characteristics of each user.
[0055] Exemplarily, the distance relationship matrix content format includes, but is not limited to: tap start time, tap end time, tap point distance information, and user identification.
[0056] After obtaining the distance relationship matrix, a gesture muscle movement model can be constructed based on the motion perception data, the distance relationship matrix, and the content parameters. The content parameters are used to represent the types of content written by the user, and the content parameters include, but are not limited to, Chinese strokes and pinyin, English subtitles, Arabic numerals, special characters, other graphic gestures, etc.
[0057] A gesture muscle movement model with strong applicability can be constructed according to the corresponding relationship between the content parameters and the gesture muscle data when different users write different content parameters. Although the writing structures of Chinese strokes, Chinese pinyin, English letters, Arabic numerals, special characters, and other graphics are simple and the types are limited, rich and diverse gesture muscle data can be collected by covering a wide range of usage scenarios and user groups (different genders, ages, etc.).
[0058] Exemplarily, Figure 2 The following is a flowchart of the training method of the gesture muscle movement model provided by an exemplary embodiment of the present disclosure, as Figure 2 shown, and specifically may include the following steps:
[0059] Step S210: Collect training data.
[0060] Exemplarily, gesture muscle data for training can be collected according to each component of each content parameter. For example, when the content parameter is an Arabic numeral, its components include: 0, 1, 2, 3... 9. The gesture muscle data for each component may include:
[0061] Distance data: The distance data can be obtained from the distance relationship matrix. The distance data includes the distance data between each point when writing a specific content (such as writing a "丿" or "2").
[0062] Each component of the content parameter: Specific elements in gesture recognition, such as: Chinese strokes (such as "丿"), English letters (such as "A"), Arabic numerals (such as "2"), special characters, or graphic gestures. The definition of the content parameter is the specific target that the model needs to recognize and learn, and they are associated with the distance data to form a unique identifier for each gesture.
[0063] Motion perception data: When performing specific gestures (such as writing a slash or the number 2), the usage data of motion perception modules such as gyroscopes, electromyogram sensors, and magnetometers obtained from smart wearable devices. For example: the rotation angle of the wrist (collected by the gyroscope sensor), the activity of finger muscles (collected by the electromyogram sensor), the movement information of the arm (collected by the accelerometer and position sensor), and magnetometer data (providing magnetic field changes during gesture execution) and the position data of the above data in three-dimensional coordinates. Motion perception data provides dynamic information during gesture execution, such as motion speed, direction, muscle activity intensity, and magnetic field changes. Motion perception data is crucial for understanding and predicting gesture actions.
[0064] Position data in three-dimensional coordinates: The specific coordinate data of each component of the content parameter calculated by combining the data of the smart wearable device in three-dimensional space.
[0065] Based on this, by using multi-dimensional training data to train the gesture muscle movement model, the trained gesture muscle movement model can accurately identify and analyze complex gesture actions. In addition, incorporating the gesture muscle data of different user groups into the training data can enhance the generalization ability and application scope of the model.
[0066] Step S220: Extract features from the training data.
[0067] Exemplarily, a convolutional neural network or a sequence data processing network can be used to extract features from the input training data to obtain gesture feature information, thereby identifying different gestures. The gesture feature information can include parameters such as speed, acceleration, direction change, force, and muscle activity intensity.
[0068] Among them, speed and acceleration can reflect the speed and dynamic changes of gesture execution. Direction change can reflect the direction adjustment of the gesture in space. Force and muscle activity intensity can be obtained through electromyogram sensor data and reflect the force differences of different gestures.
[0069] Step S230: Model establishment and output.
[0070] Exemplarily, the three-dimensional coordinate data of each component of the gesture muscle movement model is established and output. To simplify data processing, these three-dimensional coordinate data may be simplified and the result representation of each component in the system is output.
[0071] The model output results can include: content parameters, orientation information, starting coordinates, ending coordinates, intermediate point coordinates, content recognition results, and content parameter recognition result types.
[0072] Finally, the correspondence between the content parameters and the model output results is established to obtain a correspondence data table to ensure that the model can accurately reflect the user's gesture actions.
[0073] Step S240: Model optimization.
[0074] For example, in order to optimize the gesture muscle movement model, a machine learning method can be used to train and generalize the obtained model data, so that the model can be compatible with the usage habits of more users and improve its adaptability and accuracy.
[0075] Specifically, by using the large amount of data collected above for training, the model will be able to learn the gesture characteristics of different users in various usage scenarios. During the training process, cross-validation and real-time feedback can be used to adjust parameters to ensure that the model can adapt to various complex usage environments and individual differences in actual applications, thereby significantly improving the practicality and user satisfaction of gesture recognition technology.
[0076] Step S130: distance confirmation and three-dimensional coordinate calculation.
[0077] In actual applications, when users use the gesture muscle motion model, they need to confirm the distance relationship matrix again based on the data of the motion perception module to obtain distance confirmation data. By using the 2-4 microphone devices of the smart wearable device and the user's tapping action, the distance, direction and coordinates of the tapping position of the user's fingertip or pen tip and the smart wearable device in three-dimensional space are determined.
[0078] Specifically, the user taps an object or plane in the real world with the tip of a finger, a pen tip, or a teaching tool, and 2-4 microphones on the smart wearable device receive the sound produced by the tapping. Since the microphones are located in different positions, the time they receive the sound will be different. Based on the time difference of these sound signals, the distance, direction, and three-dimensional coordinates between the tapping point and the smart wearable device can be calculated through a triangulation positioning algorithm.
[0079] Step S140: performing content matching through the gesture muscle movement model and outputting content parameter recognition results.
[0080] In actual applications, the distance confirmation data when the target user uses the system, the content parameter data written by the user, and the data in the gesture muscle movement model library are matched to find the most matching content in the model library as the content actually written by the user.
[0081] Specifically, the position and size of the user's writing content are determined based on the writing orientation and three-dimensional coordinate data of the distance confirmation data. This step is to ensure that the system can accurately locate the user's writing area and perform appropriate scaling and adjustment according to the spatial range of the writing.
[0082] Furthermore, prediction is performed using the distance confirmation data, the content parameter data of the user's writing, and the data in the gesture muscle movement model library to find the content that best matches the user's actual writing as the prediction result.
[0083] For example, when recognizing special written content, a preset recognition strategy can be used: for single-stroke writing, the single-stroke writing result is first recognized. The single-stroke writing result can be directly compared with the gesture operation model for recognition. If accurate recognition of a single content parameter cannot be achieved, the connected writing recognition mode is switched to.
[0084] For cursive writing, fuzzy matching of multiple content parameters is performed on the enlarged or reduced writing results to ensure the accuracy of the recognition results.
[0085] The content parameter recognition results include data of each Chinese stroke, pinyin, English letter, number, character and graphic, and display these recognition results in sequence according to distance and position. At the same time, the attributes of the recognition results are marked according to the standards defined by the content parameters. For example, the attributes can be defined as Chinese strokes and pinyin, English letters, Arabic numbers, special characters or graphics.
[0086] Step S150: Verify the content parameter identification result to obtain the content parameter verification result.
[0087] For example, the three-dimensional coordinate data and distance confirmation data obtained in real time during the user's writing can be matched and verified with the three-dimensional coordinate and distance data in the gesture muscle movement model to ensure accurate restoration of the user's writing results and solve the problem of repeated data coverage.
[0088] Exemplarily, the three-dimensional coordinate data of the content parameters during the user's writing process is first obtained through coordinate mapping. Coordinate mapping refers to obtaining and recording the three-dimensional coordinate data of the user's finger, pen tip or the top of the teaching tool in real time during the user's writing process. The three-dimensional coordinate data represents the user's writing trajectory and position in three-dimensional space. The three-dimensional coordinate data is recorded in real time to form a series of time series data, which can be used to represent the user's writing trajectory.
[0089] The gesture muscle movement model can use the three-dimensional coordinate data to confirm the position and size of the user's writing content and reproduce the user's writing trajectory to ensure the accuracy and consistency of the writing content.
[0090] Furthermore, coordinate verification is used to check and verify the three-dimensional coordinate data acquired in real time to ensure that the coordinate data is consistent with the previously recorded data and to handle possible duplicate coverage.
[0091] The coordinate verification process may include the following situations:
[0092] 1. The new and old coordinate data are completely consistent. Compare the newly acquired coordinate data with the previously recorded coordinate data. If the new and old coordinate data are consistent, it means that the user is using the previously used three-dimensional coordinates during the writing process. At this time, the existing coordinate content parameter results are used as the content parameter verification results.
[0093] 2. The new and old coordinate data are partially consistent. Assume that the user has written something (such as a letter or a stroke) at a certain location, and the 3D coordinate data of this location has been recorded. Now, the user writes at this location again, resulting in the new and old coordinate data being partially consistent. The specific operation steps may include:
[0094] Get the original coordinate data: The system has recorded the three-dimensional coordinate data that the user previously wrote at a certain location.
[0095] Obtain new coordinate data: When the user writes at the same location again, the system obtains new three-dimensional coordinate data.
[0096] Compare new and old data: The system compares the newly acquired coordinate data with the original coordinate data recorded previously.
[0097] Partial use of original coordinates: If the user only partially writes at the original location (for example, adding a stroke to an already written letter), the system needs to merge the new writing content with the original content.
[0098] Replace the original data position: The system compares and merges the newly written data with the original data. If the newly written content covers part of the original content, the system will replace the original data position with the position covered by the new data.
[0099] For example, if the user adds a stroke to the original letter "C" and it becomes the letter "G", the system will replace the corresponding part of the original data (the coordinate data of "C") with the new data (the coordinate data of "G").
[0100] Update content parameter results: The system uses the fused data as the new content parameter verification result. For example: suppose the user writes the letter "C" at a certain position for the first time, and the system records the three-dimensional coordinate data of "C". Then, the user adds another stroke at the same position, so that "C" becomes "G". At this time, the original data is the coordinate data of "C" recorded by the system, and the new data is the coordinate data of "G" after the user adds a stroke. When the system compares the two data, it is found that the new data covers part of the original data. Therefore, the system replaces the corresponding part of the coordinate data of "C" with the coordinate data of "G", updates the entire coordinate data, and finally saves it as the "G" written by the user.
[0101] 3. Repeated overwriting of original coordinate data or inconsistent content parameter attributes. During the writing process, sometimes the user writes again at a coordinate location that has been used, or the new writing content is different from the previous content parameter attributes. Specific operation steps may include:
[0102] Detect whether coordinate data is repeatedly overwritten: detect that the user writes again at a 3D coordinate position that has been used.
[0103] Check whether the attributes are inconsistent: detect that the new writing content attributes (such as character type, stroke attributes, etc.) are inconsistent with the previously recorded content attributes.
[0104] The processing method includes: saving the data that reuses the original coordinates as a complete result record, that is, the system will not simply overwrite the old data, but record the new written content together with the old content as part of the content parameter verification result.
[0105] For example, the user writes the number "1" at a certain location for the first time, and the system records the three-dimensional coordinate data of "1". Then, the user writes the letter "I" at the same location. The user writes "I" at the location where "1" was written, causing the new writing track to repeatedly cover the original coordinate data, and the attributes of the number "1" are inconsistent with the attributes of the letter "I". At this time, the new writing content (letter "I") needs to be recorded together with the previous writing content (number "1") as a complete final output result.
[0106] 4. Consider the deformation effect of the Z coordinate axis on the content parameter output results. In three-dimensional space, the written content not only involves the position changes of the X-axis and Y-axis, but also the changes of the Z-axis (depth axis). During the writing process, the user's writing tool (such as the pen tip or finger) may move up and down on the Z-axis to a certain extent. This movement will affect the shape and position of the written content. For example, the change in the height of the pen tip when lifting and pressing when writing will result in different strokes. The thickness and connection method. Therefore, the writing content of different depths will affect the final recognition results. In the content parameter verification, it is necessary to consider the impact of the Z coordinate axis data on the writing content.
[0107] Specific steps may include:
[0108] Height difference detection: Detect the height change of the Z axis during writing. If the height change exceeds a certain threshold, the writing content needs to be adjusted.
[0109] Smoothing: Smooth the writing trajectory to eliminate discontinuity caused by Z-axis changes.
[0110] For example, when the user writes the letter "L", the tip of the pen has an obvious up and down movement on the Z-axis, resulting in the starting point being higher than the ending point. The vertical stroke part of the "L" does not look like a straight line but has a downward tilt. After detecting this change in the Z-axis, the tilted stroke can be corrected to a straight line by adjusting the Z-axis coordinates, so that the written content is accurately restored to the "L" that the user intends to write.
[0111] Finally, different content parameters can be distinguished and identified through content parameter verification to avoid confusion. The specific operation steps may include:
[0112] When the user is writing, the attributes of the written content are recognized in real time. For example, it is recognized whether the current writing is a letter, a number or a symbol.
[0113] Through three-dimensional coordinate data, the specific position and trajectory of the written content are determined, and these data are compared with the existing models.
[0114] Verification rules are set for easily confused characters. For example, the Chinese stroke "vertical" (stroke I) and the number "1" are very similar in shape, but they can be distinguished by the content parameters of the context, writing habits and subtle shape differences.
[0115] For example, when the user writes a "1" or an "I" during the writing process, it is first recognized that the user writes a vertical line (which may be the number "1" or the letter "I"). Then, through three-dimensional coordinate data, the specific position and angle of the written vertical line are determined. If the vertical line appears in a number string (such as "123"), it can be more inclined to be recognized as the number "1"; if the vertical line appears in a letter string (such as "IT"), the system can be more inclined to be recognized as the letter "I".
[0116] In an optional way, further checks can also be performed based on the verification rules. For example: if there are obvious starting and ending traces in the upper and lower parts of the vertical line, it may be closer to the Chinese stroke "vertical"; if the height and line thickness of the vertical line conform to the writing habit of the number "1", it may be more inclined to be recognized as the number "1".
[0117] Based on this, through coordinate mapping, coordinate verification and content parameter verification, the user's written content can be effectively matched and verified, solving the situation of partial duplicate coverage data and improving the accuracy of the content parameter verification result.
[0118] Step S160: Determine the target output result and send the target output result to the target terminal.
[0119] Exemplarily, the content parameter recognition result output by the gesture muscle movement model is compared with the content parameter verification result obtained in step S150. When the content parameter recognition result is consistent with the content parameter verification result, either result is used as the target output result. When the content parameter recognition result is inconsistent with the content parameter verification result, the content parameter verification result is used as the target output result.
[0120] The target terminal may be a terminal that is connected to the smart wearable device for communication in a specific scenario, such as a computer screen during a meeting or a mobile phone screen during a call.
[0121] In addition, the target output results can be stored in a certain format, or used for digital human text-to-speech interactive presentation.
[0122] One or more technical solutions provided in the exemplary embodiments of the present disclosure establish a gesture muscle movement model based on the known distance and each content of the user's handwriting input by a smart wearable device such as a bracelet or finger ring. Then, through tapping and the sound receiving function of the smart wearable device, the azimuth distance relationship matrix between the smart wearable device and the tapping position is obtained and established. Using the gesture muscle movement model and the azimuth distance relationship matrix, each content written by the user is restored. Finally, through coordinate mapping, coordinate verification and content parameter verification, the user's written content can be effectively matched and verified, the situation of partial repeated coverage of data can be solved, and the accuracy of the content parameter verification results can be improved.
[0123] Therefore, the handwriting content recognition method provided in the exemplary embodiments of the present disclosure can generate user gesture writing content in real time without relying on specific writing hardware devices and any lighting conditions.
[0124] Based on the above embodiments, the present disclosure also provides a handwritten content recognition method. Figure 3 This is a flow chart of a handwritten content recognition method provided by an exemplary embodiment of the present disclosure, such as Figure 3 As shown, the method may include the following steps:
[0125] Step S310: when it is detected that the user starts writing, obtaining the hand movement data of the user when writing the text to be recognized.
[0126] Exemplarily, the hand motion data refers to the displacement data and muscle motion data of the user's fingers and wrists during writing. The hand motion data in this embodiment is equivalent to the motion perception data in the previous embodiment.
[0127] Motion sensing data can be obtained through smart wearable devices (such as wristbands and finger rings) worn by users. Motion sensing data may include gyroscope data, electromyography sensor data, and magnetometer data.
[0128] The gyroscope data may include: a starting direction, an ending direction, a motion trajectory, a starting attitude angle, and an ending attitude angle.
[0129] The electromyographic sensor data may include: starting action potential, ending action potential, signal amplitude, signal integration, signal density, starting muscle activation time point, ending muscle activation time point, and dynamic electromyogram.
[0130] The magnetometer data may include: a starting magnetic field strength, an ending magnetic field strength, a starting inclination angle, and an ending inclination angle.
[0131] Step S320: extracting features to be recognized from the hand motion data, and matching content parameter recognition results of the text to be recognized based on the features to be recognized.
[0132] In an embodiment, a convolutional neural network or a sequence data processing network may be used to extract features from hand motion data to obtain features to be identified, thereby identifying content parameter recognition results of the text to be identified. Features to be identified refer to feature information extracted from hand motion data, which are used for subsequent matching and identification of text content. Features to be identified may include parameters such as speed, acceleration, direction change, strength, and muscle activity intensity.
[0133] Among them, speed and acceleration can reflect the speed and dynamic changes of hand movement when writing. Direction changes can reflect the direction adjustment of the hand in space. Strength and muscle activity intensity can be obtained through electromyography sensor data to reflect the difference in strength of hand movement when writing different contents.
[0134] Exemplarily, the content parameter recognition result is used to indicate the type of content written by the user, and the content parameter recognition result includes but is not limited to Chinese strokes and pinyin, English subtitles, Arabic numerals, special characters, other graphic gestures, etc.
[0135] Step S330: based on the content parameter recognition result and the hand movement data, restore the written content of the text to be recognized.
[0136] In the embodiment, the specific writing trajectory of the written content is determined by hand motion data, and the specific writing content of the text to be recognized is determined in combination with the content parameter recognition result.
[0137] Among them, the content parameter recognition result refers to the type of written content extracted by analyzing the hand movement data, such as Chinese strokes and pinyin, English subtitles, Arabic numerals, special characters, etc., which indicates the approximate classification or characteristics of the user's written content.
[0138] The written content of the text to be recognized is the text itself specifically restored based on the content parameter recognition results and the hand movement data, that is, the specific shape and form of the words or symbols actually written by the user.
[0139] For example, the Chinese stroke "vertical" (stroke I) and the number "1" are very similar in shape, but can be distinguished by the content parameter recognition result and the hand movement data. When the user writes a "1" or an "I" during the writing process, first, the hand movement data is used to recognize that the user has written a vertical line (which may be the number "1" or the letter "I").
[0140] Then, by extracting the features to be recognized from the hand movement data, the content parameter recognition result is determined. For example, if there are obvious starting and ending traces at the upper and lower parts of the vertical line, it may be closer to the Chinese stroke "vertical"; if the height and line thickness of the vertical line conform to the writing habit of the number "1", it may be more inclined to be recognized as the number "1". Thus, the written content of the text to be recognized is restored.
[0141] Based on this, by extracting and matching the hand movement features to determine the content parameters of the text to be recognized, and determining the actual written content of the text to be recognized according to the content parameter recognition result and the hand movement data, not only can the dependence on specific writing hardware devices and lighting conditions be reduced, but also the accuracy of the handwritten content recognition method can be improved, enabling the user to smoothly perform handwritten input and recognition in various environments.
[0142] Based on the above embodiments, in another embodiment provided by the present disclosure, the above step S320 may specifically include:
[0143] Obtain a pre-constructed gesture muscle movement model;
[0144] Input the hand movement data into the gesture muscle movement model to extract the features to be recognized from the hand movement data;
[0145] Based on the features to be recognized, determine the content parameter recognition result corresponding to the text to be recognized in the content parameter feature table; the content parameter feature table includes content parameters and the sample features corresponding to the content parameters.
[0146] In the embodiment, the hand movement data in this embodiment is equivalent to the motion perception data in the previous embodiment.
[0147] Exemplarily, a gesture muscle movement model of the target user can be constructed based on the motion perception data and the distance relationship matrix of the target user. The gesture muscle movement model is a model constructed based on the motion perception data and the distance relationship matrix of the target user, and is used to recognize and extract the features of the hand movement data during the user's writing process. By analyzing inputs such as the displacement data and muscle movement data of the hand, features to be recognized such as speed, acceleration, direction change, force, and muscle activity intensity are extracted, so as to determine the content parameter recognition result corresponding to the text to be recognized in the content parameter feature table, and finally restore the actual written content of the user.
[0148] Exemplarily, a method of obtaining a distance relationship matrix may include:
[0149] The user taps different planes or objects with his fingers or a pen. The 2-4 microphones in the smart wearable device capture the sound intensity of the tapping, determine the three-dimensional coordinate data of the tapping point, calculate the distance from each tapping point to the origin of the three-dimensional coordinate system, and generate a preliminary distance relationship matrix.
[0150] In the subsequent use process, more distance relationship data can be collected according to user-specific behaviors, such as time, user ID and other fields, to further improve and adjust the distance relationship matrix, so that the constructed gesture muscle movement model can better fit the movement characteristics of each user.
[0151] Exemplarily, the distance relationship matrix content format includes, but is not limited to: tap start time, tap end time, tap point distance information, and user identification.
[0152] After obtaining the distance relationship matrix, a gesture muscle movement model can be constructed based on the motion perception data, the distance relationship matrix and the content parameters. The content parameters are used to indicate the type of content written by the user, including but not limited to Chinese strokes and pinyin, English subtitles, Arabic numerals, special characters, other graphic gestures, etc.
[0153] A highly applicable gesture muscle movement model can be constructed based on the correspondence between content parameters and gesture muscle data when different users write different content parameters. Although the writing structure of Chinese strokes, Chinese pinyin, English letters, Arabic numerals, special characters and other graphics is simple and limited in variety, rich and diverse gesture muscle data can be collected by covering a wide range of usage scenarios and user groups (different genders, ages, etc.).
[0154] For example, the training process of the gesture muscle movement model can be found in the above text. Figure 2 Related content.
[0155] The gesture muscle movement model is built based on a large amount of hand movement data, and can accurately simulate and understand the user's gesture and muscle movement characteristics during the writing process.
[0156] For example, when it is detected that the user starts writing, the user's hand motion data is acquired in real time, and the hand motion data includes information such as the motion trajectory and speed of the user's fingers, wrists, etc. Subsequently, the hand motion data is input into the gesture muscle motion model, and the model processes the data and extracts features to be recognized, such as the starting point, end point, direction, speed, pressure and other feature information of the stroke.
[0157] Then, based on the extracted features to be recognized, a match is performed in the content parameter feature table. The content parameter feature table contains multiple content parameters, each of which corresponds to multiple different sample features. By comparing and matching with these sample features, the system can determine the content parameter recognition result corresponding to the user's text to be recognized.
[0158] Exemplarily, the content parameter feature table may be a database table, which contains multiple content parameters and their corresponding multiple sample features, and is used to extract the features to be identified from the hand motion data for matching and identification, and is the basis for judgment in the gesture muscle motion model recognition process.
[0159] Content parameters refer to the specific type or classification of the content written by the user, such as Chinese strokes and pinyin, English letters, Arabic numerals, special characters or graphics, etc. Content parameters are used to classify the content written by the user.
[0160] Sample features are specific feature information associated with content parameters, including stroke starting point, end point, direction, speed and pressure, etc. These features are used to compare and match with the extracted features to be identified in the gesture muscle movement model to determine the final content parameter recognition result.
[0161] The content parameter recognition results include data of each Chinese stroke, pinyin, English letter, number, character and graphic, and display these recognition results in sequence according to distance and position. At the same time, the attributes of the recognition results are marked according to the standards defined by the content parameters. For example, the attributes can be defined as Chinese strokes and pinyin, English letters, Arabic numbers, special characters or graphics.
[0162] Based on this, the gesture muscle movement model is trained by using multi-dimensional training data, so that the trained gesture muscle movement model can accurately recognize and analyze complex gesture movements. In addition, incorporating gesture muscle data of different user groups into the training data can enhance the generalization ability and application scope of the model.
[0163] Based on the above embodiment, in another embodiment provided by the present disclosure, the above determination of the content parameter recognition result corresponding to the text to be recognized in the content parameter feature table based on the feature to be recognized may specifically include:
[0164] Traverse the sample features corresponding to the content parameters in the content parameter feature table;
[0165] Obtain the similarity between the feature to be identified and the sample feature;
[0166] When the similarity is greater than a preset threshold, the content parameter associated with the corresponding sample feature in the content parameter feature table is obtained, and the content parameter is determined as a content parameter recognition result corresponding to the text to be recognized.
[0167] In the embodiment, the content parameter feature table includes a plurality of content parameters, and each content parameter corresponds to a plurality of different sample features.
[0168] First, multiple content parameters in the content parameter feature table are traversed, each of which corresponds to multiple sample features. By traversing, sample features corresponding to each content parameter are obtained one by one. Sample features can be collected and stored in advance and associated with corresponding content parameters.
[0169] Then, the similarity between the feature to be identified and each sample feature is calculated respectively. Similarity is an indicator used to quantify the degree of match between the feature to be identified and the sample feature. By calculating the similarity, the degree of similarity between two features in geometric or vector space can be evaluated. If the similarity is greater than a preset threshold, the two features are considered to have a high similarity, which helps determine the content parameter recognition result of the text to be identified.
[0170] The similarity calculation method may include calculating the Euclidean distance or the cosine similarity, etc. By calculating the similarity, the matching degree between the feature to be identified and the sample feature can be quantified.
[0171] After the similarity calculation is completed, it is determined whether each similarity is greater than a preset threshold. If the similarity exceeds the preset threshold, it can be considered that the feature to be identified has a high similarity with the sample feature. At this time, the content parameter associated with the sample feature in the content parameter feature table is obtained, and the content parameter is determined as the content parameter recognition result corresponding to the text to be identified.
[0172] Based on this, by calculating the similarity between the features to be identified and the sample features in the content parameter feature table, the content parameters of the text to be identified can be accurately matched and identified.
[0173] Based on the above embodiment, in another embodiment provided by the present disclosure, the above determining the content parameter recognition result corresponding to the text to be recognized in the content parameter feature table based on the feature to be recognized may also include:
[0174] Respectively obtain the similarity between the feature to be identified and the sample feature corresponding to the single-stroke writing content parameter;
[0175] When the similarity is less than a preset threshold, the similarity between the feature to be identified and the sample features corresponding to the cursive writing content parameters is obtained respectively.
[0176] In the embodiment, the content parameter recognition process may be further refined to improve recognition accuracy and adaptability.
[0177] First, the similarity between the feature to be recognized and the sample feature corresponding to the single-stroke writing content parameter is obtained respectively. When the similarity is greater than a preset threshold, the sample feature corresponding to the single-stroke writing content parameter can be directly determined as the content parameter recognition result of the text to be recognized.
[0178] However, if the similarity is less than a preset threshold, it is considered that the text to be recognized is not a single-stroke writing content, and then the mode of recognition of connected writing is switched to.
[0179] In the cursive writing recognition mode, the similarity between the feature to be recognized and the sample feature corresponding to the cursive writing content parameters is obtained. In order to ensure the accuracy of recognition, the writing results can be enlarged or reduced to adapt to the writing style and speed of different users, thereby enhancing the system's adaptability to various cursive writings. Even if there are cursive, broken or other complex situations in the writing process, the user's writing content can still be accurately recognized.
[0180] Based on this, by first performing single-stroke recognition and then continuous writing recognition, high-precision content recognition can be provided in various writing modes. It also enhances the system's adaptability to the writing habits of different users, ensuring that users can get a smooth handwriting content recognition experience in various environments.
[0181] Based on the above embodiment, in another embodiment provided by the present disclosure, the above handwritten content recognition method may further include:
[0182] Constructing a three-dimensional coordinate system; wherein, the gyroscope in the smart wearable device worn by the user is used as the origin of the three-dimensional coordinate system, the X-axis of the three-dimensional coordinate system is constructed based on the horizontal plane of the gyroscope, the Y-axis is established in a direction perpendicular to the X-axis, and the Z-axis is established in a direction perpendicular to both the X-axis and the Y-axis;
[0183] Based on the three-dimensional coordinate system, the triangulation positioning method is used to determine the relative position information between the writing position of the text to be recognized and the smart wearable device.
[0184] In the embodiment, in order to accurately track and obtain the user's gesture, a three-dimensional coordinate system can be constructed according to any smart wearable device worn by the user. Among them, since the gyroscope can provide stable data of the current direction of the smart wearable device, the X-axis can be established based on the gyroscope as the origin and the horizontal plane of the gyroscope, the Y-axis can be established in a direction perpendicular to the X-axis, and the Z-axis can be established in a direction perpendicular to both the X-axis and the Y-axis.
[0185] Among them, the X-axis and Y-axis are mainly used to accurately locate and compare the written content, while the Z-axis data is used to monitor the position changes during the writing process in real time, thereby assisting in determining the writing position.
[0186] Exemplarily, in the handwritten content recognition method, the relative position information between the writing position of the to-be-recognized text and the smart wearable device can be determined using a triangulation positioning method based on a three-dimensional coordinate system, which may specifically include:
[0187] First, during the writing process, the coordinate information of multiple key points (such as the starting point, end point and turning point of the stroke) can be recorded. Then, the distance between these key points and the device is measured through the sensor on the smart wearable device. These distance data combined with the three-dimensional coordinates of the smart wearable device can be used to calculate the specific location of the writing point. Using the known three-dimensional coordinates and distance data, a triangulation positioning method (such as triangulation) is applied to calculate the specific coordinates of each writing point, thereby determining the position of each stroke written by the user.
[0188] Specifically, before the user starts writing, the three-dimensional coordinate system is initialized, and its initial position and posture are determined by the sensor of the smart wearable device. During the user's writing process, the position information of the smart wearable device and the distance data from the writing point are collected in real time. The specific three-dimensional coordinates of each writing point are calculated through a triangulation positioning algorithm (for example, using multiple known points and distances for triangulation calculations). Based on the calculated writing point coordinates and the coordinates of the smart wearable device, the system determines the relative position information between the writing position and the device, which can be used for further handwriting content recognition and processing.
[0189] Based on this, the three-dimensional coordinate system not only helps to accurately locate the user's writing movements, but also monitors the vertical changes of the writing position through the Z axis, increasing the ability to capture complex gestures. In addition, the construction of this three-dimensional coordinate system not only enables the system to effectively identify and parse handwritten content in three-dimensional space, but also breaks through the traditional two-dimensional limitations, greatly improving the applicability and flexibility of handwritten content recognition methods. Secondly, through the triangulation positioning method combined with the three-dimensional coordinate system, the writing position can be determined, the accuracy of handwritten content recognition can be improved, and the system can be ensured to obtain and calculate the writing position in real time to support subsequent real-time content recognition, thereby realizing real-time handwritten content recognition without relying on specific writing surfaces or lighting conditions.
[0190] Based on the above embodiment, in another embodiment provided by the present disclosure, the above-mentioned restoration of the written content of the text to be recognized based on the content parameter recognition result and the hand movement data may specifically include:
[0191] Determine the moving distance and moving direction of the user when writing the text to be recognized based on the hand movement data and the relative position information;
[0192] Based on the recognition results of the moving distance, moving direction and content parameters, an initial content restoration result of the text to be recognized is obtained;
[0193] The preset verification strategy is used to verify and update the initial content restoration result to obtain the written content of the text to be recognized.
[0194] In the embodiment, based on the hand motion data and relative position information, the moving distance and moving direction of the user in the process of writing the text to be recognized can be confirmed. First, the user's hand motion data and relative position information are obtained in real time through the sensor on the smart wearable device, which may include the movement trajectory and speed of the user's fingers, wrists and other parts. Combined with the three-dimensional coordinate system and triangulation positioning method, the system can accurately calculate the moving distance and moving direction of each stroke of the user during the writing process.
[0195] Then, based on the moving distance, moving direction and content parameter recognition results, the initial content restoration results of the text to be recognized are obtained. Among them, the moving distance and direction provide geometric information of the handwritten strokes, and the content parameter recognition results come from the matching of the gesture muscle movement model and the content parameter feature table. Under the combined effect of the above data, the text content written by the user can be preliminarily recognized.
[0196] Subsequently, the initial content restoration result is verified and updated using a preset verification strategy to obtain the final written content. The preset verification strategy may include:
[0197] Verification based on language model: Use natural language processing technology to check the grammatical and semantic rationality of the initial recognition results and correct obvious errors.
[0198] Context verification: By analyzing the context, determine whether the content parameters of the initial recognition result are consistent with the context, and make corresponding adjustments.
[0199] User feedback verification: Users can be prompted to confirm the recognition results and corrections can be made based on user feedback.
[0200] Based on this, through the preset verification strategy, the initial recognition results can be continuously optimized and updated to ensure that the final written content is accurate, thereby improving the accuracy of recognition and enhancing the adaptability to different writing habits and styles.
[0201] Based on the above embodiment, in another embodiment provided by the present disclosure, the above-mentioned use of a preset verification strategy to verify and update the initial content restoration result to obtain the written content of the text to be recognized may specifically include:
[0202] During the writing process of the user, the three-dimensional coordinate data of the text to be recognized is obtained multiple times through coordinate mapping;
[0203] Through coordinate verification, it is determined whether two consecutive three-dimensional coordinate data are consistent;
[0204] When two consecutive three-dimensional coordinate data are consistent, the initial content restoration result is determined as the written content of the text to be recognized;
[0205] When two consecutive three-dimensional coordinate data are partially consistent, the original written content and the updated written content corresponding to the partially consistent three-dimensional coordinates are obtained; the updated written content is used to replace the original written content in the initial content restoration result to obtain the written content of the text to be recognized;
[0206] When two consecutive three-dimensional coordinate data are inconsistent, the initial content restoration results corresponding to each three-dimensional coordinate data are obtained, and the multiple initial content restoration results are combined as the written content of the text to be recognized.
[0207] In an embodiment, the three-dimensional coordinate data of the content parameters during the user's writing process can be first obtained through coordinate mapping. Coordinate mapping refers to obtaining and recording the three-dimensional coordinate data of the user's finger, pen tip or the top of the teaching tool in real time during the user's writing process. The three-dimensional coordinate data represents the user's writing trajectory and position in three-dimensional space. The three-dimensional coordinate data is recorded in real time to form a series of time series data, and the time series data can be used to represent the user's writing trajectory.
[0208] The gesture muscle movement model can use the three-dimensional coordinate data to confirm the position and size of the user's writing content and reproduce the user's writing trajectory to ensure the accuracy and consistency of the writing content.
[0209] Furthermore, coordinate verification is used to check and verify the three-dimensional coordinate data acquired in real time to ensure that the coordinate data is consistent with the previously recorded data and to handle possible duplicate coverage.
[0210] The coordinate verification process may include the following situations:
[0211] 1. The new and old coordinate data are completely consistent. Compare the newly acquired coordinate data with the previously recorded coordinate data. If the new and old coordinate data are consistent, it means that the user is using the previously used three-dimensional coordinates during the writing process. At this time, the existing coordinate content parameter results are used as the content parameter verification results.
[0212] 2. The new and old coordinate data are partially consistent. Assume that the user has written something (such as a letter or a stroke) at a certain location, and the 3D coordinate data of this location has been recorded. Now, the user writes at this location again, resulting in the new and old coordinate data being partially consistent. The specific operation steps may include:
[0213] Get the original coordinate data: The system has recorded the three-dimensional coordinate data that the user previously wrote at a certain location.
[0214] Obtain new coordinate data: When the user writes at the same location again, the system obtains new three-dimensional coordinate data.
[0215] Compare new and old data: The system compares the newly acquired coordinate data with the original coordinate data recorded previously.
[0216] Partial use of original coordinates: If the user only partially writes at the original location (for example, adding a stroke to an already written letter), the system needs to merge the new writing content with the original content.
[0217] Replace the original data position: The system compares and merges the newly written data with the original data. If the newly written content covers part of the original content, the system will replace the original data position with the position covered by the new data.
[0218] For example, if the user adds a stroke to the original letter "C" and it becomes the letter "G", the system will replace the corresponding part of the original data (the coordinate data of "C") with the new data (the coordinate data of "G").
[0219] Update content parameter results: The system uses the fused data as the new content parameter verification result. For example: suppose the user writes the letter "C" at a certain position for the first time, and the system records the three-dimensional coordinate data of "C". Then, the user adds another stroke at the same position, so that "C" becomes "G". At this time, the original data is the coordinate data of "C" recorded by the system, and the new data is the coordinate data of "G" after the user adds a stroke. When the system compares the two data, it is found that the new data covers part of the original data. Therefore, the system replaces the corresponding part of the coordinate data of "C" with the coordinate data of "G", updates the entire coordinate data, and finally saves it as the "G" written by the user.
[0220] 3. Repeated overwriting of original coordinate data or inconsistent content parameter attributes. During the writing process, sometimes the user writes again at a coordinate location that has been used, or the new writing content is different from the previous content parameter attributes. Specific operation steps may include:
[0221] Detect whether coordinate data is repeatedly overwritten: detect that the user writes again at a 3D coordinate position that has been used.
[0222] Check whether the attributes are inconsistent: detect that the new writing content attributes (such as character type, stroke attributes, etc.) are inconsistent with the previously recorded content attributes.
[0223] The processing method includes: saving the data that reuses the original coordinates as a complete result record, that is, the system will not simply overwrite the old data, but record the new written content together with the old content as part of the content parameter verification result.
[0224] For example, the user writes the number "1" at a certain location for the first time, and the system records the three-dimensional coordinate data of "1". Then, the user writes the letter "I" at the same location. The user writes "I" at the location where "1" was written, causing the new writing track to repeatedly cover the original coordinate data, and the attributes of the number "1" are inconsistent with the attributes of the letter "I". At this time, the new writing content (letter "I") needs to be recorded together with the previous writing content (number "1") as a complete final output result.
[0225] 4. Consider the deformation effect of the Z coordinate axis on the content parameter output results. In three-dimensional space, the written content not only involves the position changes of the X-axis and Y-axis, but also the changes of the Z-axis (depth axis). During the writing process, the user's writing tool (such as the pen tip or finger) may move up and down on the Z-axis to a certain extent. This movement will affect the shape and position of the written content. For example, the change in the height of the pen tip when lifting and pressing when writing will result in different strokes. The thickness and connection method. Therefore, the writing content of different depths will affect the final recognition results. In the content parameter verification, it is necessary to consider the impact of the Z coordinate axis data on the writing content.
[0226] Specific steps may include:
[0227] Height difference detection: Detect the height change of the Z axis during writing. If the height change exceeds a certain threshold, the writing content needs to be adjusted.
[0228] Smoothing: Smooth the writing trajectory to eliminate discontinuity caused by Z-axis changes.
[0229] For example, when the user writes the letter "L", the pen tip moves up and down significantly on the Z axis, causing the starting point to be higher than the ending point. The vertical stroke of "L" does not look like a straight line, but has a downward tilt. After detecting this Z-axis change, the tilted stroke can be corrected to a straight line by adjusting the Z-axis coordinate, so that the written content is accurately restored to the "L" that the user intended to write.
[0230] Finally, different content parameters can be distinguished and identified through content parameter verification to avoid confusion. Specific steps may include:
[0231] When the user writes, the attributes of the written content are recognized in real time, for example, whether the current writing is a letter, number or symbol.
[0232] The specific location and trajectory of the written content are determined through three-dimensional coordinate data, and these data are compared with the existing model.
[0233] Set verification rules for easily confused characters. For example, the Chinese stroke "vertical" (stroke I) and the number "1" are very similar in shape, but can be distinguished by the content parameters of the context, writing habits, and subtle shape differences.
[0234] For example, when the user writes a "1" or an "I" during the writing process, first identify that the user has written a vertical line (which may be the number "1" or the letter "I"). Then, through the three-dimensional coordinate data, determine the specific position and angle of the written vertical line. If the vertical line appears in a number string (such as "123"), it can be more inclined to be recognized as the number "1"; if the vertical line appears in a letter string (such as "IT"), the system can be more inclined to recognize it as the letter "I".
[0235] In an optional manner, further checks can also be performed based on the verification rules. For example: If there are obvious starting and ending pen marks on the upper and lower parts of the vertical line, it may be closer to the Chinese stroke "vertical"; if the height and line thickness of the vertical line conform to the writing habits of the number "1", it may be more inclined to be recognized as the number "1".
[0236] Based on this, through coordinate mapping, coordinate verification, and content parameter verification, the writing content of the user can be effectively matched and verified, solving the situation of partial duplicate coverage data and improving the accuracy of the content parameter verification result.
[0237] In one or more technical solutions provided by the exemplary embodiments of the present disclosure, a gesture muscle movement model is established based on the known distance and each stroke content of the smart wearable device such as a bracelet or a ring and the user's handwritten input. Then, through tapping and the sound collection function of the smart wearable device, an azimuth distance relationship matrix between the smart wearable device and the tapping position is obtained and established. Using the gesture muscle movement model and the azimuth distance relationship matrix, each stroke content written by the user is restored. Finally, through coordinate mapping, coordinate verification, and content parameter verification, the writing content of the user can be effectively matched and verified, solving the situation of partial duplicate coverage data and improving the accuracy of the content parameter verification result.
[0238] Therefore, the handwritten content recognition method provided by the exemplary embodiments of the present disclosure can generate the user's gesture writing content in real time without relying on a specific writing hardware device and any lighting conditions.
[0239] The above mainly introduces the solutions provided by the exemplary embodiments of the present disclosure. It is understandable that in order to implement the above functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.
[0240] The exemplary embodiments of the present disclosure can divide the functional units of the electronic device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the exemplary embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0241] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a handwriting content recognition device, which may be a server or a chip applied to a server. Figure 4 This is a schematic block diagram of the functional modules of a handwriting content recognition device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the handwriting content recognition device 400 includes:
[0242] The data acquisition module 410 is used to acquire the hand movement data of the user in the process of writing the text to be recognized when detecting that the user starts writing;
[0243] The data processing module 420 is used to extract the features to be recognized of the hand motion data, and match the content parameter recognition result of the text to be recognized based on the features to be recognized;
[0244] The data processing module 420 is used to restore the written content of the text to be recognized based on the content parameter recognition result and the hand movement data.
[0245] In another embodiment provided by the present disclosure, the data processing module 420 is also used to obtain a pre-constructed gesture muscle movement model; input the hand movement data into the gesture muscle movement model, and extract the features to be identified of the hand movement data; based on the features to be identified, determine the content parameter recognition results corresponding to the text to be identified in the content parameter feature table; the content parameter feature table includes content parameters, and sample features corresponding to the content parameters.
[0246] In another embodiment provided by the present disclosure, the data processing module 420 is also used to traverse the sample features corresponding to the content parameters in the content parameter feature table; obtain the similarity between the feature to be identified and the sample feature; when the similarity is greater than a preset threshold, obtain the content parameter associated with the corresponding sample feature in the content parameter feature table, and determine the content parameter as the content parameter identification result corresponding to the text to be identified.
[0247] In another embodiment provided by the present disclosure, the data processing module 420 is also used to respectively obtain the similarity between the feature to be identified and the sample feature corresponding to the single-stroke writing content parameter; when the similarity is less than a preset threshold, the similarity between the feature to be identified and the sample feature corresponding to the connected writing content parameter is respectively obtained.
[0248] In another embodiment provided by the present disclosure, the data processing module 420 is also used to construct a three-dimensional coordinate system; wherein, the gyroscope in the smart wearable device worn by the user is used as the origin of the three-dimensional coordinate system, the X-axis of the three-dimensional coordinate system is constructed based on the horizontal plane of the gyroscope, the Y-axis is established in a direction perpendicular to the upward direction of the X-axis, and the Z-axis is established in a direction perpendicular to both the X and Y axes; based on the three-dimensional coordinate system, the relative position information between the writing position of the text to be recognized and the smart wearable device is determined using a triangulation positioning method.
[0249] In another embodiment provided by the present disclosure, the data processing module 420 is also used to confirm the moving distance and moving direction of the user in the process of writing the text to be recognized based on the hand motion data and the relative position information; based on the moving distance, the moving direction and the content parameter recognition result, obtain the initial content restoration result of the text to be recognized; use a preset verification strategy to verify and update the initial content restoration result to obtain the written content of the text to be recognized.
[0250] In another embodiment provided by the present disclosure, the data processing module 420 is also used to obtain the three-dimensional coordinate data of the text to be recognized multiple times through coordinate mapping during the user's writing process; determine whether two consecutive three-dimensional coordinate data are consistent through coordinate verification; when two consecutive three-dimensional coordinate data are consistent, determine the initial content restoration result as the writing content of the text to be recognized; when two consecutive three-dimensional coordinate data are partially consistent, obtain the original writing content and updated writing content corresponding to the partially consistent three-dimensional coordinates; use the updated writing content to replace the original writing content in the initial content restoration result to obtain the writing content of the text to be recognized; when two consecutive three-dimensional coordinate data are inconsistent, obtain the initial content restoration results corresponding to each three-dimensional coordinate data, and merge multiple initial content restoration results as the writing content of the text to be recognized.
[0251] Figure 5 Schematic block diagram of a chip provided by an example of the present disclosure. Figure 5 As shown, the chip 500 includes one or more (including two) processors 501 and a communication interface 502. The communication interface 502 can support the server to perform the data sending and receiving steps in the above method, and the processor 501 can support the server to perform the data processing steps in the above method.
[0252] Optional, such as Figure 5 As shown, the chip 500 also includes a memory 503, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).
[0253] In some embodiments, Figure 5 As shown, the processor 501 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 501 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 503 may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory 503 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 5 Various buses are labeled as bus system 504 .
[0254] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.
[0255] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.
[0256] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.
[0257] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.
[0258] Figure 6The block diagram of the electronic device provided as an example of the present disclosure, the block diagram of the electronic device 600 that can be used as the server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0259] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0260] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0261] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. Each of the methods described above may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609.
[0262] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0263] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0264] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0265] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0266] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0267] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0268] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).
[0269] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A handwritten content recognition method, characterized in that: The method comprises: When detecting that the user starts writing, obtaining hand movement data of the user in the process of writing the text to be recognized; Extracting features to be recognized from the hand motion data, and matching content parameter recognition results of the text to be recognized based on the features to be recognized; Based on the content parameter recognition result and the hand movement data, the written content of the text to be recognized is restored.
2. The method according to claim 1, characterized in that The extracting the features to be recognized of the hand motion data and matching the content parameter recognition result of the text to be recognized based on the features to be recognized includes: Get a pre-built gesture muscle movement model; Inputting the hand motion data into a gesture muscle motion model to extract features to be identified from the hand motion data; Based on the features to be recognized, a content parameter recognition result corresponding to the text to be recognized is determined in a content parameter feature table; the content parameter feature table includes content parameters and sample features corresponding to the content parameters.
3. The method according to claim 2, characterized in that The determining, based on the feature to be recognized, in a content parameter feature table, a content parameter recognition result corresponding to the text to be recognized, comprises: Traversing the sample features corresponding to the content parameters in the content parameter feature table; Obtaining the similarity between the feature to be identified and the sample feature; When the similarity is greater than a preset threshold, the content parameter associated with the corresponding sample feature in the content parameter feature table is obtained, and the content parameter is determined as the content parameter recognition result corresponding to the text to be recognized.
4. The method according to claim 3, characterized in that The method further comprises: Respectively obtaining the similarity between the feature to be identified and the sample feature corresponding to the single-stroke writing content parameter; When the similarity is less than a preset threshold, the similarity between the feature to be identified and the sample feature corresponding to the cursive writing content parameter is obtained respectively.
5. The method according to claim 1, characterized in that The method further comprises: Constructing a three-dimensional coordinate system; wherein, the gyroscope in the smart wearable device worn by the user is used as the origin of the three-dimensional coordinate system, the X-axis of the three-dimensional coordinate system is constructed based on the horizontal plane of the gyroscope, the Y-axis is established in a direction perpendicular to the X-axis, and the Z-axis is established in a direction perpendicular to both the X-axis and the Y-axis; Based on the three-dimensional coordinate system, the relative position information between the writing position of the text to be recognized and the smart wearable device is determined using a triangulation positioning method.
6. The method according to claim 5, characterized in that The restoring the written content of the to-be-recognized text based on the content parameter recognition result and the hand movement data includes: Determine the moving distance and moving direction of the user when writing the text to be recognized based on the hand movement data and the relative position information; Based on the moving distance, the moving direction and the content parameter recognition result, obtaining an initial content restoration result of the text to be recognized; The initial content restoration result is verified and updated using a preset verification strategy to obtain the written content of the text to be recognized.
7. The method according to claim 6, characterized in that The step of using a preset verification strategy to verify and update the initial content restoration result to obtain the written content of the text to be recognized includes: During the writing process of the user, the three-dimensional coordinate data of the text to be recognized is obtained multiple times through coordinate mapping; Through coordinate verification, it is determined whether two consecutive three-dimensional coordinate data are consistent; When two consecutive three-dimensional coordinate data are consistent, the initial content restoration result is determined as the written content of the text to be recognized; When two consecutive three-dimensional coordinate data are partially consistent, the original written content and the updated written content corresponding to the partially consistent three-dimensional coordinates are obtained; the updated written content is used to replace the original written content in the initial content restoration result to obtain the written content of the text to be recognized; When two consecutive three-dimensional coordinate data are inconsistent, initial content restoration results corresponding to each three-dimensional coordinate data are obtained, and multiple initial content restoration results are combined as the written content of the text to be recognized.
8. A handwritten content recognition device, characterized in that: The device comprises: A data acquisition module, for acquiring hand movement data of a user in the process of writing the text to be recognized when detecting that the user starts writing; A data processing module, used for extracting features to be identified from the hand motion data, and matching content parameter identification results of the text to be identified based on the features to be identified; A data processing module is used to restore the written content of the text to be recognized based on the content parameter recognition result and the hand movement data.
9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the steps of the method according to any one of claims 1 to 7.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Writing pen touch recognition method based on gesture recognition and inertial sensor
CN121255037A
