Handwritten character recognition method and device combining point cloud and image features and medium
By combining point cloud and image features, the handwritten character recognition method solves the accuracy and efficiency problems of traditional methods in complex writing situations and diverse font styles, achieving higher recognition accuracy and work efficiency.
Patent Information
- Application Number
- CN202411440737.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-01-24
AI Technical Summary
Traditional handwritten character recognition methods have limited accuracy and efficiency when facing complex writing situations and diverse font styles.
A handwritten character recognition method that combines point cloud and image features is adopted. By obtaining point cloud data and image data of handwritten characters, feature extraction and fusion are performed, and multi-classifier blocks are used for character recognition.
It has significantly improved the ability to deal with complex backgrounds, vague text, different fonts and writing styles. The accuracy of identification results has been greatly enhanced, and the work efficiency of all walks of life has been improved.
Smart Images
Figure CN119360396A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, device and medium for handwritten character recognition combining point cloud and image features. Background Art
[0002] In today's era of rapid digital development, the communications industry has extremely high requirements for accurate and fast information processing. As one of the important ways to input information, the accuracy and efficiency of handwritten character recognition directly affect the quality of communication services.
[0003] Handwritten character recognition has broad application prospects in various fields of society, such as finance,
[0004] Medical treatment, education, office, etc. Handwritten character recognition also has a large number of application scenarios within the company, mainly concentrated in the following aspects: technical blackboard explanations at technical training sites, corresponding character recognition of blackboard content; relevant conference blackboards and LCD panel handwritten character recognition; various sign-in and name recognition and output in signature files, etc.; handwritten character recognition can quickly and accurately output electronic documents, improving the company's internal work efficiency.
[0005] However, traditional handwritten character recognition methods often show certain limitations when faced with complex writing situations and diverse font styles. Therefore, how to improve the accuracy and efficiency of handwritten character recognition results has become a problem that needs to be solved. Summary of the invention
[0006] The technical problem to be solved by the present application is to provide a handwritten character recognition method, device and medium combining point cloud and image features to solve the problems existing in the prior art in view of the above-mentioned deficiencies in the prior art.
[0007] In a first aspect, the present application provides a handwritten character recognition method combining point cloud and image features, the method comprising:
[0008] S1, obtaining point cloud data and image data of handwritten characters;
[0009] S2. Extracting features based on the point cloud data and the image data to obtain point cloud features and image features;
[0010] S3, fusing the point cloud features and the image features to obtain comprehensive features;
[0011] S4. Perform character recognition according to the comprehensive features to obtain a handwritten character recognition result.
[0012] In some embodiments, S2 includes:
[0013] Extracting features from the point cloud data using a neural network model to obtain point cloud features, wherein the point cloud features include at least geometric shape and spatial distribution;
[0014] The image data is subjected to feature extraction through a neural network model to obtain image features, wherein the image features at least include texture, edge, and contour.
[0015] In some embodiments, feature extraction is performed on the point cloud data through a neural network model to obtain point cloud features, including:
[0016] Performing feature extraction on the point cloud data through a point cloud feature extraction module in a neural network model to obtain point cloud features;
[0017] Among them, the point cloud convolution pconv of the point cloud feature extraction module is:
[0018]
[0019] Among them, the midpoint p i Represents the local neighborhood N(p i ), Represents the local neighborhood N(p i ), Represents the result of convolution; function σ represents the multi-layer perceptron MLP, function τ is the symmetric function max(), and function ρ represents the high-dimensional feature vector Mapping to low-dimensional feature vector
[0020] In some embodiments, extracting features from the image data using a neural network model to obtain image features includes:
[0021] The image data is subjected to feature extraction by an image feature extraction module using ResNet as a basic structure in a neural network model to obtain image features.
[0022] In some embodiments, S3 includes:
[0023] A fusion method based on deep learning is adopted to fuse the point cloud features and image features to obtain comprehensive features.
[0024] In some embodiments, S4 includes:
[0025] Performing character recognition on the comprehensive features through a multi-classifier block to obtain a handwritten character recognition result;
[0026] The multi-classifier block consists of a main classifier and an auxiliary classifier.
[0027] In some embodiments, after S1 and before S2, the process further includes:
[0028] Preprocessing the point cloud data and image data;
[0029] Wherein, the preprocessing of the point cloud data includes at least one of denoising and streamlining;
[0030] The preprocessing of the image data includes at least one of grayscale conversion, binarization, and normalization.
[0031] In a second aspect, the present application provides a handwritten character recognition device combining point cloud and image features, the device comprising:
[0032] A data acquisition module configured to acquire point cloud data and image data of handwritten characters;
[0033] A feature extraction module, which is configured to extract features based on the point cloud data and the image data to obtain point cloud features and image features;
[0034] A feature fusion module, which is configured to fuse the point cloud features and the image features to obtain comprehensive features;
[0035] The character recognition module is configured to perform character recognition based on the comprehensive features to obtain a handwritten character recognition result.
[0036] In a third aspect, the present application provides a handwritten character recognition device combining point cloud and image features, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the handwritten character recognition method combining point cloud and image features described in the first aspect above.
[0037] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the handwritten character recognition method combining point cloud and image features as described in the first aspect above is implemented.
[0038] The present application provides a method, device and medium for handwritten character recognition that combines point cloud and image features. The method includes: obtaining point cloud data and image data of handwritten characters; performing feature extraction based on the point cloud data and image data to obtain point cloud features and image features; performing feature fusion on the point cloud features and image features to obtain comprehensive features; performing character recognition based on the comprehensive features to obtain handwritten character recognition results. The present application proposes a method for handwritten character recognition that combines point cloud and image features. By combining handwritten character recognition with point cloud and image features, the ability to handle scenes with complex backgrounds, blurred text, different fonts and writing styles can be greatly improved. The accuracy of the recognition results has been greatly enhanced, which has improved the work efficiency of all walks of life. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0040] Figure 1 A flowchart of a handwritten character recognition method combining point cloud and image features provided in an embodiment of the present application;
[0041] Figure 2 A schematic diagram of a neural network model provided in an embodiment of the present application;
[0042] Figure 3 A schematic diagram of the structure of a handwritten character recognition device combining point cloud and image features provided in an embodiment of the present application;
[0043] Figure 4 A schematic diagram of the structure of another handwritten character recognition device combining point cloud and image features provided in an embodiment of the present application.
[0044] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the technical solution of the present application, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings.
[0046] It should be understood that the specific embodiments and drawings described herein are only used to explain the present application, rather than to limit the present application.
[0047] It can be understood that, in the absence of conflict, the various embodiments in the present application and the various features in the embodiments can be combined with each other.
[0048] It can be understood that, for the convenience of description, the drawings of the present application only show the parts related to the present application, while the parts unrelated to the present application are not shown in the drawings.
[0049] It can be understood that each unit and module involved in the embodiments of the present application may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.
[0050] It can be understood that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.
[0051] It is understandable that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present application may occur in an order different from that marked in the drawings.
[0052] It is understood that the flowcharts and block diagrams of the present application illustrate the possible architecture, functions and operations of the systems, devices, equipment and methods according to the embodiments of the present application. Among them, each box in the flowchart or block diagram may represent a unit, module, program segment, code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or by a combination of hardware and computer instructions.
[0053] It can be understood that the units and modules involved in the embodiments of the present application can be implemented by software or hardware, for example, the units and modules can be located in a processor.
[0054] Handwritten character recognition technology is a technology that uses optical technology and computer technology to read the characters written on paper and convert them into a format that can be read by computers and understood by people. This technology obtains image information through scanning, taking pictures, etc., and then uses recognition algorithms to convert the handwritten characters in the image into text format, realizing the automatic entry and processing of text information.
[0055] With the rapid development of deep learning, artificial intelligence and other technologies, the recognition accuracy and efficiency of handwritten character recognition technology have been significantly improved. In particular, the recognition ability has been greatly enhanced in dealing with complex backgrounds, blurred text, different fonts and writing styles.
[0056] The current handwritten character recognition technology mainly uses a neural network model for images, which simply treats handwritten characters as images and ignores the point cloud properties of handwritten characters. Handwritten characters are composed of sparse lines, and lines are composed of a series of two-dimensional point clouds. The ability of handwritten characters represented by image features is limited, while point cloud features can make up for the shortcomings of image features. Moreover, the existing point cloud convolution is mainly for three-dimensional point clouds.
[0057] This application proposes a handwritten character recognition method that combines point cloud and image features, which can well solve the above problems:
[0058] (1) The vector representation that combines point cloud features with image features contains both spatial information and visual information, and can more accurately represent handwritten characters.
[0059] (2) The point cloud convolution of the present invention is specially designed for handwritten characters, mapping high-dimensional vectors to low-bit space and eliminating redundant information.
[0060] The method of the present application can be applied to various scenarios in the communications industry, such as handwriting input in intelligent customer service, handwritten text message recognition on mobile terminals, handwritten document processing in office automation, etc., to improve the intelligence level of communication services and user experience.
[0061] In addition, the method of this application can be further combined with communication technologies, such as the low latency characteristics of 5G networks, to achieve real-time handwritten character recognition services. At the same time, explore the integration with cloud computing, edge computing and other technologies to optimize the allocation and utilization of computing resources. By continuously improving and optimizing methods, paying attention to the development and demand changes in the communications industry, and constantly updating and expanding data sets to adapt to new font styles, writing tools and application scenarios, we can provide more powerful technical support for the digital transformation of the communications industry.
[0062] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0063] The present application provides a handwritten character recognition method that combines point cloud and image features. The working process of the method can be implemented by electronic devices, such as computers, handheld smart terminals, etc. For the convenience of explanation, the embodiments of the present application are described with the method execution subject being a computer.
[0064] Figure 1 A schematic diagram of a handwritten character recognition method combining point cloud and image features provided in an embodiment of the present application, such as Figure 1As shown, the present application provides a handwritten character recognition method combining point cloud and image features, the method includes S1-S4, which are as follows:
[0065] S1, obtaining point cloud data and image data of handwritten characters;
[0066] First, handwritten text is captured in real time through high-precision scanning equipment or sensors to obtain its point cloud data and image data. Point cloud data can accurately record the three-dimensional spatial coordinate information of handwritten text, while image data can reflect the visual characteristics of handwritten text such as texture and color.
[0067] In some embodiments, after S1 and before S2, it also includes: preprocessing the point cloud data and image data; wherein the preprocessing of the point cloud data includes at least one of denoising and simplification; the preprocessing of the image data includes at least one of grayscale, binarization, and normalization.
[0068] Specifically, the point cloud data and image data are preprocessed to improve the quality of the data and ensure the accuracy and efficiency of subsequent processing and analysis.
[0069] For the preprocessing of the point cloud data, at least one of the following operations is performed:
[0070] (1) Denoising: Point cloud data may contain noise points due to the influence of equipment performance, environmental factors or operational errors during data collection. Denoising operation uses algorithms to identify and remove these abnormal points to reduce the impact of noise on subsequent analysis and ensure the smoothness and accuracy of point cloud data.
[0071] (2) Simplification: Point cloud data usually contains a large number of points, which may make the data processing and analysis process complicated and time-consuming. Simplification can optimize the data processing process by reducing the number of data points while maintaining the basic shape and structural characteristics of handwritten characters. This can be achieved through a variety of methods, such as uniform sampling, spatial clustering, etc.
[0072] In addition, for preprocessing of image data, perform at least one of the following operations:
[0073] (1) Grayscale conversion: Converting a color image to a grayscale image can simplify the image data processing process because a grayscale image only contains brightness information and does not involve color channels. This step helps reduce computational complexity. At the same time, for handwritten text recognition, color information is often not a key factor.
[0074] (2) Binarization: Converting a grayscale image into a binary image containing only black and white can further simplify the image data, making the text outline in the image clearer and facilitating subsequent feature extraction and recognition. During the binarization process, it is necessary to select an appropriate threshold to ensure that the text information is effectively retained.
[0075] (3) Normalization: Adjust the size, brightness, and contrast of the image data to make it conform to a certain standard format. The normalization operation helps to eliminate data differences caused by different writing conditions and improve the generalization ability of the handwritten text recognition system.
[0076] Through the above preprocessing steps, a cleaner, more consistent and easier to process data set can be obtained, which is convenient for subsequent handwritten text feature extraction, analysis and recognition processing.
[0077] S2. Extracting features based on the point cloud data and the image data to obtain point cloud features and image features;
[0078] In some embodiments, S2 includes: performing feature extraction on the point cloud data through a neural network model to obtain point cloud features, wherein the point cloud features include at least geometric shape and spatial distribution; performing feature extraction on the image data through a neural network model to obtain image features, wherein the image features include at least texture, edge, and contour.
[0079] In this step, the feature extraction of the point cloud data is performed through a neural network model, which can deeply mine the three-dimensional structure information in the point cloud data, thereby obtaining a series of highly representative point cloud features.
[0080] Among them, the neural network model can recognize and extract basic geometric shapes in point cloud data, such as planes, curved surfaces, edges and corners. These geometric shape features are crucial for understanding and recognizing the structure of handwritten text, because they can reflect the thickness of the strokes, the way they are connected, and the overall shape of the text.
[0081] In addition, the spatial distribution characteristics of point cloud data describe the arrangement and relative position of data points in three-dimensional space. By learning these distribution characteristics, the neural network model can capture the continuity and change trend of handwritten text in space, which is of great significance for identifying different writing styles and subtle differences in handwriting.
[0082] Similarly, by performing feature extraction on the image data through a neural network model, key planar visual feature information can be extracted from the image to form a set of image features.
[0083] Among them, texture features reflect the arrangement and repetitive patterns of pixels in an image, which can reveal the surface characteristics of handwritten text, such as the roughness of the brushstrokes, the texture of the paper, etc. By learning these texture features, the neural network model can enhance the recognition ability of handwritten text, especially in complex backgrounds or low-resolution images.
[0084] In addition, edge features are important information about the outline of objects in an image, which define the boundaries of the strokes of a character. By extracting edge features, the neural network model can clearly identify the stroke structure of handwritten characters, which is crucial for distinguishing similar characters and identifying connected writing.
[0085] In addition, contour features are an abstract description of the shape of objects in an image, which includes not only edge information but also closed curves and shapes of objects. By learning contour features, the neural network model can better understand the overall shape and structure of handwritten text, thereby improving the recognition accuracy.
[0086] Through the above feature extraction process, the neural network model can extract highly abstract and discriminative features from point cloud data and image data. These features provide strong support for subsequent handwritten text recognition, classification and analysis.
[0087] Figure 2 A schematic diagram of a neural network model provided in an embodiment of the present application, such as Figure 2 As shown, the neural network model includes an image feature extraction module, a point cloud feature extraction module, and a multi-classifier block.
[0088] In some embodiments, feature extraction is performed on the image data through a neural network model to obtain image features, including: feature extraction is performed on the image data through an image feature extraction module that uses ResNet as a basic structure in the neural network model to obtain image features.
[0089] Specifically, the image feature extraction module uses ResNet as the basic structure and inputs the handwritten character image S px ∈R W ×H , output image feature vector f px σR Dpx .
[0090] In some embodiments, extracting features from the point cloud data through a neural network model to obtain point cloud features includes: extracting features from the point cloud data through a point cloud feature extraction module in the neural network model to obtain point cloud features;
[0091] Among them, the point cloud convolution pconv of the point cloud feature extraction module is:
[0092]
[0093] Among them, the midpoint p i Represents the local neighborhood N(p i ), Represents the local neighborhood N(p i The features of each point in 0, represents the result of convolution; the function σ represents a multi-layer perceptron (MLP), which must perform convolution on the local neighborhood N (p i ) to achieve the effect of feature transformation, the function τ is the symmetric function max(), and the function ρ represents the high-dimensional feature vector Mapping to low-dimensional feature vector Low-dimensional features are suitable for representing sparse character point clouds.
[0094] The point cloud convolution (pconv) in the point cloud feature extraction module is an extension of the point cloud convolution in PointNet++. The pconv in PointNet++ can be expressed as follows:
[0095]
[0096] Among them, the midpoint p i Represents the local neighborhood N(p i ), Represents the local neighborhood N(p i ), represents the result of convolution; the function σ represents the multi-layer perceptron MLP, which must perform convolution on the local neighborhood N(p i ) to achieve the effect of feature transformation, and the function τ is the symmetric function max().
[0097] refer to Figure 2 , the point cloud feature extraction module of this application contains three point set convolutional layers: pconv1, pconv2 and pconv3. The number of parameters of functions σ and ρ (i.e., MLP) has been described in Figure 2 Before the pconv convolution layer, M is selected by the farthest point sampling method. i centers, and assign K i In the present application, (M1, M2, M3) = (128, 64, 1), and (K1, K2, K3) = (32, 32, 64).
[0098] Given a handwritten character image S px, first record all covered pixel coordinates, sample N points for all pixel coordinates, and obtain the sketch point set S pt The point cloud feature extraction module inputs an S pt , output a point cloud feature vector f pt ∈R Dpt .
[0099] S3, fusing the point cloud features and the image features to obtain comprehensive features;
[0100] In some embodiments, S3 includes: using a deep learning-based fusion method to fuse the point cloud features and image features to obtain comprehensive features.
[0101] For example, advanced network structures in deep learning frameworks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformer networks, can be used to build a fusion model that can simultaneously process point cloud and image data, and then the point cloud features and image features are fused through the fusion model to obtain comprehensive features.
[0102] In the process of feature fusion, the following steps can be taken:
[0103] (1) Feature alignment: Since point cloud features and image features may be misaligned in space and time, we first use technical means such as spatial transformation and temporal synchronization to ensure that the two features have a good correspondence before fusion.
[0104] (2) Feature embedding: In order to enable the point cloud features and image features to be fused in the same feature space, they are converted into feature representations with the same dimension through the embedding layer, which facilitates the subsequent fusion operation.
[0105] (3) Feature interaction: After the embedding layer, multimodal feature interaction strategies are used, such as feature concatenation, feature weighted summation, or more complex attention mechanisms, to promote information exchange between point cloud features and image features. These interaction strategies can effectively extract complementary information between the two features and enhance the expressiveness of the fused features.
[0106] (4) Feature abstraction: Through a series of deep learning layers, such as fully connected layers, convolutional layers, or self-attention layers, the interacted features are further abstracted to obtain higher-level comprehensive features. These hierarchical features can capture abstract concepts and deep relationships in the data.
[0107] (5) Feature optimization: In order to ensure that the integrated features after fusion have the best discriminability and generalization ability, the fusion model is trained and optimized through the design of loss function and adjustment of optimization algorithm. This step aims to reduce overfitting and improve the performance of the model on unknown data.
[0108] Finally, through this series of feature fusion operations, a comprehensive feature is obtained, which not only contains the geometric shape and spatial distribution information of the point cloud data, but also integrates the texture, edge and contour information of the image data. Such comprehensive features provide richer and more comprehensive inputs for subsequent tasks, such as handwritten text recognition, 3D object detection, scene understanding, etc., thus significantly improving the performance and accuracy of the tasks.
[0109] S4. Perform character recognition according to the comprehensive features to obtain a handwritten character recognition result.
[0110] In some embodiments, S4 includes: performing character recognition on the comprehensive features through a multi-classifier block to obtain a handwritten character recognition result; wherein the multi-classifier block is composed of a main classifier and an auxiliary classifier.
[0111] refer to Figure 2 , the multi-classifier block contains three classifiers: the main classifier C main , auxiliary classifier C px and C pt The two auxiliary classifiers also play an important role, which can respectively ensure that the image features f px And point cloud features f pt Thus, the performance of the main classifier is improved.
[0112] When predicting, only the main classifier C is used main Predict the category of each handwritten character image. Each classifier is a computational sequence that includes a fully connected layer (FC) and a Softmax function layer. The main classifier input feature f px and f pt , input probability vector O main . Classifier C px Input f px , output O px . Classifier C pt Input f pt , output O pt .
[0113] It is understood that in this application Figure 2The neural network model shown can be trained in an end-to-end manner. However, training this model from scratch and updating all weights at once can easily lead to overfitting. Therefore, a pre-trained ResNet and a three-step training strategy are used.
[0114] Specifically, the three-step training strategy is as follows:
[0115] (1) The weights in the image feature extraction module are initialized by ResNet, which is pre-trained on the ImageNet dataset. First, only the image feature extraction module and classifier C are updated. px The loss function in the first step is L px .
[0116] (2) Only update the point set feature extraction module and classifier C pt The weights of are adjusted until convergence, and the other weights are frozen. The loss function in the second step is L pt .
[0117] (3) Update all weights in the model until convergence, the loss function is
[0118] L=L main +L px +L pt
[0119] Among them, L main Indicates the output value O main and the cross entropy loss of the true value G, L px Indicates the output value O px and the cross entropy loss of the true value G, L pt Indicates the output value O pt and the cross entropy loss of the true value G.
[0120] This application proposes a handwritten character recognition method that combines point cloud and image features. By combining point cloud and image features for handwritten character recognition, the ability to handle scenes with complex backgrounds, blurred text, different fonts and writing styles can be greatly improved. The accuracy of the recognition results has been greatly enhanced, thereby improving work efficiency in all walks of life.
[0121] It should be understood that, although the various steps in the flowcharts in the above-described embodiments are sequentially displayed according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily to be carried out sequentially, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0122] Figure 3 A schematic diagram of a handwritten character recognition device combining point cloud and image features provided in an embodiment of the present application, such as Figure 3 As shown, the present application provides a handwritten character recognition device combining point cloud and image features, the device comprising:
[0123] A data acquisition module 11, which is configured to acquire point cloud data and image data of handwritten characters;
[0124] A feature extraction module 12, which is configured to extract features based on the point cloud data and the image data to obtain point cloud features and image features;
[0125] A feature fusion module 13, which is configured to fuse the point cloud features and the image features to obtain comprehensive features;
[0126] The character recognition module 14 is configured to perform character recognition according to the comprehensive features to obtain a handwritten character recognition result.
[0127] Regarding the limitation of the handwritten character recognition device combining point cloud and image features, reference may be made to the limitation of the handwritten character recognition method combining point cloud and image features in the above-mentioned embodiments of the present application, which will not be repeated in this embodiment.
[0128] Figure 4 Another schematic diagram of a handwritten character recognition device combining point cloud and image features provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, in some embodiments, the present application provides a handwritten character recognition device combining point cloud and image features, including a memory 22 and a processor 21, the memory storing a computer program, and the processor being configured to run the computer program to execute the handwritten character recognition method combining point cloud and image features in the above-mentioned embodiments of the present application.
[0129] The memory is connected to the processor, the memory may be a flash memory or a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.
[0130] In some embodiments, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the handwritten character recognition method combining point cloud and image features in the above-mentioned embodiments of the present application is implemented.
[0131] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.
[0132] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.
Claims
1. A handwritten character recognition method combining point cloud and image features, characterized in that: The method comprises: S1, obtaining point cloud data and image data of handwritten characters; S2. Extracting features based on the point cloud data and the image data to obtain point cloud features and image features; S3, fusing the point cloud features and the image features to obtain comprehensive features; S4. Perform character recognition according to the comprehensive features to obtain a handwritten character recognition result.
2. The handwritten character recognition method combining point cloud and image features according to claim 1, characterized in that: S2, including: Extracting features from the point cloud data using a neural network model to obtain point cloud features, wherein the point cloud features include at least geometric shape and spatial distribution; The image data is subjected to feature extraction through a neural network model to obtain image features, wherein the image features at least include texture, edge, and contour.
3. The handwritten character recognition method combining point cloud and image features according to claim 2, characterized in that: The point cloud data is subjected to feature extraction through a neural network model to obtain point cloud features, including: Performing feature extraction on the point cloud data through a point cloud feature extraction module in a neural network model to obtain point cloud features; Among them, the point cloud convolution pconv of the point cloud feature extraction module is: Among them, the midpoint p i Represents the local neighborhood N(p i ), Represents the local neighborhood N(p i ), Represents the result of convolution; function σ represents the multi-layer perceptron MLP, function τ is the symmetric function max(), and function ρ represents the high-dimensional feature vector Mapping to low-dimensional feature vector 4. The handwritten character recognition method combining point cloud and image features according to claim 2, characterized in that: The image data is subjected to feature extraction by a neural network model to obtain image features, including: The image data is subjected to feature extraction by an image feature extraction module using ResNet as a basic structure in a neural network model to obtain image features.
5. The handwritten character recognition method combining point cloud and image features according to claim 1, characterized in that: S3, including: A fusion method based on deep learning is adopted to fuse the point cloud features and image features to obtain comprehensive features.
6. The handwritten character recognition method combining point cloud and image features according to claim 1, characterized in that: S4, including: Performing character recognition on the comprehensive features through a multi-classifier block to obtain a handwritten character recognition result; The multi-classifier block consists of a main classifier and an auxiliary classifier.
7. The handwritten character recognition method combining point cloud and image features according to any one of claims 1 to 6, characterized in that: After S1 and before S2, it also includes: Preprocessing the point cloud data and image data; Wherein, the preprocessing of the point cloud data includes at least one of denoising and streamlining; The preprocessing of the image data includes at least one of grayscale conversion, binarization, and normalization.
8. A handwritten character recognition device combining point cloud and image features, characterized in that: The device comprises: A data acquisition module configured to acquire point cloud data and image data of handwritten characters; A feature extraction module, which is configured to extract features based on the point cloud data and the image data to obtain point cloud features and image features; A feature fusion module, which is configured to fuse the point cloud features and the image features to obtain comprehensive features; The character recognition module is configured to perform character recognition based on the comprehensive features to obtain a handwritten character recognition result.
9. A handwritten character recognition device combining point cloud and image features, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the handwritten character recognition method combining point cloud and image features as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the handwritten character recognition method combining point cloud and image features according to any one of claims 1 to 7 is implemented.