Air Writing Trajectory Recognition Method, Device, Storage Medium and Electronic Device

Abstract

Embodiments of the present application disclose a method, apparatus, storage medium, and electronic device for recognizing an air writing trajectory. The method includes: receiving attitude sensing parameters sent by a smart device, where the attitude sensing parameters are generated when the smart device senses an air writing operation, generating writing trajectory data based on the attitude sensing parameters, inputting the writing trajectory data into a writing recognition model, and inputting the writing trajectory data into the writing recognition model, and based on the writing recognition model, outputting a writing recognition result corresponding to the writing trajectory data. By using the embodiments of the present application, the accuracy of air writing trajectory recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to an air writing trajectory recognition method, apparatus, storage medium, and electronic device. Background Art

[0002] As a new means of human-computer interaction, air writing is different from traditional human-computer interaction methods. Air handwriting allows users to manipulate intelligent devices (such as wearable devices) to write in the air in a natural and unrestrained manner. A display device (such as a smart TV or a large-screen display) can recognize the air writing trajectory when the user writes in the air based on the intelligent device, so as to obtain the writing data that the user expects to input; air writing provides a more intuitive, convenient, and comfortable interaction experience. Summary of the Invention

[0003] Embodiments of this application provide an air writing trajectory recognition method, apparatus, storage medium, and electronic device. The technical solutions of the embodiments of this application are as follows:

[0004] In a first aspect, an embodiment of this application provides an air writing trajectory recognition method, and the method includes:

[0005] Receiving attitude sensing parameters sent by an intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an air writing operation;

[0006] Generating writing trajectory data based on the attitude sensing parameters;

[0007] Inputting the writing trajectory data into a writing recognition model, and outputting a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0008] In a second aspect, an embodiment of this application provides an air writing trajectory recognition method, and the method includes:

[0009] Sensing an air writing operation on the display device, and collecting attitude sensing data corresponding to the air writing operation;

[0010] Sending the attitude sensing data to the display device, so that the display device generates writing trajectory data corresponding to the attitude sensing data, and inputs the writing trajectory data into a writing recognition model to output a writing recognition result.

[0011] In a third aspect, an embodiment of this application provides an air writing trajectory recognition apparatus, and the apparatus includes:

[0012] An induction parameter receiving module, configured to receive attitude sensing parameters sent by an intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an air writing operation;

[0013] A writing data generation module, configured to generate writing trajectory data based on the posture sensing parameters;

[0014] A writing data output module, configured to input the writing trajectory data into a writing recognition model, and output a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0015] In a fourth aspect, an embodiment of the present application provides an in-air writing trajectory recognition device, where the device includes:

[0016] A data sensing module, configured to sense an in-air writing operation on the display device and collect posture sensing data corresponding to the in-air writing operation;

[0017] A data sending module, configured to send the posture sensing data to the display device, so that the display device generates writing trajectory data corresponding to the posture sensing data, and inputs the writing trajectory data into a writing recognition model to output a writing recognition result.

[0018] In a fifth aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores multiple instructions, and the instructions are suitable for being loaded and executed by a processor to perform the above method steps.

[0019] In a sixth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein, the memory stores a computer program, and the computer program is suitable for being loaded and executed by the processor to perform the above method steps.

[0020] The beneficial effects brought by the technical solutions provided by some embodiments of the present application at least include:

[0021] In one or more embodiments of the present application, a display device may receive gesture sensing parameters sent by a smart device, where the gesture sensing parameters are generated when the smart device senses an in-air writing operation; then generate writing trajectory data based on the gesture sensing parameters; and then input the writing trajectory data into a writing recognition model to output an accurate writing recognition result corresponding to the writing trajectory data. Additionally, in some embodiments, the writing recognition model may be obtained through transfer training based on third writing data in an in-air medium and fourth writing data in a planar medium. Based on the foregoing method, problems such as low accuracy and low recognition processing efficiency of in-air writing trajectory recognition in related technologies can be avoided, and the convenience of cross-device collaborative control can be improved; and writing data recognition can be achieved only based on gesture sensing data collected by the smart device, without the need to collect three-dimensional data of the smart device or remap the three-dimensional data to two dimensions for recognition, greatly reducing the computational amount of in-air writing recognition and improving the recognition processing efficiency; and in the in-air writing recognition stage, a writing recognition model based on transfer learning is adopted, and a general pre-trained model with average performance can be obtained through existing relevant data (such as fourth writing data in a planar medium), and then the model is fine-tuned using a small amount of domain data to obtain a model with better matching for the current data type. This also improves the robustness of the model, and only a small amount of in-air writing data is required to ensure the accuracy of in-air writing trajectory recognition and improve the recognition processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0023] Figure 1 is a flowchart of a method for recognizing an in-air writing trajectory provided by an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of a handwriting sampling scenario involved in the method for recognizing an in-air writing trajectory provided by an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a scenario for separating the fall and lift of a stroke involved in the method for recognizing an in-air writing trajectory provided by an embodiment of the present application;

[0026] Figure 4 is a flowchart of another method for recognizing an in-air writing trajectory provided by an embodiment of the present application;

[0027] Figure 5It is a schematic diagram of parameters in an air writing scenario involved in the air writing trajectory recognition method provided by an embodiment of this application;

[0028] Figure 6 It is a schematic diagram of a handwriting sampling scenario involved in the air writing trajectory recognition method provided by an embodiment of this application;

[0029] Figure 7 It is a schematic diagram of a handwriting adjustment scenario involved in the air writing trajectory recognition method provided by an embodiment of this application;

[0030] Figure 8 It is a scenario usage diagram for displaying the writing recognition result involved in the air writing trajectory recognition method provided by an embodiment of this application;

[0031] Figure 9 It is a schematic flowchart of an air writing trajectory recognition method provided by an embodiment of this application;

[0032] Figure 10 It is a schematic flowchart of an air writing trajectory recognition method provided by an embodiment of this application;

[0033] Figure 11 It is a schematic structural diagram of an air writing trajectory recognition device provided by an embodiment of this application;

[0034] Figure 12 It is a schematic structural diagram of an induction parameter receiving module provided by an embodiment of this application;

[0035] Figure 13 It is a schematic structural diagram of a parameter determination unit provided by an embodiment of this application;

[0036] Figure 14 It is a schematic structural diagram of another air writing trajectory recognition device provided by an embodiment of this application;

[0037] Figure 15 It is a schematic structural diagram of another air writing trajectory recognition device provided by an embodiment of this application;

[0038] Figure 16 It is a schematic structural diagram of an electronic device provided by an embodiment of this application;

[0039] Figure 17 It is a schematic structural diagram of an operating system and user space provided by an embodiment of this application;

[0040] Figure 18 is Figure 16 the architecture diagram of the Android operating system in;

[0041] Figure 19 is Figure 16 the architecture diagram of the IOS operating system in;

[0042] Figure 20 It is a schematic structural diagram of another electronic device provided by an embodiment of the present application. Detailed implementation manners

[0043] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0044] In the description of the present application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In the description of the present application, it should be noted that unless otherwise clearly specified and limited, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes unlisted steps or units, or optionally further includes other steps or units inherent to these processes, methods, products or devices. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific situations. In addition, in the description of the present application, unless otherwise stated, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0045] The present application will be described in detail below with reference to specific embodiments.

[0046] In one embodiment, as Figure 1 shown, a method for recognizing an air writing trajectory is proposed, which is applied to a display device. This method can be implemented depending on a computer program and can run on an air writing trajectory recognition device based on the von Neumann architecture. This computer program can be integrated in an application or run as an independent tool application.

[0047] The above-mentioned air writing trajectory recognition device may be a display device, and the display device includes but is not limited to: personal computers, tablet computers, smart TVs, vehicle-mounted devices, large-screen display devices, computing devices, or other processing devices connected to a wireless modem, etc. In different networks, the display device may be called by different names. For example: user equipment, access terminal, user unit, user station, mobile station, mobile device, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), display device in a 5G network or future evolved network, etc.

[0048] Specifically, the air writing trajectory recognition method includes:

[0049] Step S101: Receive the attitude sensing parameters sent by the intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an air writing operation.

[0050] The intelligent device and the display device can be used in cooperation. The intelligent device can be an electronic device that facilitates users to perform air writing operations. As a new means of human-computer interaction, air writing is different from traditional human-computer interaction methods. In this application, air handwriting allows users to manipulate the intelligent device to write in the air in a natural and unrestrained manner, thus providing a more intuitive, convenient, and comfortable interaction experience. For example, in some scenarios, the display device can be a weak input device (such as a TV, conference tablet), that is, it is not convenient for users to directly input relevant information, such as writing characters, drawing, etc. Therefore, in some interaction scenarios, air writing based on the intelligent device is an interaction form that can achieve the purpose of complex information instruction input. Air writing includes writing input forms with paper or electronic media as the writing surface, and input forms of air handwriting by waving fingers / wrists / arms, etc. in three-dimensional air media. The recognition technology for writing on a flat medium has been relatively mature, such as the touch screen input methods carried by various smartphones. In some implementation scenarios, with the maturity of technology products in the fields of virtual reality and the Internet of Things, the feature that air writing based on intelligent devices (such as smart bracelets, smart watches, smart pens, etc.) is not limited to a flat surface can be well combined with display devices that are not convenient for users to directly write (such as TVs, conference tablets, etc.) and applied.

[0051] In addition, in some implementation scenarios, the display device may not actively collect the relevant pose data corresponding to the smart device or the user using the smart device. For example, when the display device includes an image acquisition device, it may not actively collect the pose data through the image acquisition device. Instead, when the smart device detects an air writing operation by the user, the smart device collects the pose sensing data corresponding to the air writing operation through the included sensor device, and then sends the pose sensing data to the display device for identifying the air writing trajectory during the air writing operation.

[0052] The pose sensing data refers to the position, pose, angular velocity data, acceleration data, etc. sensed by a movable or rotatable object (such as a smart device) based on the included sensor device at the current sampling time point. In this application, it can be understood as the position of the smart device in space and / or its pose at this position when the user is operating the smart device. In this embodiment, the pose sensing data can be used to characterize or quantify the current pose of the smart device. For example, the change in angular velocity of the smart device (the rotation angle between two adjacent sampling moments), the change in moving speed, the change in moving direction, the angle parameter of the smart device relative to the reference two-dimensional plane (which can be the reference two-dimensional plane of the display device or the surrounding environmental objects (such as walls)) (the included angle between the display device's pointing direction and the normal line of the reference two-dimensional plane such as the display device is α), and the distance parameter (such as the distance R between the smart device and the reference two-dimensional plane such as the display device), etc.

[0053] Furthermore, the smart device has a variety of sensor devices for real-time detecting the current pose information. The sensor devices include, but are not limited to, acceleration sensors, magnetic sensors, gyroscopes, physical quantity sensors, etc. The control device obtains the current physical quantity parameters through the electronic components. The physical quantity parameters can be acceleration parameters, magnetic parameters, angular velocity parameters, relative distance and relative angle relative to the reference two-dimensional plane such as the display device, etc., and performs pose calculation on the physical quantity parameters, that is, the pose sensing data of the smart device can be obtained.

[0054] Specifically, when the smart device senses that the user operates the smart device for an air writing operation, the smart device can sense the current pose sensing data through the included sensor device. In practical applications, the smart device can establish a communication connection with the display device (such as a Bluetooth connection, a wireless local area network connection, etc.), and then after collecting the current pose sensing data, transmit the pose sensing data to the display device based on this communication connection, so that the display device can receive the pose sensing data of the display device through this communication connection.

[0055] Step S102: Generate writing trajectory data based on the pose sensing parameters.

[0056] In practical applications, the attitude sensing data during air writing operations is collected in a three-dimensional space, corresponding to the data of the attitude change of the three-dimensional space device itself. In this application, air writing handwriting recognition is to map the attitude sensing parameters during its air writing to a two-dimensional plane based on the attitude sensing data, so as to determine the writing trajectory data on the two-dimensional plane corresponding to the air writing operation, such as writing trajectory data of character handwriting, Chinese character handwriting, English handwriting, picture handwriting, etc. In this application, the movement trajectory is mainly determined based on the angular velocity in the attitude sensing data, and then the writing trajectory data corresponding to the movement trajectory is formed. Among them, the writing trajectory data is pictures representing the movement trajectory, data in a predetermined format (such as txt format), etc.

[0057] In a specific implementation scenario, the display device can determine the angular velocity of the intelligent device and the auxiliary recognition parameters during air writing based on the attitude sensing parameters. Based on the angular velocity and the auxiliary recognition parameters, determine the movement trajectory corresponding to the angular velocity, and generate the writing trajectory data corresponding to the movement trajectory.

[0058] The auxiliary recognition parameters are used to calculate or determine the movement trajectory. In some embodiments of this application, the auxiliary recognition parameters may be a preset recognition resolution. In addition, the recognition resolution is not related to the screen resolution of the display device. The recognition resolution is an empirical value for assisting in calculating the movement trajectory determined based on the actual air writing environment of the display device and the intelligent device. Further, this recognition resolution can also be called the recognition sensitivity parameter. In the case of triggering an air writing operation, the sampling rate of the intelligent device's sensor components is generally relatively high, such as greater than 100 Hz. In this way, the angular velocity change between adjacent two angular velocity handwriting sampling points is relatively small, and the movement speed and amplitude of natural writing both change relatively small. Therefore, the movement trajectory can be calculated based on the auxiliary recognition parameters of the display device determined by taking a large number of sample data and the actually determined angular velocity. For example, the product of the angular velocity and the auxiliary recognition parameter can be used as the distance of the movement trajectory corresponding to the adjacent two angular velocity handwriting sampling points. In this way, based on the initial sampling point (such as the position of the sampling point at the start of writing), the handwriting writing direction corresponding to the writing, and the distance of the movement trajectory corresponding to the adjacent two angular velocity handwriting sampling points, the movement trajectory corresponding to the adjacent two angular velocity handwriting sampling points can be determined, and thus the writing trajectory data corresponding to the movement trajectory is generated.

[0059] In practical applications, the intelligent device usually corresponds to a sampling rate, such as 200 Hz. The intelligent device measures the angular velocity θ within each sampling time point through the included sensing components, and then based on the angular velocity θ corresponding to each sampling time point, as Figure 2 shown Figure 2This is a schematic diagram of a handwriting sampling scenario involved in this application. Based on a preset sampling rate, at sampling time point t1, the corresponding sampling point in the figure is a1 (the initial sampling point), at sampling time point t2, the corresponding sampling point in the figure is a2... at sampling time point tn, the corresponding sampling point in the figure is an (the end sampling point). Schematically, for the sampling point a2 corresponding to the sampling time point t2, the intelligent device can sense the angular velocity θ2 at the current sampling time point t2, that is, the rotation angle θ2 from the sampling point a1 to the sampling point a2.

[0060] In this way, based on the angular velocity θ and the recognition resolution r or the recognition sensitivity parameter r, the sampling movement displacement x corresponding to the current sampling time point t relative to the previous sampling time point t-1 can be calculated. Taking the angular velocity θ2 at the current sampling time point t2 as an example, the sampling movement displacement x corresponding to the current sampling time point t2 relative to the previous sampling time point 1 can be calculated, that is, the displacement from the sampling point a1 to the sampling point a2;

[0061] That is, x = θ * r; where r can be determined based on the distance R between the intelligent device and the reference writing plane (such as the plane of the display device) and the error adjustment coefficient k in the actual application environment.

[0062] Among them, in the actual environment, when the air writing operation is triggered, the sampling rate of the intelligent device sensor components is usually relatively high, such as greater than 100Hz. In this way, the amplitude change between two adjacent handwriting sampling points is relatively small, and the distance R from the reference writing plane (such as the plane of the display device) is usually within a fixed range (for example, for a TV display device, the distance R between each user and the TV is not much different, and the error in determining the handwriting can be relatively ignored). Therefore, an empirical value can be determined based on a large number of sample data in the actual application environment: the auxiliary recognition parameter of the display device - the recognition resolution r. It can be applied to the scenario where the natural writing amplitude change is not large when the user writes through the intelligent device. The specific measurement can be judged by setting amplitude change parameters, such as setting the angular velocity change threshold, speed change threshold, writing distance threshold, etc.

[0063] In another specific implementation scenario, when identifying the writing trajectory corresponding to the intelligent device within a period of acquisition time based on the attitude sensing parameters, usually based on the sampling movement displacement x and the angular velocity θ between each pair of sampling points, the position of the current sampling point can be determined. For example, the position coordinates of the current sampling point in the coordinate system can be determined. The intelligent device usually corresponds to a sampling rate, such as 200Hz. Sampling starts from the initial sampling moment t1 based on the preset sampling rate. At the sampling moment t1, the corresponding Figure 2The sampling point a1 (the initial sampling point). In practical applications, a coordinate system representing the change of the movement trajectory can be established with the initial sampling point as the coordinate origin. Assuming the sampling point a1 is (0, 0), the next sampling time point t2 corresponding to the sampling time point t1 corresponds to the sampling point a2 in the figure... The sampling time point tn corresponds to the sampling point an (the end sampling point) in the figure. Schematically, the sampling time point t i-1 The corresponding sampling point a i-1 , the intelligent device can sense the angular velocity θi at the current sampling time point ti, that is, the rotation angle θi from the sampling point a i-1 to the sampling point ai. In this way, based on the angular velocity θ and the recognition resolution r or the recognition sensitivity parameter r, the sampling movement displacement x corresponding to the current sampling time point ti relative to the previous sampling time point t i-1 can be calculated. After determining the sampling movement displacement x and the rotation angle θi from the sampling point a i-1 to the sampling point ai, since the coordinates of the previous sampling point a i-1 are known, the coordinates of the current sampling point ai can be determined based on the coordinates of the previous sampling point a i-1 , the rotation angle θi, and the sampling movement displacement x. Assuming the coordinates of a i-1 are (a, b), the coordinates of the current sampling point ai can usually be expressed as (a + xsinθi, b + xcosθi).

[0064] Referring to the foregoing interpretation, assume that the intelligent device has moved a displacement S in the air and the time taken is T. This distance may not be a regular straight line but may be a relatively curved line, corresponding to the strokes of numbers, letters, and Chinese characters, etc. According to the principle of calculus, if the set sampling time is smaller, the trajectory of S is more likely to approach a straight line. As long as the time T can meet the required accuracy, such as 0.1 second, within this 0.1 second, based mainly on the angular velocity collected by the intelligent device, the display device can determine that its movement trajectory can basically be regarded as a linear motion. The display device can simply determine the displacement x between the current sampling time point and the previous sampling time point based on the angular velocity and the auxiliary recognition parameter (such as the recognition resolution) at each sampling time point. In this way, the displacement trajectory between two adjacent sampling points of the intelligent device can be conveniently determined. Then, during a long period of time of writing in the air, the trajectory curve of the movement of the intelligent device (which can be regarded as the trajectory of the user's writing in the air) is divided into several very small parts according to the principle of calculus. The sum of all these very small displacements is the movement trajectory of the intelligent device. That is, the entire process can be based on the coordinates of the previous sampling point a i-1 , the rotation angle θi, and the sampling movement displacement x to sequentially determine the coordinates of the current sampling point ai. When the coordinates of all sampling points are connected pairwise according to the sampling sequence, a movement trajectory will be obtained within the sampling time, that is, the aforementioned movement trajectory data,

[0065] Optionally, in some embodiments, the particularity of writing situations is taken into account, such as the particularity of characters. For example, Chinese characters have many independent strokes of the same character, and different Chinese characters also need to be relatively independent. When the user controls the movement of the smart device, the movement trajectory of the smart device is continuous. If the motion trajectory of the smart device is not treated differently during the sampling process, the motion trajectory image transmitted to the display device for recognition will be connected, which will make it very difficult for the display device to recognize it, and the speed and efficiency of recognition will be reduced. Based on this, a drop signal corresponding to the lifting state of the air-written stroke (the handwriting after the stroke is lifted does not fall into the reference) and a lift signal corresponding to the falling state (the handwriting after the stroke falls must fall into the reference) can be set. The smart device can be provided with interval operation instructions and can be based on the interactive object control to give the writing drop signal and lift signal. The interactive object control can be a physical button, or it can be a single click, double click, long press and other actions on the touch screen, or it can be a virtual button area on the touch screen of the smart device, or it can be distinguished by setting a time threshold, such as Figure 3 As shown, Figure 3 It is a schematic diagram of a scenario in which strokes are dropped and lifted to separate them involved in the present application; when a user needs to input a character (such as when inputting in cursive script), the strokes of the character are connected. After each stroke is input, releasing the key for a short time is considered to be an input lift signal (the duration of the key being released is less than the threshold value T0). The smart default is that the key release at this time is only a transition between different strokes of the same character, and it is not the completion of a character input. When the smart device collects data, it only needs to mark the time point of the input lift signal. In this way, when transmitting to the display device, the display device can intelligently clear the handwriting between the lift signal and the next drop signal corresponding to the lift signal. Schematically, Figure 3 For example, the display device recognizes the gesture sensing data, and the recognized moving track is roughly the character "王".

[0066] Step S103: input the writing trajectory data into a writing recognition model, and output a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0067] In some embodiments, the handwriting recognition model may be a neural network model. Further, the handwriting recognition model may be obtained by transfer training based on the third handwriting data of the air medium and the fourth handwriting data of the plane medium, that is, the handwriting recognition module may be obtained by creatively adopting a transfer learning method to train the handwriting recognition model in combination with the actual application environment and taking into account the small number of three-dimensional air handwriting training sets.

[0068] The described writing recognition model can be obtained through transfer learning-based model conceptions using a large amount of third writing data based on the air medium and fourth writing data based on the flat medium for transfer training. For example, the writing recognition model can be implemented by one or more of a Logistic Regression (LR) model, a Support Vector Machine (SVM), a decision tree, a Naive Bayes classifier, a Convolutional Neural Network (CNN), a Recurrent Neural Networks (RNN), etc. In some embodiments, an initial writing recognition model can be trained based on sample data with already annotated handwriting labels (such as standard characters, standard patterns) to obtain a trained writing recognition model. Among them, the specific model training process can refer to other embodiments of this application.

[0069] Among them, the air medium is also the air writing medium. Generally, when a user writes on the air medium in the three-dimensional space of a smart device held, in short, when the user operates the smart device in the air to write sample handwriting (characters, English, numbers, strokes, gestures, etc.), the corresponding sample data is the writing trajectory data.

[0070] The flat medium, that is, the flat writing medium, such as when a user can directly write sample handwriting (characters, English, numbers, strokes, etc.) on the screen of a device such as a mobile phone, a television, a tablet, etc., the corresponding sample data is the writing recognition result. It can be understood that when writing based on the flat medium, the recognition of the writing handwriting is relatively simple compared to the air medium, and the sample data is usually massive; while for the recognition of air writing handwriting, it is usually necessary to map the posture data of air writing to a two-dimensional plane to generate the writing handwriting on the two-dimensional plane. There will usually be a certain error when converting from three dimensions to two dimensions, and such sample data is less.

[0071] In practical applications, for the air writing recognition based on the posture sensing data collected between the sensors of the smart device, only converting it to the writing trajectory data on the two-dimensional plane, that is, restoring the posture sensing data into the corresponding plane trajectory, however, their readability is still very poor. It is difficult to identify the specific writing content from the waveform just by human eye observation or machine recognition. Therefore, in this application, there is a pre-trained writing recognition model based on transfer training. The trained writing recognition model can effectively recognize the writing trajectory data. In practical applications, only by inputting the writing trajectory data into the writing recognition model, the writing recognition result can be output. The readability and visibility of the writing recognition result are higher than those of the writing trajectory data.

[0072] In a specific implementation scenario, the display device can perform air writing trajectory recognition by the following steps:

[0073] Step S1001: Receive the attitude sensing parameters sent by the intelligent device, where the attitude sensing parameters are generated by the intelligent device when it senses an air writing operation.

[0074] For details, refer to step S101, which will not be elaborated here.

[0075] Step S1002: Obtain the angular velocity of the intelligent device during air writing from the attitude sensing parameters and obtain a first auxiliary recognition parameter, where the first auxiliary recognition parameter includes a preset recognition resolution.

[0076] In some actual usage scenarios, when the user writes through the intelligent device, the natural writing amplitude changes little. At this time, the display device obtains the angular velocity included in the attitude sensing parameters and obtains an auxiliary recognition parameter, that is, the preset recognition resolution. In addition, the recognition resolution has nothing to do with the screen resolution of the display device. The recognition resolution is mainly an empirical value for assisting in calculating the movement trajectory determined based on the actual air writing environment between the display device and the intelligent device. Further, this recognition resolution can also be called the recognition sensitivity parameter. When an air writing operation is triggered, the sampling rate of the intelligent device's sensor components is generally high, such as greater than 100 Hz. In this way, the angular velocity change between adjacent two angular velocity handwriting sampling points is small, and both the movement speed and amplitude of natural writing change little. Therefore, the movement trajectory can be calculated based on the recognition resolution of the display device determined by taking a large number of sample data and the actually determined angular velocity. For example, the product of the angular velocity and the recognition resolution can be used as the distance of the movement trajectory corresponding to adjacent two angular velocity handwriting sampling points. In this way, based on the initial sampling point (such as the position of the sampling point at the start of writing), the handwriting direction corresponding to the writing, and the distance of the movement trajectory corresponding to adjacent two angular velocity handwriting sampling points, the movement trajectory corresponding to adjacent two angular velocity handwriting sampling points can be determined, thereby generating the writing trajectory data corresponding to the movement trajectory.

[0077] Step S1003: Use the product of the angular velocity and the recognition resolution as the first instantaneous displacement of movement, and determine the first movement trajectory based on the first instantaneous displacement of movement.

[0078] In an actual environment, when an in-air writing operation is triggered, the sampling rate of the sensor components of the intelligent device is generally relatively high, such as greater than 100 Hz. In this case, the amplitude change between two adjacent handwriting sampling points is small, and the distance R from the reference writing plane (such as the plane of the display device) is usually within a fixed range (for example, for a TV display device, the distance R between each user and the TV is not much different, and the error in determining the handwriting can be relatively ignored). Therefore, an empirical value can be determined based on a large number of sample data in the actual application environment: the auxiliary recognition parameter of the display device - the recognition resolution r. This can be applied to scenarios where the natural writing amplitude change is not significant when the user writes through the intelligent device. Based on this concept, the time for calculating the writing trajectory can be significantly saved when determining the in-air writing trajectory. Among them, the technical feature of "using the product of the angular velocity and the recognition resolution as the first instantaneous displacement of movement" and the derivation principle of the recognition resolution can refer to other embodiments of this application.

[0079] Step S1004: Input the writing trajectory data into the writing recognition model, and output a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0080] For details, please refer to step S103, which will not be elaborated here.

[0081] In a specific implementation scenario, the display device can perform in-air writing trajectory recognition by the following steps:

[0082] Step S2001: Receive the attitude sensing parameters sent by the intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an in-air writing operation.

[0083] For details, please refer to step S101, which will not be elaborated here.

[0084] Step S2002: Obtain the angular velocity of the intelligent device during in-air writing and the second auxiliary recognition parameter from the attitude sensing parameters. The second auxiliary recognition parameter includes the normal angle and the distance from the intelligent device. The normal angle is the normal angle of the intelligent device relative to the display device.

[0085] According to some embodiments, usually in such an extended writing trajectory recognition scenario, it is possible to accurately quantify the writing amplitude when the user writes through a smart device. At this time, the display device can obtain the angular velocity included in the attitude sensing parameters as the corresponding change parameter and the second auxiliary recognition parameter when the smart device writes in the air. Among them, in some embodiments, the second auxiliary recognition parameter obtained by the display device can be from the attitude sensing parameters, that is, when the smart device collects the angular velocity corresponding to the air writing operation, it also collects the normal angle (the normal angle α of the smart device relative to the reference two-dimensional plane (such as the display device)) and the distance R between the smart device and the reference two-dimensional plane (such as the display device). As Figure 5 shown, Figure 5 is a schematic diagram of parameters in the air writing scenario involved in this application. The relationship between the parameters is as shown in the figure, and the angular velocity is θ. In some embodiments, the second auxiliary recognition parameter obtained by the display device can be measured by the sensor devices (infrared devices, ranging devices, etc.) included in the display device.

[0086] Schematically, the distance between the smart device and the display device is R, the angle between the direction of the smart device and the normal line of the display device is α, and the instantaneous rotation angle of the interval device at a certain sampling moment, that is, the angular velocity, is θ.

[0087] Step S2003: Input the angular velocity, the normal angle, and the distance into the target instantaneous displacement formula, output the second moving instantaneous displacement, and determine the second moving trajectory based on the second moving instantaneous displacement.

[0088] Taking Figure 5 as an example, the moving distance x of the moving instantaneous displacement corresponding to each sampling time point on the reference two-dimensional plane is calculated as:

[0089] x = Rtan(α + θ) - Rtanα

[0090]

[0091] In this way, the target instantaneous displacement formula is obtained.

[0092]

[0093] Among them, x is the second moving instantaneous displacement, R is the distance, α is the normal angle, and θ is the angular velocity.

[0094] Based on the above target instantaneous displacement formula, input the angular velocity, the normal angle, and the interval distance into the target instantaneous displacement formula. In practical applications, input the angular velocity value, the normal angle, and the interval distance at each sampling time point corresponding to the sampling period of the display device into the target instantaneous displacement formula, and output a small segment of displacement trajectory x corresponding to each sampling time point. Then, based on the angular velocity, the direction of "a small segment of displacement trajectory x" (i.e., the second moving instantaneous displacement) can be determined. Then, based on the determined direction and "a small segment of displacement trajectory x" corresponding to each sampling time point, accumulate them in sequence to obtain the second moving trajectory.

[0095] Step S1004: Input the writing trajectory data into the writing recognition model, and output a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0096] Specifically, refer to step S103, which will not be elaborated here.

[0097] In the embodiment of the present application, the display device can receive the attitude sensing parameters sent by the intelligent device, and the attitude sensing parameters are generated by the intelligent device when sensing an air writing operation; then generate writing trajectory data based on the attitude sensing parameters; then input the writing trajectory data into the writing recognition model to output an accurate writing recognition result. In addition, the writing recognition model is obtained by transfer training based on the third writing data of the air medium and the fourth writing data of the plane medium. Based on the foregoing method, the problems of low accuracy and low recognition processing efficiency of air writing trajectory recognition in the related art can be avoided, and the convenience of cross-device collaborative control is improved; and the recognition of writing data can be realized only based on the attitude sensing data collected by the intelligent device. When writing recognition is performed, it is not necessary to collect the three-dimensional data of the intelligent device nor map the three-dimensional data to two dimensions for recognition, which greatly reduces the calculation amount of air writing recognition and improves the recognition processing efficiency; and in the air writing recognition stage, a writing recognition model based on transfer learning is used. A general pre-trained model with general performance can be obtained through existing relevant data (such as the fourth writing data of the plane medium), and then the model is fine-tuned with a small amount of domain data to obtain a model with better matching for the current data type. It also improves the robustness of the model, and only a small amount of air writing data is required to ensure the accuracy of air writing trajectory recognition and improve the recognition processing efficiency.

[0098] Please refer to Figure 4 , Figure 4 is a schematic flowchart of another embodiment of an air writing trajectory recognition method proposed by the present application. Specifically:

[0099] Step S201: Receive the attitude sensing parameters sent by the intelligent device, and the attitude sensing parameters are generated by the intelligent device when sensing an air writing operation.

[0100] For details, refer to step S101, which will not be elaborated here.

[0101] Step S202: Based on the posture sensing parameters and the preset posture parameter thresholds, determine the writing type of the air writing operation.

[0102] Among them, the posture parameter thresholds are used to measure the degree of writing change when the user manipulates the intelligent device to write, so as to determine the writing type of the air writing operation, and then adopt the handwriting determination methods corresponding to different writing types to calculate or determine the movement trajectory corresponding to the posture sensing parameters.

[0103] Among them, the first writing type can be understood as a scenario where the natural writing amplitude changes little when the user writes through the intelligent device. When it is the first writing type, the preset recognition resolution and the angular velocity parameter in the posture sensing parameters can be used to determine the movement trajectory;

[0104] Among them, the second writing type can be understood as a scenario where the natural writing amplitude changes greatly when the user writes through the intelligent device. When it is the second writing type, it is only calculated with a large error through the preset recognition resolution. At this time, it is necessary to determine the movement trajectory based on the second auxiliary recognition parameter and the angular velocity. The second auxiliary recognition parameter includes the normal angle and the distance from the intelligent device.

[0105] Schematically, the type of the posture sensing parameters can be various reference posture parameters (speed, acceleration, distance, etc.) involved in the above embodiments. The posture sensing parameters only need to include the angular velocity at least. The posture parameter thresholds are the threshold values or critical values of the reference posture parameters used to measure the writing type, and the number can be multiple. When the reference posture parameter (at least one of speed, acceleration, distance, etc.) is greater than the posture parameter threshold of the corresponding type of the reference posture parameter, it is considered that the natural writing amplitude changes greatly when the user writes through the intelligent device. At this time, the air writing operation is determined to be the second writing type; on the contrary, when the reference posture parameter (at least one of speed, acceleration, distance, etc.) is less than or equal to the posture parameter threshold of the corresponding type of the reference posture parameter, it is considered that the natural writing amplitude changes little when the user writes through the intelligent device. At this time, the air writing operation is determined to be the first writing type.

[0106] Step S203: When the writing type is the second writing type, obtain the angular velocity included in the posture sensing parameters and the second auxiliary recognition parameter. The second auxiliary recognition parameter includes the normal angle and the distance from the intelligent device. The normal angle is the normal angle of the intelligent device relative to the display device.

[0107] Specifically, when the writing type is the second writing type, usually in this scenario, when the user writes through a smart device, the natural writing amplitude changes greatly. At this time, the display device obtains the angular velocity and the second auxiliary recognition parameter included in the attitude sensing parameter. Among them, in some embodiments, the second auxiliary recognition parameter obtained by the display device can be obtained from the attitude sensing parameter, that is, when the smart device collects the angular velocity corresponding to the air writing operation, it also collects the normal angle (the normal angle α of the smart device relative to the reference two-dimensional plane (such as the display device)) and the distance R between the smart device and the reference two-dimensional plane (such as the display device). As Figure 5 shown, Figure 5 is a schematic diagram of parameters in the air writing scenario involved in the present application. The relationship between the parameters is as shown in the figure, and the angular velocity is θ. In some embodiments, the second auxiliary recognition parameter obtained by the display device can be calculated by the display device through the included sensor devices (infrared devices, ranging devices, etc.).

[0108] Schematically, the distance between the smart device and the display device is R, the angle between the direction of the smart device and the normal of the display device is α, and the instantaneous rotation angle of the interval device at a certain sampling moment, that is, the angular velocity, is θ.

[0109] Step S204: Input the angular velocity, the normal angle, and the interval distance into the target instantaneous displacement formula, output the second moving instantaneous displacement, and determine the second moving trajectory based on the second moving instantaneous displacement.

[0110] In practical applications, the smart device usually corresponds to a sampling rate, such as 200 Hz. The smart device calculates the angular velocity θ within each sampling time point through the included sensing devices, and then based on the angular velocity θ corresponding to each sampling time point, as Figure 6 shown, Figure 6This is a schematic diagram of a handwriting sampling scenario involved in the present application. Based on a preset sampling rate, at sampling time point t1, it corresponds to sampling point a1 (sampling starting point) in the figure, at sampling time point t2, it corresponds to sampling point a2 in the figure... at sampling time point tn, it corresponds to sampling point an (sampling ending point) in the figure. Schematically, for the sampling point a2 corresponding to the sampling time point t2, the intelligent device can sense the angular velocity θ2 at the current sampling time point t2, that is, the rotation angle θ2 from sampling point a1 to sampling point a2. Additionally, assume that the intelligent device has moved a displacement S in the air over a time T. This distance may not be a regular straight line but may be a relatively curved line, corresponding to the strokes of numbers, letters, and Chinese characters, etc. According to the principle of calculus, if the set sampling time is smaller, the displacement trajectory of S is more likely to approach a straight line. As long as the time T can meet the required accuracy, such as 0.1 seconds, within this 0.1 seconds, based mainly on the angular velocity collected by the intelligent device, the display device can determine that its movement trajectory can basically be regarded as a linear motion, and the display device can simply calculate this trajectory S of the intelligent device based on the angular velocity and auxiliary recognition parameters (such as recognition resolution). Then, during a long period of in-air writing time, the curve of the intelligent movement trajectory (which can be regarded as the user's in-air writing trajectory) is divided into several very small parts according to the principle of calculus, and the sum of all several very small displacements x is the movement trajectory of the intelligent device.

[0111] Therefore, the key to determining the movement trajectory lies in determining the instantaneous displacement x of the movement.

[0112] Take Figure 5 as an example. The moving distance x of the instantaneous displacement of the movement corresponding to each sampling time point on the reference two-dimensional plane is calculated as:

[0113] x = Rtan(α + θ) - Rtanα

[0114]

[0115] In this way, the target instantaneous displacement formula is obtained.

[0116]

[0117] Among them, x is the second instantaneous displacement of the movement, R is the interval distance, α is the normal angle, and θ is the angular velocity.

[0118] Based on the above target instantaneous displacement formula, input the angular velocity, the normal angle, and the interval distance into the target instantaneous displacement formula. In actual applications, input the angular velocity values, the normal angle, and the interval distance at each sampling time point corresponding to the sampling period of the display device into the target instantaneous displacement formula, and output a small segment of displacement trajectory x corresponding to each sampling time point. And based on the angular velocity, the direction of "a small segment of displacement trajectory x" (i.e., the second moving instantaneous displacement) can be determined. Then, accumulate according to the sequence based on the determined direction and "a small segment of displacement trajectory x" corresponding to each sampling time point, which is the second moving trajectory.

[0119] Step S205: When the writing type is the first writing type, obtain the angular velocity included in the attitude sensing parameter and the first auxiliary recognition parameter, where the first auxiliary recognition parameter includes a preset recognition resolution.

[0120] Step S206: Multiply the angular velocity by the recognition resolution as the first moving instantaneous displacement, and determine the first moving trajectory based on the first moving instantaneous displacement.

[0121] Specifically, when the writing type is the first writing type, usually in this scenario, when the user writes through the intelligent device, the natural writing amplitude changes little. At this time, it is sufficient for the display device to obtain the angular velocity included in the attitude sensing parameter.

[0122] In some embodiments of the present application, the auxiliary recognition parameter may be a preset recognition resolution. Additionally, the recognition resolution has nothing to do with the screen resolution of the display device. The recognition resolution is an empirical value for assisting in calculating the moving trajectory based on the actual air writing environment between the display device and the intelligent device. Further, this recognition resolution can also be called the recognition sensitivity parameter. In the case of triggering an air writing operation, generally, the sampling rate of the intelligent device sensor is relatively high, such as greater than 100 Hz. In this way, the angular velocity change between adjacent two angular velocity handwriting sampling points is small, and both the action speed and amplitude of natural writing change little. Therefore, the moving trajectory can be calculated based on the recognition resolution of the display device determined by taking a large number of sample data and the actually determined angular velocity. For example, the product of the angular velocity and the recognition resolution can be used as the distance of the moving trajectory corresponding to the adjacent two angular velocity handwriting sampling points. In this way, based on the initial sampling point (such as the position of the sampling point at the start of writing), the handwriting writing direction corresponding to writing, and the distance of the moving trajectory corresponding to the adjacent two angular velocity handwriting sampling points, the moving trajectory corresponding to the adjacent two angular velocity handwriting sampling points can be determined, thereby generating the writing trajectory data corresponding to the moving trajectory.

[0123] In practical applications, when the writing type is the first writing type, it is actually based on the approximate equivalence of angular velocity in the derivation process of the above target instantaneous displacement formula. And when the writing type is the first writing type, the approximate determination of the movement trajectory in this way has relatively little error. Specifically as follows,

[0124] Taking Figure 5 as an example, the moving distance x of the moving instantaneous displacement corresponding to each sampling time point on the reference two-dimensional plane is calculated as:

[0125] x = Rtan(α + θ) - Rtanα

[0126] ≈ k(R(α + θ) - Rα)

[0127] = kRθ

[0128] = rθ

[0129] In the actual environment, when the in-air writing operation is triggered, the sampling rate of the intelligent device sensor is usually relatively high, such as greater than 100Hz. In this way, the amplitude change between adjacent two handwriting sampling points is small, and the distance R relative to the reference writing plane (such as the plane of the display device) is usually within a fixed range (for example, for a display device TV, the distance R between each user and the TV is not much different, and this error can be relatively ignored when determining the handwriting). Therefore, an empirical value can be determined based on a large number of sample data in the actual application environment: the auxiliary recognition parameter of the display device - the recognition resolution r. It can be applied to the scenario where the natural writing amplitude change is not large when the user writes through the intelligent device. Based on this concept, when determining the in-air writing trajectory, the time for calculating the trajectory can be greatly saved. At the same time, after generating the writing trajectory data, in order to improve the accuracy of trajectory recognition, it is also input into the writing recognition model to output the writing recognition result.

[0130] Step S207: Generate the writing trajectory data corresponding to the movement trajectory, and perform trajectory style adjustment processing on the writing trajectory data to obtain the adjusted target writing trajectory data. Wherein the movement trajectory is the above first movement trajectory or the second movement trajectory.

[0131] Among them, the generation of the writing trajectory data corresponding to the movement trajectory can refer to step S102, which will not be elaborated here.

[0132] In a specific implementation manner, after generating the writing trajectory data corresponding to the movement trajectory, due to objective factors such as the writing style and writing mood when the user manipulates the intelligent device, in order to further improve the visibility of the writing data, the trajectory style adjustment processing can also be performed on the writing trajectory data. In addition, based on the adjusted writing data input into the writing recognition model, the accurate recognition of the writing recognition result can be improved.

[0133] Since writing based on a three-dimensional air medium is quite different from writing on a two-dimensional planar medium, there are also significant differences in the writing result writing trajectory data. For example, Figure 6 , the left figure shows handwritten characters on paper, and the right figure shows the air writing result recognized by the display device. It can be seen that due to differences in writing tools, media, and user styles, handwritten data will show variations in color shade, character size, and stroke thickness; while for characters displayed by an electronic media display device, their stroke thickness and color are unified. Directly using the air writing result recognized by the display device usually has a high probability of poor readability and visibility of the writing trajectory data. Therefore, style unification can be carried out, and the writing trajectory data can be processed by adjusting the trajectory style, which can be:

[0134] 1. Perform character detection processing on the grayscale image corresponding to the writing trajectory data to determine the target trajectory area;

[0135] In practical applications, binary conversion is first performed: converting the grayscale image corresponding to the original writing trajectory data into a binary image. One way can be to set the grayscale value of the pixel points of the original character image as, and the conversion method is:

[0136]

[0137] where τ is the conversion threshold, which can be custom-set, such as τ = 230.

[0138] After binary conversion, then detect the effective rectangular area of the moving trajectory (such as characters, numbers) in the writing trajectory data, that is, the target trajectory area. The target trajectory area can just accommodate the moving trajectory (such as characters, numbers, etc.), and the blank area is cropped and deleted to determine the target trajectory area.

[0139] 2. Perform trajectory scaling processing on the target trajectory area to generate target writing trajectory data of a preset specification. In practical applications, taking the trajectory as the character type as an example, scale the above target trajectory area proportionally to the pixel size of the preset specification, such as 192×192 pixel size, and then supplement blank areas on the periphery, such as supplementing corresponding pixel lengths of blanks above, below, left, and right of the target trajectory area, to form a handwritten character image of a unified size (256×256) of the preset specification, that is, the first target writing data.

[0140] Optionally, before performing trajectory scaling processing on the target trajectory area, the reference supplementary full line corresponding to the target trajectory area (which can be the longest character side of the target trajectory area) can also be determined, and with the reference supplementary full line as a reference, adjust the target trajectory area to a preset shape. Schematically, the target trajectory area (such as the character effective rectangular area) can be detected, and with the longer side as the reference, keep the character centered and supplement the shorter side to the length of the longer side to form a square area.

[0141] Then, perform trajectory scaling processing on the target trajectory area of the preset shape to generate target writing trajectory data of a preset specification. Scale the target trajectory area of the above-mentioned preset shape (such as a square shape) to the pixel size of the preset specification, such as 192×192 pixel size, and then supplement blank areas on the peripheral sides. For example, supplement blanks with corresponding pixel lengths above, below, left, and right of the target trajectory area to form a handwritten character picture of the preset specification with a unified size (256×256), that is, the first target writing data, such as Figure 7 shown Figure 7 The left figure is the intercepted target trajectory area. Then, based on the longer side as the benchmark, keep the character centered and complete the shorter side to the length of the longer side to form a target trajectory area in the shape of a square.

[0142] Step S208: Input the target writing trajectory data into the writing recognition model and output the writing recognition result.

[0143] Specifically, refer to step S103, which will not be elaborated here.

[0144] Step S209: Display at least one standard writing object corresponding to the writing recognition result.

[0145] The standard writing object can be at least a standard writing Chinese character, standard writing character, standard writing number, standard writing pattern, etc. corresponding to the writing recognition result.

[0146] In a specific implementation scenario, the display device can provide at least one standard writing object determined based on the air writing trajectory for display. The display device can provide an input display box for the smart device. During the air writing process, the display device can display at least one standard writing object corresponding to the recognized writing recognition result, such as at least one standard writing character, in the input display box in real time, such as Figure 8 shown Figure 8 is a scenario usage diagram for displaying the writing recognition result provided by this application. In Figure 8 the display device can be a smart TV or a smart display screen. The user carries a smart device such as a wearable device to perform air writing operations. The smart device such as a wearable device collects attitude sensing data corresponding to the air writing operations and sends it to the display device (such as a smart TV). The smart TV determines the writing recognition result based on the attitude sensing data and displays at least one standard writing character corresponding to the writing recognition result in Figure 8The writing character display area shown; alternatively, based on the recognition result of the writing recognition result, the display device determines multiple standard writing objects with high probabilities, such as candidate options for high-probability character recognition results. Further, the user can control the cursor within the display area on the display device through a mobile intelligent device such as a wearable device, and combine interactive controls (such as buttons) to confirm the selection of the recognition result. If the standard writing object desired by the user is not included in the current candidate options, the user can also control and re-enter the recognition through the control, and so on.

[0147] In the embodiment of the present application, the display device can receive the attitude sensing parameters sent by the intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an air writing operation; then generate writing trajectory data based on the attitude sensing parameters; and then input the writing trajectory data into the writing recognition model to output an accurate writing recognition result. In addition, the writing recognition model is obtained by transfer training based on the third writing data of the air medium and the fourth writing data of the planar medium. Based on the foregoing method, the problems of low accuracy and low recognition processing efficiency of the air writing trajectory recognition in the related art can be avoided, and the convenience of cross-device collaborative control is improved; and the recognition of the writing data can be realized only based on the attitude sensing data collected by the intelligent device. When performing writing recognition, it is not necessary to collect the three-dimensional data of the intelligent device nor map the three-dimensional data to two dimensions for recognition, which greatly reduces the calculation amount of air writing recognition and improves the recognition processing efficiency; and in the air writing recognition stage, a writing recognition model based on transfer learning is adopted. A general pre-trained model with average performance can be obtained through existing relevant data (such as the fourth writing data of the planar medium), and then the model is fine-tuned using a small amount of domain data to obtain a model with better matching for the current data type. It also improves the robustness of the model, and only a small amount of air writing data is required to ensure the accuracy of air writing trajectory recognition and improve the recognition processing efficiency.

[0148] Please refer to Figure 9 , Figure 9 which is a schematic flowchart of another embodiment of an air writing trajectory recognition method proposed by the present application. Specifically:

[0149] Step S301: Obtain the third writing data based on the air medium and the fourth writing data of the planar medium.

[0150] In some embodiments, the aerial medium, i.e., the aerial writing medium, generally refers to the sample data, i.e., the third trajectory data, corresponding to the writing of a sample handwriting (characters, English, numbers, strokes, gestures, etc.) in the aerial medium in three-dimensional space by a smart device held by a user. Further, generally, the amount of data of the third writing data based on the aerial medium is small. In this application, the third writing data can be generated based on the samples collected by the smart device in the corresponding experimental environment under the actual application environment, and standard object labels (such as standard characters, labeled numbers, labeled patterns, etc.) can be marked for the third writing data.

[0151] The planar medium, i.e., the planar writing medium, refers to the sample data, i.e., the fourth writing data, corresponding to the writing of a sample handwriting (characters, English, numbers, strokes, etc.) directly on the screens of devices such as mobile phones, televisions, and tablets by a user. It can be understood that when writing based on the planar medium, the recognition of the writing handwriting is relatively simple compared to the aerial medium, and the sample data is usually massive. For the recognition of aerial writing handwriting, it is usually necessary to map the posture data of the aerial writing to a two-dimensional plane to generate the writing handwriting on the two-dimensional plane. There will usually be a certain error when converting from three dimensions to two dimensions, and this kind of sample data is less. In the two-dimensional plane, especially the handwritten data set of paper media is relatively complete, and the fourth writing data based on the planar medium can be obtained from the corresponding public data sets.

[0152] In this application, the model training based only on the writing trajectory data of the aerial writing medium. Usually, the handwriting recognition scheme needs to be driven by a large amount of data. The deep neural network is trained by learning a large number of writing results with result labels, and then the recognition is completed through the network. For aerial writing, the same large amount of this type of data is also required. Taking the GB2312 common Chinese character list as an example, there are approximately 7,000 common characters in total. If each Chinese character requires 200 different samples, a total of 1.4 million samples are required. Constructing an aerial writing data set of this scale requires a large amount of work. In the two-dimensional plane, especially the handwritten data set of paper media is relatively complete. Therefore, using the handwritten data set to train a model that can perform aerial writing recognition can greatly reduce the work overhead of system implementation.

[0153] In a specific implementation manner, before model training, the obtained sample data can also be processed by adjusting the trajectory style. That is, the third sample data based on the aerial medium can be obtained, and then the first sample data is processed by adjusting the trajectory style to obtain the adjusted third writing data. And, the fourth sample data based on the planar medium can be obtained, and the fourth sample data is processed by adjusting the trajectory style to obtain the adjusted fourth writing data.

[0154] Among them, the processing of adjusting the trajectory style can refer to step S206, which is similar to the processing process of the writing trajectory data. In practical applications, the grayscale image corresponding to the writing data is subjected to trajectory detection processing to determine the target trajectory area; the reference supplementary full line corresponding to the target trajectory area is determined, and the target trajectory area is adjusted to a preset shape with reference to the reference supplementary full line. The target trajectory area is subjected to trajectory scaling processing to generate target writing data of a preset specification.

[0155] Step S302: Input the fourth writing data into the neural network model for training to generate an initial writing recognition model.

[0156] In this application, it is creatively considered that the training model of the writing trajectory data based only on the air writing medium usually requires a large amount of data for the handwritten recognition scheme. The neural network is trained by learning a large number of writing results with result tags, and then the recognition is completed through the network. For air writing, the same large amount of this type of data is also required. Taking the GB2312 common Chinese character list as an example, there are approximately 7,000 common Chinese characters in total. If each Chinese character requires 200 different samples, a total of 1.4 million samples are required. Constructing an air writing dataset of this scale requires a lot of work. In the two-dimensional plane, especially the handwritten dataset on paper media is relatively complete. Therefore, using the handwritten dataset to train and obtain an initial writing recognition model that can perform writing recognition, and then inputting the third writing data based on the air medium into the initial writing recognition model to obtain a trained writing recognition model can greatly reduce the work overhead of system implementation.

[0157] The transfer learning refers to applying the knowledge or patterns learned in a certain field or task to different but related fields or tasks. In this embodiment, the fourth writing data based on the planar medium is usually massive, while the third writing data based on the air medium is less than the fourth writing data. The fourth writing data can be called the source data in transfer learning, and the third writing data can be called the target data. The user terminal first inputs the source data - the fourth writing data into the neural network model for training to obtain a trained initial writing recognition model, and then inputs the target data - the fourth writing data into the initial writing recognition model to train the initial writing recognition model and adjust the parameters of the initial writing recognition model, and then generates a writing recognition model to achieve a better fitting effect.

[0158] The neural network model can be a fitting implementation based on one or more of the following models: Convolutional Neural Network (CNN) model, Deep Neural Network (DNN) model, Recurrent Neural Networks (RNN) model, embedding model, Gradient Boosting Decision Tree (GBDT) model, Logistic Regression (LR) model, etc. In this embodiment, a deep neural network can be adopted, and the error backpropagation algorithm is introduced to optimize the existing neural network model. According to the loss function between the actually output recognition result and the true label, backpropagation (BP) is performed, and the network parameters are iteratively adjusted to improve the performance, thereby improving the recognition accuracy of the neural network model.

[0159] In a feasible implementation manner, the fourth writing data is divided into at least one first data training set and a first data test set;

[0160] Among them, the first data training set is used to train the neural network model, and the first data test set is used to verify the recognition effect of the neural network model during the training phase. Further, the number division of the data training set and the data test set can be in a certain ratio, such as 7:3, etc., and can be specifically determined according to the actual application environment.

[0161] Train the neural network model based on the current first data training set, and use the first data test set to test the first recognition accuracy of the neural network model;

[0162] Illustratively, the neural network model can be composed of at least an input layer, a hidden layer, and an output layer. The input layer is used to extract writing features according to the first writing training set input to the deep neural network, and calculate the output value input to the hidden layer unit at the bottom layer based on the writing features. The input layer usually includes multiple input units, and the input units are used to calculate the output value input to the hidden layer unit at the bottom layer according to the input writing features. After the speech features are input to the input unit, the input unit calculates the output value output to the bottom hidden layer according to its own weighting value and the writing features input to the input unit.

[0163] There are usually multiple hidden layers, and each hidden layer includes multiple hidden units. The hidden units receive the input values from the hidden units in the next hidden layer. The input values from the hidden units in the next hidden layer are weighted and summed according to the weight values of this layer, and the result of the weighted sum is used as the output value output to the hidden units in the upper hidden layer.

[0164] The output layer includes multiple output units. The output units receive the input values from the hidden units in the topmost hidden layer, perform weighted summation on the input values from the hidden units in the topmost hidden layer according to the weight values of this layer, calculate the actual output value based on the result of the weighted summation, and backpropagate from the output layer based on the error between the expected output value and the actual output value and adjust the connection weight values and thresholds of each layer along the output path.

[0165] Specifically, in this embodiment, a DNN-HMM model introducing the error backpropagation algorithm can be used to create an initial model. After extracting the writing features of the fourth writing data, the writing features are input into the neural network model. The training process of the neural network model usually consists of two parts: forward propagation and backpropagation. In the forward propagation process, the electronic device inputs the sample fourth writing data. After passing through the transfer function (also known as the activation function or transformation function) of the hidden layer neurons (also called nodes) in the neural network model, it is transmitted to the output layer. The state of each layer of neurons affects the state of the next layer of neurons. In the output layer, the actual output value - the recognized writing result is calculated, and the expected error between the actual output value and the expected output value is calculated. Based on the expected error, the parameters of the neural network model are adjusted. The parameters include the weight values and thresholds of each layer. After the dataset training is completed, the first recognition accuracy of the neural network model is tested using the first data test set.

[0166] Determination of the first recognition accuracy: Input the first data test set into the neural network model, and calculate the correct rate according to the recognized result actually output and the labeled true label corresponding to the first data test set, that is, the first recognition accuracy.

[0167] If the first recognition accuracy is less than the first preset threshold, obtain the next first data training set of the current first data training set, use the next first data training set as the current first data training set, and execute the step of training the neural network model based on the current first data training set;

[0168] If the first recognition accuracy is greater than or equal to the first preset threshold, it is considered that the expected recognition effect is achieved, then the training is stopped, the parameters of the current neural network model are saved, and the neural network model is used as the initial writing recognition model.

[0169] In addition, in some embodiments, the number of training rounds of the neural network model can be set for the dataset to determine the training stop.

[0170] Step S303: Input the third writing data into the initial writing recognition model for transfer learning to generate a writing recognition model.

[0171] In a feasible embodiment, the third writing data is divided into at least one second data training set and a second data test set;

[0172] Among them, the second data training set is used to train the neural network model, and the second data test set is used to verify the recognition effect of the neural network model during the training phase. Further, the number division of the data training set and the data test set can be in a certain ratio, such as 7:3, etc., and can be specifically determined according to the actual application environment.

[0173] Then, based on the current second data training set, the initial writing recognition model is trained, and the second data test set is used to test the second recognition accuracy of the initial writing recognition model;

[0174] Determination of the second recognition accuracy: Input the second data test set into the neural network model, and calculate the correct rate according to the actual output recognition result and the labeled true label corresponding to the second data test set, that is, the second recognition accuracy.

[0175] If the second recognition accuracy is less than the second preset threshold, obtain the next second data training set of the current second data training set, use the next second data training set as the current second data training set, and execute the step of training the initial writing recognition model based on the current second data training set;

[0176] If the second recognition accuracy is greater than or equal to the second preset threshold, use the initial writing recognition model as the writing recognition model.

[0177] Step S304: Receive the attitude sensing parameters sent by the intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an in-air writing operation.

[0178] Step S305: Generate writing trajectory data based on the attitude sensing parameters, input the writing trajectory data into the writing recognition model, and output a writing recognition result.

[0179] Specifically, steps S101 - S103 can be referred to.

[0180] In an embodiment of the present application, a display device may receive attitude sensing parameters sent by a smart device, where the attitude sensing parameters are generated when the smart device senses an in-air writing operation; then generate writing trajectory data based on the attitude sensing parameters; and then input the writing trajectory data into a writing recognition model to output an accurate writing recognition result. In addition, the writing recognition model is obtained by transfer training based on third writing data of an in-air medium and fourth writing data of a planar medium. Based on the foregoing method, problems such as low accuracy and low recognition processing efficiency of in-air writing trajectory recognition in related technologies can be avoided, and the convenience of cross-device collaborative control is improved; and the recognition of writing data can be achieved only based on the attitude sensing data collected by the smart device, and three-dimensional data of the smart device does not need to be collected during writing recognition, nor does the three-dimensional data need to be remapped to two dimensions for recognition, which greatly reduces the computational amount of in-air writing recognition and improves the recognition processing efficiency; and in the stage of in-air writing recognition, a writing recognition model based on transfer learning is adopted. A general pre-trained model with general performance can be obtained through existing related data (such as fourth writing data of a planar medium), and then the model is fine-tuned with a small amount of domain data to obtain a model with better matching for the current data type. It also improves the robustness of the model, and only a small amount of in-air writing data is required to ensure the accuracy of in-air writing trajectory recognition and improve the recognition processing efficiency.

[0181] In one embodiment, as Figure 10 shown, a method for in-air writing trajectory recognition is specifically proposed, which is applied to a smart device. This method can be implemented depending on a computer program and can run on an in-air writing trajectory recognition device based on the von Neumann architecture. The computer program can be integrated in an application or run as an independent tool-type application.

[0182] The smart device can be an electronic device with communication functions. The smart device can have complete functions and can implement complete or partial functions without relying on a display device (such as a smartphone), such as a smart watch, smart glasses, smart speaker, smart toothbrush, smart pen, smart bracelet, etc., and only focus on a certain type of application function and need to cooperate with other devices such as a smartphone. For example, various smart bracelets, smart jewelry, smart schoolbags, etc. for operation monitoring are smart devices.

[0183] Specifically, the method for in-air writing trajectory recognition includes:

[0184] Step S401: Sense an in-air writing operation on the display device and collect attitude sensing data corresponding to the in-air writing operation.

[0185] In some embodiments, the intelligent device and the display device can be used in cooperation. The intelligent device can be an electronic device that facilitates the user to perform air writing operations. As a new means of human-computer interaction, air writing is different from traditional human-computer interaction methods. In this application, air handwriting allows the user to write in the air with the intelligent device in a natural and unrestrained manner, providing a more intuitive, convenient, and comfortable interaction experience. For example, in some scenarios, the display device can be a weak input device (such as a TV, a conference tablet), that is, it is not convenient for the user to directly input relevant information, such as writing characters, drawing, etc. Therefore, in some interaction scenarios, air writing based on the intelligent device can be an interaction form that can achieve the purpose of complex information instruction input. Air writing includes writing input forms with paper or electronic media as the writing surface, and input forms of waving fingers / wrists / arms, etc. in three-dimensional air media for air handwriting. The recognition technology for writing on a flat medium has been relatively mature, such as the touch screen input methods carried by various smartphones. In some implementation scenarios, with the maturity of technology products in the fields of virtual reality and the Internet of Things, the feature that air writing based on intelligent devices (such as smart bracelets, smart watches, smart pens, etc.) is not limited to a flat surface can be well combined with display devices (such as TVs, conference tablets, etc.) that are not convenient for users to directly write.

[0186] In some embodiments, the pose sensing data refers to the position, pose, angular velocity data, acceleration data, etc. sensed by a movable or rotatable object (such as an intelligent device) based on the included sensor components at the current sampling time point. In this application, it can be understood as the position of the intelligent device in space and / or its pose at that position when the user is operating the intelligent device. In this embodiment, the pose sensing data can be used to characterize or quantify the current pose of the intelligent device. Such as the change in the angular velocity of the intelligent device (the rotation angle between two adjacent sampling moments), the change in the moving speed, the change in the moving direction, the angle parameter of the intelligent device relative to the reference two-dimensional plane (which can be the reference two-dimensional plane of the display device, the surrounding environment objects (such as walls)) (the included angle between the display device pointing direction and the normal line of the reference two-dimensional plane such as the display device is α), and the distance parameter (such as the distance R between the intelligent device and the reference two-dimensional plane such as the display device), etc.

[0187] Furthermore, the intelligent device has a variety of sensor components for real-time detecting the current pose information. The sensor components include but are not limited to acceleration sensors, magnetic sensors, gyroscopes, physical quantity sensors, etc. The control device obtains the current physical quantity parameters through the electronic components. The physical quantity parameters can be acceleration parameters, magnetic parameters, angular velocity parameters, relative distance and relative angle relative to the reference two-dimensional plane such as the display device, etc., and performs pose calculation on the physical quantity parameters, that is, the pose sensing data of the intelligent device can be obtained.

[0188] Step S402: Send the attitude sensing data to the display device, so that the display device generates writing trajectory data corresponding to the attitude sensing data, and inputs the writing trajectory data into a writing recognition model to output a writing recognition result.

[0189] Specifically, when the intelligent device senses that the user operates the intelligent device for in-air writing, the intelligent device can sense the current attitude sensing data through the included sensor components. In practical applications, the intelligent device can establish a communication connection with the display device (such as a Bluetooth connection, a wireless local area network connection, etc.). Then, after collecting the current attitude sensing data, the attitude sensing data is transmitted to the display device based on this communication connection, so that the display device can receive the attitude sensing data of the display device through this communication connection.

[0190] The communication connection includes but is not limited to: Bluetooth communication connection, wireless local area network connection, ZigBee communication connection, etc.

[0191] Among them, the writing recognition model is obtained by transfer training based on third writing data in an air medium and fourth writing data in a planar medium.

[0192] In a feasible implementation manner, the intelligent device can generate writing trajectory data based on the attitude sensing data, input the writing trajectory data into the writing recognition model, output the writing recognition result, and send the writing recognition result to the display device.

[0193] In a feasible implementation manner, the intelligent device can generate writing trajectory data based on the attitude sensing data, and send the writing trajectory data to the display device, so that the display device inputs the writing trajectory data into the writing recognition model to output a writing recognition result.

[0194] In the embodiments of the present application, when the intelligent device senses an in-air writing operation, it collects the attitude sensing parameters of the in-air writing operation and then can send them to the display device. The display device can receive the attitude sensing parameters sent by the intelligent device; then generate writing trajectory data based on the attitude sensing parameters; and then input the writing trajectory data into the writing recognition model to output an accurate writing recognition result. In addition, the writing recognition model is obtained by transfer training based on the third writing data of the in-air medium and the fourth writing data of the planar medium. Based on the foregoing method, the problems of low accuracy and low recognition processing efficiency of in-air writing trajectory recognition in the related art can be avoided, and the convenience of cross-device collaborative control is improved; and the recognition of writing data can be realized only based on the attitude sensing data collected by the intelligent device. When recognizing writing, it is not necessary to collect the three-dimensional data of the intelligent device nor map the three-dimensional data to two dimensions for recognition, which greatly reduces the calculation amount of in-air writing recognition and improves the recognition processing efficiency; and in the in-air writing recognition stage, a writing recognition model based on transfer learning is adopted. A general pre-trained model with average performance can be obtained through existing relevant data (such as the fourth writing data based on the planar medium), and then the model is fine-tuned with a small amount of domain data to obtain a model with better matching for the current data type. It also improves the robustness of the model. Only a small amount of in-air writing data is required to ensure the accuracy of in-air writing trajectory recognition and improve the recognition processing efficiency.

[0195] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.

[0196] Please refer to Figure 11 , which shows a schematic structural diagram of an in-air writing trajectory recognition device provided by an exemplary embodiment of the present application. The in-air writing trajectory recognition device can be implemented as all or part of the device through software, hardware, or a combination of both. The device 1 includes a sensing parameter receiving module 11, a writing data generating module 12, and a writing data output module 13.

[0197] The sensing parameter receiving module 11 is configured to receive the attitude sensing parameters sent by the intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an in-air writing operation;

[0198] The writing data generating module 12 is configured to generate writing trajectory data based on the attitude sensing parameters;

[0199] The writing data output module 13 is configured to input the writing trajectory data into the writing recognition model and output a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0200] Optionally, as Figure 12As shown, the induction parameter receiving module 11 includes:

[0201] A parameter determination unit 111, configured to determine the angular velocity of the intelligent device and auxiliary recognition parameters during air writing based on the attitude induction parameters;

[0202] A data generation unit 112, configured to determine a movement trajectory corresponding to the angular velocity based on the angular velocity and the auxiliary recognition parameters, and generate writing trajectory data corresponding to the movement trajectory.

[0203] Optionally, the parameter determination unit 111 is specifically configured to:

[0204] Obtain the angular velocity of the intelligent device during air writing from the attitude induction parameters and obtain a first auxiliary recognition parameter, where the first auxiliary recognition parameter includes a preset recognition resolution.

[0205] Optionally, the parameter determination unit 111 is specifically configured to:

[0206] Obtain the angular velocity of the intelligent device during air writing from the attitude induction parameters and obtain a first auxiliary recognition parameter, where the first auxiliary recognition parameter includes a preset recognition resolution.

[0207] Optionally, the data generation unit 112 is specifically configured to:

[0208] Use the product of the angular velocity and the recognition resolution as a first instantaneous movement displacement, and determine a first movement trajectory based on the first instantaneous movement displacement.

[0209] Optionally, the parameter determination unit 111 is specifically configured to:

[0210] Obtain the angular velocity of the intelligent device during air writing from the attitude induction parameters and obtain a second auxiliary recognition parameter, where the second auxiliary recognition parameter includes a normal angle and a distance from the intelligent device, and the normal angle is the normal angle of the intelligent device relative to the display device.

[0211] Optionally, as Figure 13 shown, the parameter determination unit 111 includes:

[0212] A type determination subunit 1111, configured to determine the writing type of the air writing operation based on the attitude induction parameters and a preset attitude parameter threshold;

[0213] A parameter determination subunit 1112, configured to obtain the angular velocity and a first auxiliary recognition parameter included in the attitude induction parameters when the writing type is a first writing type, where the first auxiliary recognition parameter includes a preset recognition resolution;

[0214] The parameter determination subunit 1112 is also used to obtain the angular velocity and the second auxiliary recognition parameter included in the attitude sensing parameter when the writing type is the second writing type. The second auxiliary recognition parameter includes the normal angle and the distance from the intelligent device. The normal angle is the normal angle of the intelligent device relative to the display device.

[0215] Optionally, when the writing type is the first writing type, the data generation unit 112 is specifically configured to:

[0216] Multiply the angular velocity by the recognition resolution as the first instantaneous displacement of movement, and determine the first movement trajectory based on the first instantaneous displacement of movement.

[0217] Optionally, when the writing type is the second writing type, the data generation unit 112 is specifically configured to:

[0218] Input the angular velocity, the normal angle, and the distance into the target instantaneous displacement formula, output the second instantaneous displacement of movement, and determine the second movement trajectory based on the second instantaneous displacement of movement;

[0219] Wherein, the target instantaneous displacement formula is:

[0220]

[0221] Wherein, x is the second instantaneous displacement of movement, R is the distance, α is the normal angle, and θ is the angular velocity.

[0222] Optionally, the device 1 is specifically configured to:

[0223] Perform trajectory style adjustment processing on the writing trajectory data to obtain the adjusted target writing trajectory data;

[0224] Input the target writing trajectory data into the writing recognition model.

[0225] Optionally, the device 1 is specifically configured to:

[0226] Perform trajectory detection processing on the grayscale image corresponding to the writing trajectory data to determine the target trajectory area;

[0227] Perform trajectory scaling processing on the target trajectory area to generate the target writing trajectory data of a preset specification.

[0228] Optionally, the device 1 is specifically configured to:

[0229] Determine a reference supplementary full line corresponding to the target trajectory area, and adjust the target trajectory area to a preset shape with reference to the reference supplementary full line.

[0230] Optionally, as Figure 14 shown, the device 1 further includes:

[0231] A model training module 14, configured to obtain third writing data based on an air medium and the fourth writing data based on a planar medium;

[0232] The model training module 14 is configured to input the fourth writing data into a neural network model for training to generate an initial writing recognition model;

[0233] A transfer learning module 15, configured to input the third writing data into the initial writing recognition model for transfer learning to generate a writing recognition model.

[0234] Optionally, the model training module 14 is specifically configured to:

[0235] Obtain first sample data based on the air medium, perform trajectory style adjustment processing on the first sample data to obtain adjusted writing trajectory data; and,

[0236] Obtain second sample data based on the planar medium, perform trajectory style adjustment processing on the second sample data to obtain an adjusted writing recognition result.

[0237] Optionally, the model training module 14 is specifically configured to:

[0238] Divide the fourth writing data into at least one first data training set and a first data test set;

[0239] Train a neural network model based on the current first data training set, and use the first data test set to test the first recognition accuracy of the neural network model;

[0240] If the first recognition accuracy is less than a first preset threshold, obtain the next first data training set of the current first data training set, use the next first data training set as the current first data training set, and perform the step of training the neural network model based on the current first data training set;

[0241] If the first recognition accuracy is greater than or equal to the first preset threshold, use the neural network model as the initial writing recognition model.

[0242] Optionally, the transfer learning module 15 is specifically configured to:

[0243] Divide the third writing data into at least one second data training set and a second data test set;

[0244] Train the initial writing recognition model based on the current second data training set, and test the second recognition accuracy of the initial writing recognition model using the second data test set;

[0245] If the second recognition accuracy is less than the second preset threshold, obtain the next second data training set of the current second data training set, use the next second data training set as the current second data training set, and perform the step of training the initial writing recognition model based on the current second data training set;

[0246] If the second recognition accuracy is greater than or equal to the second preset threshold, use the initial writing recognition model as the writing recognition model.

[0247] Optionally, the device 1 is further configured to: display at least one standard writing object corresponding to the writing recognition result.

[0248] It should be noted that when the above-described air writing trajectory recognition device provided in the above embodiment executes the air writing trajectory recognition method, only the above-mentioned division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the air writing trajectory recognition device provided in the above embodiment and the air writing trajectory recognition method embodiment belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be repeated here.

[0249] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0250] Please refer to Figure 15 , which shows a schematic structural diagram of an air writing trajectory recognition device provided by an exemplary embodiment of the present application. The air writing trajectory recognition device can be implemented as all or part of the device through software, hardware, or a combination of both. The device 2 includes a data sensing module 21 and a data sending module 22.

[0251] The data sensing module 21 is configured to sense an air writing operation on the display device and collect attitude sensing data corresponding to the air writing operation;

[0252] The data sending module 22 is configured to send the attitude sensing data to the display device, so that the display device generates writing trajectory data corresponding to the attitude sensing data, and inputs the writing trajectory data into a writing recognition model to output a writing recognition result; the writing recognition model is obtained by transfer training based on third writing data in the air medium and fourth writing data on a flat medium.

[0253] Optionally, the device 2 is specifically configured to:

[0254] Generate writing trajectory data based on the attitude sensing data, input the writing trajectory data into the writing recognition model, output the writing recognition result, and send the writing recognition result to the display device; or,

[0255] Generate writing trajectory data based on the attitude sensing data, and send the writing trajectory data to the display device, so that the display device inputs the writing trajectory data into the writing recognition model to output a writing recognition result.

[0256] It should be noted that when the above-mentioned air writing trajectory recognition device provided in the above embodiment executes the air writing trajectory recognition method, only the above-mentioned division of each functional module is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the above-mentioned air writing trajectory recognition device provided in the above embodiment and the air writing trajectory recognition method embodiment belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be elaborated here.

[0257] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0258] The embodiment of the present application also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor to perform the air writing trajectory recognition method as described in the above Figures 1 - 10 shown embodiment. The specific execution process can refer to the specific description of the Figures 1 - 10 shown embodiment, which will not be elaborated here.

[0259] The present application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor to perform the air writing trajectory recognition method as described in the above Figures 1 - 10 shown embodiment. The specific execution process can refer to the specific description of the Figures 1 - 10 shown embodiment, which will not be elaborated here.

[0260] Please refer to Figure 16, which shows a structural block diagram of an electronic device provided by an exemplary embodiment of the present application. The electronic device in the present application may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected through the bus 150.

[0261] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts within the entire electronic device, and by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, as well as calling data stored in the memory 120, it executes various functions of the electronic device 100 and processes data. Optionally, the processor 110 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 110 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for the rendering and drawing of display content; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 110 and may be implemented separately through a communication chip.

[0262] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems. The data storage area may also store data created by the electronic device during use, such as a phone book, audio and video data, chat record data, etc.

[0263] See also Figure 17 As shown, the memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, different application scenarios in the same third-party application also have different requirements for system resources. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0264] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0265] Taking the Android operating system as an example, the programs and data stored in the memory 120 are as follows: Figure 17As shown, the memory 120 may store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360 and an application layer 380, wherein the Linux kernel layer 320, the system runtime library layer 340 and the application framework layer 360 belong to the operating system space, and the application layer 380 belongs to the user space. The Linux kernel layer 320 provides underlying drivers for various hardware of electronic devices, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, power management, etc. The system runtime library layer 340 provides the main feature support for the Android system through some C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D drawing support, and the Webkit library provides browser kernel support, etc. The Android runtime library (Android runtime) is also provided in the system runtime library layer 340, which mainly provides some core libraries that allow developers to use the Java language to write Android applications. The application framework layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the application layer 380. These applications can be native applications that come with the operating system, such as contact applications, text messaging applications, clock applications, camera applications, etc.; they can also be third-party applications developed by third-party developers, such as game applications, instant messaging applications, photo beautification applications, etc.

[0266] Taking the operating system as an IOS system as an example, the programs and data stored in the memory 120 are as follows: Figure 17As shown in the figure, the iOS system includes: Core OS layer 420, Core Services layer 440, Media layer 460, and Cocoa Touch Layer 480. The Core OS layer 420 includes the operating system kernel, drivers, and underlying program frameworks, which provide functions closer to the hardware for the program frameworks in the Core Services layer 440 to use. The Core Services layer 440 provides the system services and / or program frameworks required by applications, such as the Foundation framework, Account framework, Advertising framework, Data Storage framework, Network Connection framework, Location framework, Motion framework, and so on. The Media layer 460 provides interfaces related to audio-visual aspects for applications, such as interfaces related to graphics and images, audio technology-related interfaces, video technology-related interfaces, and the AirPlay interface for audio and video transmission technology. The Cocoa Touch Layer 480 provides various commonly used interface-related frameworks for application development and is responsible for the touch interaction operations of users on electronic devices. For example, local notification services, remote push services, advertising frameworks, game tool frameworks, Message User Interface (UI) frameworks, User Interface UIKit frameworks, Map frameworks, and so on.

[0267] In Figure 18 Among the frameworks shown, the frameworks related to most applications include, but are not limited to: the Foundation framework in the Core Services layer 440 and the UIKit framework in the Cocoa Touch Layer 480. The Foundation framework provides many basic object classes and data types, provides the most basic system services for all applications, and is independent of the UI. The classes provided by the UIKit framework are the basic UI class libraries, used to create touch-based user interfaces. iOS applications can provide the UI based on the UIKit framework, so it provides the infrastructure for applications to build user interfaces, draw, process, and handle user interaction events, respond to gestures, and so on.

[0268] Among them, the methods and principles for implementing data communication between third-party applications and the operating system in the iOS system can refer to the Android system, which will not be elaborated in this application.

[0269] Among them, the input device 130 is used to receive input instructions or data. The input device 130 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is used to output instructions or data. The output device 140 includes, but is not limited to, a display device, a speaker, etc. In one example, the input device 130 and the output device 140 can be combined. The input device 130 and the output device 140 are a touch display screen, which is used to receive touch operations of a user using any suitable object such as a finger or a stylus on or near it, and to display the user interfaces of various application programs. The touch display screen is usually arranged on the front panel of the electronic device. The touch display screen can be designed as a full-screen, a curved screen, or a special-shaped screen. The touch display screen can also be designed as a combination of a full-screen and a curved screen, or a combination of a special-shaped screen and a curved screen. The embodiments of the present application do not limit this.

[0270] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the drawings, or combine certain components, or have different component arrangements. For example, the electronic device further includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, etc., which will not be elaborated here.

[0271] In the embodiments of the present application, the execution subject of each step may be the electronic device introduced above. Optionally, the execution subject of each step is the operating system of the electronic device. The operating system may be an Android system, an IOS system, or other operating systems. The embodiments of the present application do not limit this.

[0272] The electronic device according to the embodiment of the present application may also be equipped with a display device, which can be various devices capable of realizing the display function, such as: cathode ray tube display (CR for short), light-emitting diode display (LED for short), electronic ink screen, liquid crystal display (LCD for short), plasma display panel (PDP for short), etc. Users can use the display device on the electronic device 101 to view information such as text, images, and videos displayed. The electronic device may be a smart phone, a tablet computer, a game device, an AR (Augmented Reality) device, an automobile, a data storage device, an audio playback device, a video playback device, a notebook, a desktop computing device, a wearable device such as an electronic watch, electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, an electronic clothing, etc.

[0273] In Figure 17 In the electronic device shown, the electronic device may be a terminal, and the processor 110 may be used to call the air writing trajectory recognition application program stored in the memory 120 and specifically perform the following operations:

[0274] Receive the attitude sensing parameters sent by the smart device, where the attitude sensing parameters are generated when the smart device senses an air writing operation;

[0275] Generate writing trajectory data based on the attitude sensing parameters;

[0276] Input the writing trajectory data into a writing recognition model, and output a writing recognition result corresponding to the writing trajectory data based on the writing recognition model.

[0277] In one embodiment, when the processor 110 executes generating writing trajectory data based on the attitude sensing parameters, the following steps are specifically executed:

[0278] Based on the attitude sensing parameters, determine the angular velocity of the smart device and auxiliary recognition parameters during air writing;

[0279] Based on the angular velocity and the auxiliary recognition parameters, determine the movement trajectory corresponding to the angular velocity, and generate writing trajectory data corresponding to the movement trajectory.

[0280] In one embodiment, when the processor 110 executes determining the angular velocity and auxiliary recognition parameters based on the attitude sensing parameters, it includes:

[0281] Determine the writing type of the air writing operation based on the posture sensing parameters and preset posture parameter thresholds;

[0282] When the writing type is the first writing type, obtain the angular velocity and the first auxiliary recognition parameter included in the posture sensing parameters, where the first auxiliary recognition parameter includes a preset recognition resolution;

[0283] When the writing type is the second writing type, obtain the angular velocity and the second auxiliary recognition parameter included in the posture sensing parameters, where the second auxiliary recognition parameter includes a normal angle and the distance from the intelligent device, and the normal angle is the normal angle of the intelligent device relative to the display device.

[0284] In one embodiment, when the processor 110 executes the writing type as the first writing type and determines the movement trajectory corresponding to the angular velocity based on the angular velocity and the auxiliary recognition parameter, the following operations are performed:

[0285] Multiply the angular velocity by the recognition resolution as the first instantaneous displacement of movement, and determine the first movement trajectory based on the first instantaneous displacement of movement.

[0286] In one embodiment, when the processor 110 executes the writing type as the second writing type and determines the movement trajectory corresponding to the angular velocity based on the angular velocity and the auxiliary recognition parameter, it includes:

[0287] Input the angular velocity, the normal angle, and the distance into the target instantaneous displacement formula to output the second instantaneous displacement of movement, and determine the second movement trajectory based on the second instantaneous displacement of movement;

[0288] Wherein, the target instantaneous displacement formula is:

[0289]

[0290] Wherein, x is the second instantaneous displacement of movement, R is the distance, α is the normal angle, and θ is the angular velocity.

[0291] In one embodiment, when the processor 110 executes the operation of inputting the writing trajectory data into the writing recognition model, the following steps are specifically performed:

[0292] Perform trajectory style adjustment processing on the writing trajectory data to obtain the adjusted target writing trajectory data;

[0293] Input the target writing trajectory data into the writing recognition model.

[0294] In one embodiment, when the processor 110 executes the processing of adjusting the trajectory style of the writing trajectory data to obtain the adjusted target writing trajectory data, it includes:

[0295] Perform trajectory detection processing on the grayscale image corresponding to the writing trajectory data to determine the target trajectory area;

[0296] Perform trajectory scaling processing on the target trajectory area to generate target writing trajectory data of a preset specification.

[0297] In one embodiment, before the processor 110 executes the character scaling processing on the first character in the target trajectory area, it also performs the following operations:

[0298] Determine the reference supplementary line corresponding to the target trajectory area, and adjust the target trajectory area to a preset shape with reference to the reference supplementary line.

[0299] In one embodiment, before the processor 110 executes the receiving of the attitude sensing parameters sent by the intelligent device, it also performs the following operations:

[0300] Obtain the third writing data based on the air medium and the fourth writing data based on the plane medium;

[0301] Input the fourth writing data into a neural network model for training to generate an initial writing recognition model;

[0302] Input the third writing data into the initial writing recognition model for training to generate a writing recognition model.

[0303] In one embodiment, when the processor 110 executes the obtaining of the third writing data based on the air medium and the fourth writing data based on the plane medium, it specifically performs the following steps:

[0304] Obtain the third sample data based on the air medium, perform trajectory style adjustment processing on the third sample data to obtain the adjusted third writing data; and,

[0305] Obtain the fourth sample data based on the plane medium, perform trajectory style adjustment processing on the fourth sample data to obtain the adjusted fourth writing data.

[0306] In one embodiment, when the processor 110 executes the inputting of the fourth writing data into a neural network model for training to generate an initial writing recognition model, it specifically performs the following steps:

[0307] Divide the fourth writing data into at least one first data training set and a first data test set;

[0308] Train a neural network model based on the current first data training set, and use the first data test set to test the first recognition accuracy of the neural network model;

[0309] If the first recognition accuracy is less than the first preset threshold, obtain the next first data training set of the current first data training set, use the next first data training set as the current first data training set, and execute the step of training the neural network model based on the current first data training set;

[0310] If the first recognition accuracy is greater than or equal to the first preset threshold, use the neural network model as the initial handwriting recognition model.

[0311] In one embodiment, when the processor 110 executes the step of inputting the third handwriting data into the initial handwriting recognition model for training to generate a handwriting recognition model, it specifically executes the following steps: divide the third handwriting data into at least one second data training set and a second data test set;

[0312] Train the initial handwriting recognition model based on the current second data training set, and use the second data test set to test the second recognition accuracy of the initial handwriting recognition model;

[0313] If the second recognition accuracy is less than the second preset threshold, obtain the next second data training set of the current second data training set, use the next second data training set as the current second data training set, and execute the step of training the initial handwriting recognition model based on the current second data training set;

[0314] If the second recognition accuracy is greater than or equal to the second preset threshold, use the initial handwriting recognition model as the handwriting recognition model.

[0315] In one embodiment, after the processor 110 executes the output of the handwriting recognition result, it also performs the following operations:

[0316] Display at least one standard handwriting object corresponding to the handwriting recognition result.

[0317] In an embodiment of the present application, a display device may receive attitude sensing parameters sent by a smart device, where the attitude sensing parameters are generated when the smart device senses an in-air writing operation; then generate writing trajectory data based on the attitude sensing parameters; and then input the writing trajectory data into a writing recognition model to output an accurate writing recognition result. In addition, the writing recognition model is obtained by transfer training based on third writing data of an in-air medium and fourth writing data of a planar medium. Based on the foregoing method, problems such as low accuracy and low recognition processing efficiency of in-air writing trajectory recognition in related technologies can be avoided, and the convenience of cross-device collaborative control can be improved; and the recognition of writing data can be achieved only based on the attitude sensing data collected by the smart device. When recognizing writing, it is not necessary to collect three-dimensional data of the smart device nor map the three-dimensional data to two dimensions for recognition, which greatly reduces the computational amount of in-air writing recognition and improves the recognition processing efficiency; and in the in-air writing recognition stage, a writing recognition model based on transfer learning is adopted. A general pre-trained model with general performance can be obtained through existing related data (such as fourth writing data of a planar medium), and then the model is fine-tuned with a small amount of domain data to obtain a model with better matching for the current data type. It also improves the robustness of the model, and only a small amount of in-air writing data is required to ensure the accuracy of in-air writing trajectory recognition and improve the recognition processing efficiency.

[0318] Please refer to Figure 15 , which is a schematic structural diagram of another electronic device provided by an embodiment of the present application. As Figure 15 shown, the electronic device 2000 may include: at least one processor 2001, at least one network interface 2004, a user interface 2003, a memory 2005, and at least one communication bus 2002.

[0319] Among them, the communication bus 2002 is used to realize the connection and communication between these components.

[0320] Among them, the user interface 2003 may include a display screen (Display), and optionally the user interface 2003 may further include a standard wired interface and a wireless interface.

[0321] Among them, the network interface 2004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0322] Among them, the processor 2001 may include one or more processing cores. The processor 2001 is connected to various parts within the entire server 2000 through various interfaces and circuits. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 2005, and by calling the data stored in the memory 2005, it executes various functions of the server 2000 and processes data. Optionally, the processor 2001 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 2001 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 2001 and may be implemented separately through a single chip.

[0323] Among them, the memory 2005 may include random access memory (RAM) and may also include read-only memory. Optionally, the memory 2005 includes a non-transitory computer-readable storage medium. The memory 2005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store the data involved in the above-mentioned various method embodiments. Optionally, the memory 2005 may also be at least one storage device located far from the aforementioned processor 2001. As Figure 15 shown, the memory 2005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an air writing trajectory recognition application program.

[0324] In Figure 15In the electronic device 2000 shown, the user interface 2003 is mainly used to provide an interface for the user to input and obtain the data input by the user; and the processor 2001 can be used to call the air writing trajectory recognition application program stored in the memory 2005 and specifically perform the following operations:

[0325] Sense an air writing operation on the display device and collect the attitude sensing data corresponding to the air writing operation;

[0326] Send the attitude sensing data to the display device so that the display device generates writing trajectory data corresponding to the attitude sensing data and inputs the writing trajectory data into a writing recognition model to output a writing recognition result; the writing recognition model is obtained by transfer training based on third writing data of an air medium and fourth writing data of a planar medium.

[0327] In one embodiment, when the processor 2001 executes the air writing trajectory recognition method, it also performs the following operations:

[0328] Generate writing trajectory data based on the attitude sensing data, input the writing trajectory data into the writing recognition model, output the writing recognition result, and send the writing recognition result to the display device; or,

[0329] Generate writing trajectory data based on the attitude sensing data, and send the writing trajectory data to the display device, so that the display device inputs the writing trajectory data into the writing recognition model to output a writing recognition result.

[0330] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0331] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0332] The foregoing are only exemplary embodiments of the present disclosure, and thus cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An air writing trajectory recognition method, characterized in that, The method includes: Receiving attitude sensing parameters sent by an intelligent device, where the attitude sensing parameters are generated when the intelligent device senses an in-air writing operation, and the attitude sensing data includes angular velocity; Based on the attitude sensing parameters, determining the angular velocity of the intelligent device during in-air writing and auxiliary recognition parameters, and based on the angular velocity and the auxiliary recognition parameters, determining the movement trajectory corresponding to the angular velocity and generating writing trajectory data corresponding to the movement trajectory; Inputting the writing trajectory data into a writing recognition model, and based on the writing recognition model, outputting a writing recognition result corresponding to the writing trajectory data; The step of based on the attitude sensing parameters, determining the angular velocity of the intelligent device during in-air writing and auxiliary recognition parameters, and based on the angular velocity and the auxiliary recognition parameters, determining the movement trajectory corresponding to the angular velocity, includes: obtaining the angular velocity of the intelligent device during in-air writing and obtaining a first auxiliary recognition parameter from the attitude sensing parameters, where the first auxiliary recognition parameter includes a preset recognition resolution, taking the product of the angular velocity and the recognition resolution as a first instantaneous movement displacement, and determining a first movement trajectory based on the first instantaneous movement displacement; or, The step of based on the attitude sensing parameters, determining the angular velocity of the intelligent device during in-air writing and auxiliary recognition parameters, and based on the angular velocity and the auxiliary recognition parameters, determining the movement trajectory corresponding to the angular velocity and generating writing trajectory data corresponding to the movement trajectory, includes: obtaining the angular velocity of the intelligent device during in-air writing and obtaining a second auxiliary recognition parameter from the attitude sensing parameters, where the second auxiliary recognition parameter includes a normal angle and a distance from the intelligent device, and the normal angle is the normal angle of the intelligent device relative to the display device; inputting the angular velocity, the normal angle, and the distance into a target instantaneous displacement formula to output a second instantaneous movement displacement, and determining a second movement trajectory based on the second instantaneous movement displacement; Wherein, the target instantaneous displacement formula is: Where x is the second instantaneous movement displacement, R is the distance, α is the normal angle, and θ is the angular velocity.

2. The method according to claim 1, characterized in that, The step of based on the attitude sensing parameters, determining the angular velocity of the intelligent device during in-air writing and auxiliary recognition parameters, and based on the angular velocity and the auxiliary recognition parameters, determining the movement trajectory corresponding to the angular velocity, includes: Based on the attitude sensing parameters and a preset attitude parameter threshold, determining the writing type of the in-air writing operation; When the writing type is a second writing type, obtaining the angular velocity and a second auxiliary recognition parameter included in the attitude sensing parameters, where the second auxiliary recognition parameter includes a normal angle and a distance from the intelligent device, and the normal angle is the normal angle of the intelligent device relative to the display device, taking the product of the angular velocity and the recognition resolution as a first instantaneous movement displacement, and determining a first movement trajectory based on the first instantaneous movement displacement; When the writing type is the first writing type, obtain the angular velocity and the first auxiliary recognition parameter included in the attitude sensing parameter. The first auxiliary recognition parameter includes a preset recognition resolution. Input the angular velocity, the normal angle, and the interval distance into the target instantaneous displacement formula to output the second moving instantaneous displacement, and determine the second moving trajectory based on the second moving instantaneous displacement. The target instantaneous displacement formula is as follows: where x is the second moving instantaneous displacement, R is the interval distance, α is the normal angle, and θ is the angular velocity. When the user writes through the intelligent device in the first writing type, the natural writing amplitude change is greater than that in the second writing type.

3. The method according to claim 1, characterized in that, The step of inputting the writing trajectory data into the writing recognition model includes: Perform trajectory style adjustment processing on the writing trajectory data to obtain the adjusted target writing trajectory data; Input the target writing trajectory data into the writing recognition model.

4. The method according to claim 3, wherein The step of performing trajectory style adjustment processing on the writing trajectory data to obtain the adjusted target writing trajectory data includes: Perform trajectory detection processing on the grayscale image corresponding to the writing trajectory data to determine the target trajectory area; Perform trajectory scaling processing on the target trajectory area to generate the target writing trajectory data of a preset specification.

5. The method according to claim 4, characterized in that Before performing the trajectory scaling processing on the target trajectory area to generate the target writing trajectory data of a preset specification, it further includes: Determine the reference supplementary line corresponding to the target trajectory area, and adjust the target trajectory area to a preset shape with reference to the reference supplementary line.

6. The method according to claim 1, characterized in that, Before receiving the attitude sensing parameter sent by the intelligent device, it further includes: Obtain the third writing data based on the air medium and the fourth writing data based on the plane medium; Input the fourth writing data into the neural network model for training to generate the initial writing recognition model; Input the third writing data into the initial writing recognition model for training to generate the writing recognition model.

7. The method according to claim 6, characterized in that, The step of obtaining the third writing data based on the air medium and the fourth writing data based on the plane medium includes: Obtain the third sample data based on the air medium, and perform trajectory style adjustment processing on the third sample data to obtain the adjusted third writing data; and Obtain the fourth sample data based on the plane medium, and perform trajectory style adjustment processing on the fourth sample data to obtain the adjusted fourth writing data.

8. The method according to claim 6, wherein The step of inputting the fourth writing data into the neural network model for training to generate the initial writing recognition model includes: Divide the fourth writing data into at least one first data training set and a first data test set; Train the neural network model based on the current first data training set, and test the first recognition accuracy of the neural network model using the first data test set; If the first recognition accuracy rate is less than the first preset threshold, obtain the next first data training set of the current first data training set, use the next first data training set as the current first data training set, and execute the step of training the neural network model based on the current first data training set; If the first recognition accuracy rate is greater than or equal to the first preset threshold, use the neural network model as the initial writing recognition model.

9. The method according to claim 6, characterized in that, The step of inputting the third writing data into the initial writing recognition model for training to generate a writing recognition model includes: Dividing the third writing data into at least one second data training set and a second data test set; Training the initial writing recognition model based on the current second data training set, and using the second data test set to test the second recognition accuracy rate of the initial writing recognition model; If the second recognition accuracy rate is less than the second preset threshold, obtain the next second data training set of the current second data training set, use the next second data training set as the current second data training set, and execute the step of training the initial writing recognition model based on the current second data training set; If the second recognition accuracy rate is greater than or equal to the second preset threshold, use the initial writing recognition model as the writing recognition model.

10. The method according to claim 1, wherein After outputting a writing recognition result corresponding to the writing trajectory data based on the writing recognition model, it further includes: Displaying at least one standard writing object corresponding to the writing recognition result.

11. An aerial writing character recognition method, characterized in that, The method includes: Sensing an air writing operation on a display device, and collecting attitude sensing data corresponding to the air writing operation, where the attitude sensing data includes an angular velocity; Sending the attitude sensing data to the display device, so that the display device generates writing trajectory data corresponding to the attitude sensing data, and inputting the writing trajectory data into a writing recognition model to output a writing recognition result; The display device generating the writing trajectory data corresponding to the attitude sensing data includes: Based on the attitude sensing parameters, determining the angular velocity of the intelligent device and auxiliary recognition parameters during air writing, based on the angular velocity and the auxiliary recognition parameters, determining a movement trajectory corresponding to the angular velocity, and generating writing trajectory data corresponding to the movement trajectory; The step of determining the angular velocity of the intelligent device and auxiliary recognition parameters during air writing based on the attitude sensing parameters, and determining a movement trajectory corresponding to the angular velocity based on the angular velocity and the auxiliary recognition parameters includes: Obtaining the angular velocity of the intelligent device during air writing and obtaining a first auxiliary recognition parameter from the attitude sensing parameters, where the first auxiliary recognition parameter includes a preset recognition resolution, using the product of the angular velocity and the recognition resolution as a first instantaneous movement displacement, and determining a first movement trajectory based on the first instantaneous movement displacement; or, Based on the attitude sensing parameters, determine the angular velocity of the intelligent device and auxiliary recognition parameters during air writing. Based on the angular velocity and the auxiliary recognition parameters, determine the movement trajectory corresponding to the angular velocity, and generate writing trajectory data corresponding to the movement trajectory, including: obtaining the angular velocity of the intelligent device during air writing and obtaining second auxiliary recognition parameters from the attitude sensing parameters, where the second auxiliary recognition parameters include a normal angle and a distance from the intelligent device, and the normal angle is the normal angle of the intelligent device relative to the display device; input the angular velocity, the normal angle, and the distance into a target instantaneous displacement formula to output a second instantaneous displacement of movement, and determine a second movement trajectory based on the second instantaneous displacement of movement; wherein, the target instantaneous displacement formula is: where x is the second instantaneous displacement of movement, R is the distance, α is the normal angle, and θ is the angular velocity.

12. The method according to claim 11, wherein The method further includes: generating writing trajectory data based on the attitude sensing data, inputting the writing trajectory data into the writing recognition model, outputting the writing recognition result, and sending the writing recognition result to the display device; or, generating writing trajectory data based on the attitude sensing data, sending the writing trajectory data to the display device, so that the display device inputs the writing trajectory data into the writing recognition model to output a writing recognition result.

13. A computer storage medium, characterized in that, The computer storage medium stores multiple instructions, and the instructions are adapted to be loaded and executed by a processor to perform the method steps of any one of claims 1 to 10 or 11 to 12.

14. An electronic device, characterized in that, including: a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to perform the method steps of any one of claims 1 to 10 or 11 to 12.

Images