Scene classification method and device applied to touch system
By using data augmentation and deep learning to train neural network models, the problem of low scene classification efficiency in traditional touch systems has been solved, achieving efficient and accurate automatic scene classification and reducing resource consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RAYDIUM SEMICON
- Filing Date
- 2025-01-24
- Publication Date
- 2026-06-19
AI Technical Summary
Traditional touch systems rely on manual feature collection for scene classification, which makes it difficult to improve the efficiency and accuracy of classification judgment.
The scene data is preprocessed using data augmentation technology, a neural network model is trained using deep learning, and the model is lightweighted through an unspecified data frame filtering mechanism to achieve automatic scene classification.
It improves the accuracy of scene classification and reduces the consumption of computing and storage resources in software and hardware.
Smart Images
Figure CN122241295A_ABST
Abstract
Description
[Technical Field]
[0001] This invention relates to touch control, and in particular to a method and apparatus for scene classification applied to touch control systems. [Background Technology]
[0002] Generally speaking, traditional touch systems (such as wearable devices with touch functions) still rely on human feature collection methods to classify and judge different user operation scenarios (such as finger drawing, water mist operation, water flow operation, etc.), which makes it difficult to effectively improve the efficiency and accuracy of their classification and judgment, and needs to be further improved. [Summary of the Invention]
[0003] Therefore, the present invention proposes a scene classification method and apparatus for touch systems, thereby effectively solving the above-mentioned problems encountered by the prior art.
[0004] A preferred embodiment of the present invention provides a scene classification method for a touch system. In this embodiment, the scene classification method includes the following steps: (a) preprocessing scene data using data augmentation technology to generate a training scene dataset and a test scene dataset; (b) providing the training scene dataset to a neural network model for deep learning training; (c) providing the test scene dataset to the trained neural network model for scene classification inference to evaluate the accuracy of the neural network model in judging the operating scene; and (d) applying a non-specific data frame filtering mechanism to the neural network model to reduce its weight.
[0005] In one embodiment, the data augmentation technique is a horizontal flipping technique, a vertical flipping technique, or a combination of horizontal and vertical flipping techniques.
[0006] In one embodiment, step (b) further includes: when the neural network model is being trained, extracting features of different scenarios from the training scenario dataset to adjust the model parameters of the neural network model.
[0007] In one embodiment, when the neural network model determines that the operation scenario is a specific operation scenario, the touch system enters a specific operation mode corresponding to the specific operation scenario.
[0008] In one embodiment, the touch system will only enter a specific operation mode corresponding to a specific operation scenario when multiple consecutive frames are determined by the neural network model to be a specific operation scenario.
[0009] In one embodiment, the specific operation scenario is a finger operation scenario, a water mist operation scenario, a water flow operation scenario, or a near-field communication (NFC) charging scenario, and the specific operation mode is a finger operation mode, a water mist operation mode, a water flow operation mode, or an NFC charging mode.
[0010] In one embodiment, step (b) further includes providing the difference between two consecutive frames in the NFC charging scenario to the neural network model.
[0011] In one embodiment, step (b) further includes providing the neural network model with the difference between the two frames and the data after moving average when the specific operation scenario is too complex.
[0012] Another preferred embodiment of the present invention provides a scene classification device for a touch system. In this embodiment, the scene classification device includes a data preprocessing module, a model training module, a model inference module, and a data frame filtering module. The data preprocessing module is used to preprocess scene data using data augmentation technology to generate training scene datasets and test scene datasets. The model training module is coupled to the data preprocessing module and is used to provide the training scene datasets to a neural network model for deep learning training. The model inference module is coupled to the data preprocessing module and the model training module and is used to provide the test scene datasets to the trained neural network model for scene classification inference to evaluate the accuracy of the neural network model in judging the operation scene. The data frame filtering module is coupled to the model inference module and is used to apply an unspecified data frame filtering mechanism to the neural network model to reduce the weight of the neural network model.
[0013] In one embodiment, the data augmentation technique is a horizontal flipping technique, a vertical flipping technique, or a combination of horizontal and vertical flipping techniques.
[0014] In one embodiment, when the neural network model is trained, the model training module also extracts features of different scenarios from the training scenario dataset to adjust the model parameters of the neural network model.
[0015] In one embodiment, when the model inference module determines that the operation scenario is a specific operation scenario through the neural network model, the touch system enters a specific operation mode corresponding to the specific operation scenario.
[0016] In one embodiment, the touch system will only enter a specific operation mode corresponding to a specific operation scenario when the model inference module determines through the neural network model that multiple consecutive frames are all specific operation scenarios.
[0017] In one embodiment, the specific operation scenario is a finger operation scenario, a water mist operation scenario, a water flow operation scenario, or an NFC charging scenario, and the specific operation mode is a finger operation mode, a water mist operation mode, a water flow operation mode, or an NFC charging mode.
[0018] In one embodiment, the model training module also provides the neural network model with the difference between two consecutive frames in the NFC charging scenario.
[0019] In one embodiment, when a particular operation scenario is too complex, the model training module also provides the difference between two consecutive frames and the data after moving average to the neural network model.
[0020] Compared to existing technologies, the scene classification method and device proposed in this invention for touch systems first collects data of known scenes and performs data preprocessing using data augmentation technology. Then, it uses deep learning to train a convolutional neural network model to automatically learn scene features and uses model inference to make scene classification judgments. Furthermore, it can apply non-specific data filtering mechanisms to the neural network model to achieve the specific effects of model lightweighting and reducing the consumption of software and hardware computing and storage resources. [Attached Image Description]
[0021] Figure 1 A flowchart illustrating a scene classification method applied to a touch system in a specific embodiment of the present invention is shown.
[0022] Figure 2 A functional block diagram illustrating a scene classification device applied to a touch system in another specific embodiment of the present invention is shown.
[0023] Figure 3 The diagram illustrates how the data preprocessing module performs horizontal and vertical flipping of scene data using data augmentation technology.
[0024] Figure 4 A schematic diagram illustrating one embodiment of the non-specific data frame filtering mechanism employed by the data frame filtering module.
[0025] Figure 5 A schematic diagram illustrating an embodiment of a convolutional neural network model.
[0026] Figure 6 A functional block diagram illustrating an embodiment of the actual hardware architecture of the scene classification device of the present invention is shown.
Detailed Implementation Methods
[0027] A preferred embodiment of the present invention provides a scene classification method for a touch system. In this embodiment, the touch system can be any device or apparatus with touch functionality, such as a wearable device, a touch controller, or a touch and display driver integrated chip (TDDI), but is not limited thereto.
[0028] Please refer to Figure 1 , Figure 1 A flowchart illustrating the scene classification method applied to a touch system in this embodiment is shown. Figure 1 As shown, scene classification methods may include, but are not limited to, the following steps:
[0029] Step S10: Perform data preprocessing on the scene data using data augmentation technology to generate training scene datasets and test scene datasets;
[0030] Step S12: Provide the training scenario dataset to the neural network model for deep learning training;
[0031] Step S14: Provide the test scenario dataset to the trained neural network model for scenario classification and inference, in order to evaluate the accuracy of the neural network model in judging the operation scenario; and
[0032] Step S16: Apply an unspecified data frame filtering mechanism to the neural network model to lighten the neural network model.
[0033] In one embodiment, when the neural network model determines that the operation scenario is a specific operation scenario (e.g., finger operation scenario, water mist operation scenario, water flow operation scenario, or NFC charging scenario), the touch system will enter a specific operation mode corresponding to that specific operation scenario (e.g., finger operation mode, water mist operation mode, water flow operation mode, or NFC charging mode), but this is not a limitation. Furthermore, to avoid misjudgments / misoperations, the scene classification method of the present invention can also be configured so that the touch system only enters the specific operation mode corresponding to that specific operation scenario when multiple consecutive frames are determined by the neural network model to be the same specific operation scenario, but this is not a limitation.
[0034] In practical applications, the data augmentation technology used in step S10 can be horizontal flipping technology, vertical flipping technology, or a combination of horizontal and vertical flipping technology, but is not limited thereto. Furthermore, step S12 may also include: when the neural network model is being trained, extracting features from the training scenario dataset to adjust the model parameters of the neural network model; providing the difference between two consecutive frames in an NFC charging scenario to the neural network model; and when a specific operation scenario is too complex, providing the difference between two consecutive frames and the data after moving average to the neural network model, etc., but is not limited thereto.
[0035] Another preferred embodiment of the present invention is a scene classification device applied to a touch system. In this embodiment, the touch system can be any device or equipment with touch functionality, such as a wearable device, a touch controller, or a touch and display driver integrated chip (TDDI), but is not limited thereto.
[0036] Please refer to Figure 2 , Figure 2 A functional block diagram of the scene classification device applied to a touch system in this embodiment is shown. For example... Figure 2 As shown, the scene classification device 2 may include a data preprocessing module 20, a model training module 22, a model inference module 24, and a data frame filtering module 26.
[0037] The data preprocessing module 20 is coupled to the model training module 22 and the model inference module 24, respectively. The model training module 22 is coupled to the data preprocessing module 20 and the model inference module 24, respectively. The model inference module 24 is coupled to the data preprocessing module 20 and the data frame filtering module 26, respectively. The data frame filtering module 26 is coupled to the model inference module 24.
[0038] The data preprocessing module 20 receives scene data SD and preprocesses it using data augmentation techniques to generate a training scene dataset D1 and a test scene dataset D2. The model training module 22 provides the training scene dataset D1 to the neural network model for deep learning training. The model inference module 24 provides the test scene dataset D2 to the trained neural network model for scene classification inference, evaluating the accuracy of the neural network model in judging the operating scene. The data frame filtering module 26 applies a non-specific data frame filtering mechanism to the neural network model to reduce its weight.
[0039] In practical applications, the data augmentation techniques employed by the data preprocessing module 20 can be horizontal flipping, vertical flipping, or a combination of both, but are not limited to these. For example, such as Figure 3 As shown, the data preprocessing module 20 can perform horizontal and vertical flipping data preprocessing on the received scene data SD through data augmentation technology to generate training scene dataset D1 and test scene dataset D2, but is not limited thereto.
[0040] When the neural network model is trained, the model training module 22 will also extract features of different scenarios from the training scenario dataset D1 to adjust the model parameters of the neural network model, but this is not the only way. The model training module 22 will also provide the difference between two consecutive frames in the NFC charging scenario to the neural network model, but this is not the only way. When a specific operation scenario is too complex, the model training module 22 will also provide the difference between two consecutive frames and the data after moving average to the neural network model, but this is not the only way.
[0041] In one embodiment, when the model inference module 24 determines through the neural network model that the operation scenario is a specific operation scenario (e.g., finger operation scenario, water mist operation scenario, water flow operation scenario, or NFC charging scenario), the touch system will enter a specific operation mode corresponding to the specific operation scenario (e.g., finger operation mode, water mist operation mode, water flow operation mode, or NFC charging mode). Furthermore, to avoid misjudgments / misoperations, the touch system will only enter the specific operation mode corresponding to the specific operation scenario when the model inference module 24 determines through the neural network model that multiple consecutive frames are all within the specific operation scenario.
[0042] Please refer to Figure 4 , Figure 4 This diagram illustrates one embodiment of an unspecified data frame filtering mechanism employed by the data frame filtering module. Figure 4 As shown, the data frames are first processed by baseline deletion and frame parsing to be divided into data frames for different operation scenarios (such as line drawing machine, water flow, water mist, etc.). Then, after arbitrary 1:1 sampling to obtain dataset samples, they are divided into training dataset, effective dataset, and test dataset according to the ratio. The training data loader, effective data loader, and test data loader respectively perform the data loading program.
[0043] In one embodiment, such as Figure 5 As shown, a Convolutional Neural Network (CNN) model consists of one or more convolutional layers and a fully connected layer at the top. It should be noted that CNNs are a special type of neural network, named for using a mathematical operation called convolution in at least one layer instead of traditional matrix multiplication. They are primarily used in image recognition and image processing to process pixel data.
[0044] Please refer to Figure 6 , Figure 6 A functional block diagram illustrating one embodiment of the actual hardware architecture of the scene classification device of the present invention is shown. Figure 6As shown, since the weights W of the convolutional neural network are stored in the flash memory, the weights W are initially read from the flash memory and written to the SRAM. The other RAM is used to store the touch data TD.
[0045] The Artificial Intelligence (AI) classifier CF reads the corresponding weights W and touch data TD from the SRAM and RAM respectively. The INT2FP conversion module then converts the 16-bit signed integers (int16) into 16-bit floating-point numbers (float16) and stores them in multiple SRAMs 1 through SRAM 3 within the AI classifier CF. The processor PS within the AI classifier CF uses multiple processing cores PE Core#0 through PE Core#3 to form a systolic array for efficient convolution operations. Finally, the INT2FP conversion module converts the output activation function back from 16-bit floating-point numbers (float16) to 16-bit signed integers (int16) and stores it back in the SRAM.
[0046] Compared to existing technologies, the scene classification method and device proposed in this invention for touch systems first collects data of known scenes and performs data preprocessing using data augmentation technology. Then, it uses deep learning to train a convolutional neural network model to automatically learn scene features and classifies scenes through model inference. Finally, it applies an unspecified data filtering mechanism to the neural network model to achieve the effects of model lightweighting and reducing the consumption of software and hardware computing and storage resources. [Symbol Explanation] S10…Step S12…Step S14…Step S16…Step 2…Scene Classification Device 20…Data Preprocessing Module 22…Model Training Module 24… Model Inference Module 26…Data Frame Filtering Module SD...Scene Data D1… Training Scenario Dataset D2…Test Scenario Dataset FLASH…Flash memory W…weight SRAM…memory RAM…memory TD…Touch Data CF… Artificial Intelligence (AI) Classifier INT2FP… conversion module FSM... Finite State Machine SRAM1~SRAM3… memory PS… processor PE Core#0~PE Core#3… Processing component cores INT2FP… conversion module
Claims
1. A scene classification method applied to a touch system, comprising the following steps: (a) Preprocessing scene data using data augmentation techniques to generate training scene datasets and test scene datasets; (b) Provide the training scenario dataset to the neural network model for deep learning training; (c) Provide the test scenario dataset to the trained neural network model to perform scenario classification inference, so as to evaluate the accuracy of the neural network model in judging the operation scenario; as well as (d) Apply an unspecified data frame filtering mechanism to the neural network model to lighten the neural network model.
2. The scene classification method according to claim 1, wherein the data augmentation technology is a horizontal flipping technology, a vertical flipping technology, or a combination of horizontal and vertical flipping technology.
3. The scene classification method according to claim 1, wherein step (b) further includes: When the neural network model is trained, features from different scenarios are extracted from the training scenario dataset to adjust the model parameters of the neural network model.
4. The scene classification method according to claim 1, wherein when the neural network model determines that the operation scene is a specific operation scene, the touch system enters a specific operation mode corresponding to the specific operation scene.
5. The scene classification method according to claim 4, wherein the touch system will only enter the specific operation mode corresponding to the specific operation scene when multiple consecutive frames are judged by the neural network model as the specific operation scene.
6. The scene classification method according to claim 4, wherein the specific operation scene is a finger operation scene, a water mist operation scene, a water flow operation scene, or a near field communication (NFC) charging scene, and the specific operation mode is a finger operation mode, a water mist operation mode, a water flow operation mode, or an NFC charging mode.
7. The scene classification method according to claim 6, wherein step (b) further includes: The difference between two consecutive frames in this NFC charging scenario is provided to the neural network model.
8. The scene classification method according to claim 6, wherein step (b) further includes: When the specific operation scenario is too complex, the difference between the two frames and the data after moving average are provided to the neural network model.
9. A scene classification device, applied to a touch system, comprising: The data preprocessing module is used to preprocess scene data using data augmentation technology to generate training scene datasets and test scene datasets. The model training module, coupled to the data preprocessing module, is used to provide the training scenario dataset to the neural network model for deep learning training. The model inference module, coupled to the data preprocessing module and the model training module, is used to provide the test scenario dataset to the trained neural network model for scenario classification inference, so as to evaluate the accuracy of the neural network model in judging the operation scenario. as well as The data frame filtering module, coupled to the model inference module, is used to apply an unspecified data frame filtering mechanism to the neural network model in order to reduce the weight of the neural network model.
10. The scene classification device according to claim 9, wherein the data augmentation technology is a horizontal flipping technology, a vertical flipping technology, or a horizontal and vertical flipping technology.
11. The scene classification apparatus according to claim 9, wherein when the neural network model is trained, the model training module further extracts features of different scenes from the training scene dataset to adjust the model parameters of the neural network model.
12. The scene classification device according to claim 9, wherein when the model inference module determines that the operation scene is a specific operation scene through the neural network model, the touch system enters a specific operation mode corresponding to the specific operation scene.
13. The scene classification device according to claim 12, wherein when the model inference module determines through the neural network model that multiple consecutive frames are the specific operation scene, the touch system will enter the specific operation mode corresponding to the specific operation scene.
14. The scene classification device according to claim 12, wherein the specific operation scene is a finger operation scene, a water mist operation scene, a water flow operation scene, or an NFC charging scene, and the specific operation mode is a finger operation mode, a water mist operation mode, a water flow operation mode, or an NFC charging mode.
15. The scene classification apparatus according to claim 14, wherein the model training module also provides the difference between two consecutive frames in the NFC charging scene to the neural network model.
16. The scene classification device according to claim 14, wherein when the specific operation scene is too complex, the model training module also provides the difference between the two consecutive frames and the data after moving average to the neural network model.