TinyML model adaptive management application method and device

Abstract

The present application relates to the field of computer vision, and specifically provides a kind of TinyML model adaptive management application method, different functions of image recognition and data processing are deployed to different Internet of Things clients using CS architecture, any client can download different models remotely according to the different functions deployed, or download corresponding models in real time according to the change of running environment, to adapt to the change of different functions or external environment of client. Compared with the prior art, the present application can also realize the addition of equipment at any time, plug and play, and has good management effect in large-scale industrial scene application.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically providing a TinyML model adaptive management application method and apparatus. Background Technology

[0002] With the development of machine learning, TinyML has become widely popular and has become an emerging discipline. It can implement low-resource-consumption and low-power machine learning algorithms on resource-constrained microcontrollers. TinyML is widely used in ubiquitous IoT devices and can process sensor data from various locations in real time.

[0003] However, IoT devices are often tailored to specific tasks. Industrial IoT networks involve hundreds or thousands of devices, and it is often difficult to process them after fine-tuning the model. Embedded devices are usually customized for specific use cases, and their ability to optimize computation and efficiency varies. They also differ in the inference speed and performance of the same model. They have different sensors, RAM and flash memory sizes, computing power, and are affected by heterogeneity and limited resources, making it difficult to implement many complex models on IoT devices.

[0004] In addition, TinyML models generally have different structures, which makes it difficult for many models to achieve their best results in practical applications. Summary of the Invention

[0005] This invention addresses the shortcomings of the prior art by providing a highly practical TinyML model adaptive management application method.

[0006] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable TinyML model adaptive management application device.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] A TinyML model adaptive management application method adopts a client-server architecture, which deploys image recognition and data processing with different functions to different IoT clients. Any client can make a request to the server to remotely download different models according to the different functions deployed, or download the corresponding model in real time according to changes in the operating environment, so as to adapt to the client to implement different functions or changes in the external environment.

[0009] Furthermore, the IoT device includes a light intensity sensor, a humidity sensor, and a camera. The light intensity sensor, humidity sensor, and camera are all connected to an ARM processor, which is also connected to a GPU and a display.

[0010] Furthermore, the system uses an ARM processor to read camera data and sends it to a GPU for inference. It uses a light intensity sensor and a humidity sensor to determine the surrounding environment information and sends an environmental description to the server. The customer selects a function, and the server indexes the database based on the selected function, the environmental description information, and the GPU model. The corresponding model is then pushed to the ARM processor, which works with the GPU to complete the inference and count the number of people.

[0011] Furthermore, the IoT terminal pushes the video stream and inference results to the server via the network, and the client views them on a web page by connecting to the server;

[0012] The IoT terminal uses RTMP to push the video stream to the server, and the client can remotely view the video by pulling the stream from the server.

[0013] Furthermore, TinyML models use identifiers to uniquely identify model IDs, converting models from different training frameworks into a unified intermediate format NNEF or ONNX. Then, using pickle, a data persistence method in Python, the model objects are serialized and saved in pkl format, and uploaded to the server for model storage according to a unified classification format.

[0014] Furthermore, the TinyML model is first divided according to function. After being divided according to function, it is further divided according to model size in different function groups. Larger models are used in IoT devices with better CPU performance, while smaller models can be used in IoT devices with weaker MCU performance. After determining the model size, the model selection is further refined according to environmental parameters.

[0015] Furthermore, the database is a MySQL database. The data structure stored in the MySQL database is as follows: Function represents the function of the model, Size represents the size of the model, Para represents the parameters of the model, and Addr represents the storage location of the model on the server.

[0016] Furthermore, when a client needs a model of type A11, the client connects to the server via IP and port number. After a successful connection, the client sends the string A11, along with the strings of the environment descriptor and user descriptor, indicating that the client needs a model of type A11.

[0017] After the server receives the A11 data from the client, it parses the string, looks up the storage address of the A11 model in the database, binds the address to the local byte input stream object FileInputStream, and sends the read A11.pkl file to the client through the network object output stream OutputStream.

[0018] A TinyML model adaptive management application device includes: at least one memory and at least one processor;

[0019] The at least one memory is used to store a machine-readable program;

[0020] The at least one processor is used to call the machine-readable program to execute a TinyML model adaptive management application method.

[0021] Compared with existing technologies, the TinyML model adaptive management application method and apparatus of the present invention have the following outstanding advantages:

[0022] This invention reduces the memory and performance requirements of IoT devices through collaborative processing between the client and server, while expanding the functionality of the client. In addition, in IoT application scenarios where devices need to be added or removed frequently, this invention can also effectively enable the addition of devices at any time, making them plug-and-play, and achieving good management results in large-scale industrial applications. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Appendix Figure 1 This is a flowchart (I) of a TinyML model adaptive management application method;

[0025] Appendix Figure 2 This is a flowchart (II) of a TinyML model adaptive management application method;

[0026] Appendix Figure 3 This is a multi-terminal schematic diagram of a TinyML model adaptive management application method. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The following is a preferred embodiment:

[0029] like Figure 1-3 As shown in this embodiment, a TinyML model adaptive management application method adopts a client-server architecture, which deploys image recognition and data processing with different functions to different IoT clients. Any client can remotely download different models after submitting a request to the server according to the different functions deployed, or download the corresponding model in real time according to changes in the operating environment, so as to adapt to the client to implement different functions or changes in the external environment.

[0030] The IoT device includes a light intensity sensor, a humidity sensor, and a camera. The light intensity sensor, humidity sensor, and camera are all connected to an ARM processor, which is also connected to a GPU and a display.

[0031] This embodiment uses an ARM processor to read camera data and sends it to a GPU for inference. A light intensity sensor and a humidity sensor determine the surrounding environment. Environmental descriptions are sent to the server during daylight, nighttime, sunny weather, and rainy weather. The client selects functions such as pedestrian counting. The server indexes its database based on this function, environmental descriptions, and GPU model, and pushes the corresponding model to the ARM processor. The ARM processor, in conjunction with the GPU, completes the inference and counts the number of people. The video stream and inference results are then pushed to the server via the network, allowing other clients to view the data on a web browser. The IoT terminal uses RTMP to push the video stream to the server, and other clients can remotely view the data by pulling the stream from the server. When a user wants to change a function, such as using facial recognition for access control, they only need to configure the desired function on the IoT device and repeat the above steps to download the corresponding model file from the server and implement the new function.

[0032] Identifiers are unique IDs used to identify models. TinyML models contain information such as the model's location, training dataset, creation date, model creator, and human-readable descriptions, which simplify model classification. Different deep learning frameworks generate image recognition models in different storage formats. For example, models trained under the TensorFlow framework are mostly stored in .ckpt format, while models trained under the PyTorch framework are mostly stored in pkl format. To enable image recognition and artificial intelligence models to flow between different frameworks, model conversion is almost always used for industrial deployment in practical applications. This involves the unified management of the model, connecting it from the training framework to the deployment-side inference framework. This invention converts models from different training frameworks into a unified intermediate format, NNEF or ONNX. Then, using the Python data persistence method pickle, the model object is serialized and saved in pkl format, and uploaded to the server for model storage according to a unified classification format.

[0033] First, the models are divided according to their functions, such as pedestrian recognition models, animal recognition models, and people counting models. After dividing them according to functions, they are further divided according to model size within different functional groups. Larger models can be used in IoT devices with better CPU performance, while smaller models can be used in IoT devices with weaker performance, such as MCUs. After determining the model size, the selection of models can be further refined based on environmental parameters.

[0034] The database is a MySQL database, and the data structure stored in a MySQL database follows this format. Here, Function represents the model's functionality, Size represents the model's size, Para represents the model's parameters, and Addr represents the model's storage location on the server.

[0035] When Client 1 needs a model of type A11, Client 1 connects to the server via IP address and port number. After a successful connection, it sends the string "A11" along with environment descriptors, user descriptors, and other strings, indicating that the client needs a model of type A11. When the server receives the "A11" data from Client 1, it parses the string, looks up the storage address of the A11 model in the database, binds the address to a local byte input stream object FileInputStream, and sends the read A11.pkl file to the client via a network object output stream OutputStream.

[0036] Based on the above method, a TinyML model adaptive management application device includes: at least one memory and at least one processor;

[0037] The at least one memory is used to store a machine-readable program;

[0038] The at least one processor is used to call the machine-readable program to execute a TinyML model adaptive management application method.

[0039] The specific embodiments described above are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the specific embodiments described above. Any appropriate changes or substitutions made by those skilled in the art that conform to the claims of the TinyML model adaptive management application method and apparatus of the present invention should fall within the patent protection scope of the present invention.

[0040] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A TinyML model adaptive management application method, characterized in that, By adopting a client-server architecture, image recognition and data processing with different functions are deployed on different IoT clients. Each client can make a request to the server to remotely download different models according to the different functions deployed, or download the corresponding model in real time according to changes in the operating environment, so as to adapt to the client to implement different functions or changes in the external environment. TinyML models use identifiers to uniquely identify model IDs, converting models from different training frameworks into a unified intermediate format NNEF or ONNX. Then, using pickle, a data persistence method in Python, the model objects are serialized and saved in pkl format, and uploaded to the server for model storage according to a unified classification format. First, the TinyML models are divided according to their functions. After division according to functions, they are further divided according to model size within different function groups. Larger models are used in IoT devices with better CPU performance, while smaller models can be used in IoT devices with weaker MCU performance. After determining the model size, the model selection is further refined based on environmental parameters. The database is a MySQL database. The data structure stored in the MySQL database is as follows: Function represents the function of the model, Size represents the size of the model, Para represents the parameters of the model, and Addr represents the storage location of the model on the server.

2. The TinyML model adaptive management application method according to claim 1, characterized in that, The IoT device includes a light intensity sensor, a humidity sensor, and a camera. The light intensity sensor, humidity sensor, and camera are all connected to an ARM processor, which is also connected to a GPU and a display.

3. The TinyML model adaptive management application method according to claim 2, characterized in that, The system uses an ARM processor to read camera data and sends it to a GPU for inference. It uses a light intensity sensor and a humidity sensor to determine the surrounding environment information and sends an environmental description to the server. The customer selects a function, and the server indexes the database based on the selected function, the environmental description information, and the GPU model. The corresponding model is then pushed to the ARM processor. The ARM processor works with the GPU to complete the inference and count the number of people.

4. The TinyML model adaptive management application method according to claim 3, characterized in that, The IoT terminal pushes video streams and inference results to the server via the network, and the client views them on a web page by connecting to the server. The IoT terminal uses RTMP to push the video stream to the server, and the client can remotely view the video by pulling the stream from the server.

5. The TinyML model adaptive management application method according to claim 4, characterized in that, When a client needs a model of type A11, the client connects to the server via IP and port number. After a successful connection, the client sends the string A11, along with the strings of the environment descriptor and user descriptor, indicating that the client needs a model of type A11. After the server receives the A11 data from the client, it parses the string, looks up the storage address of the A11 model in the database, binds the address to the local byte input stream object FileInputStream, and sends the read A11.pkl file to the client through the network object output stream OutputStream.

6. A TinyML model adaptive management application device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to perform the method according to any one of claims 1 to 5.

Images