Data storage and detection method and device
By adding data type identification to IoT devices and using message queues to push data to the target service of data storage server, the data loss problem when IoT devices and data storage devices are connected concurrently is solved, and data transmission efficiency is achieved and reliability.
Patent Information
- Application Number
- CN201911268198.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-11
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-12-11
AI Technical Summary
When IoT devices are connected concurrently with data storage devices, it may cause excessive load on the data storage device, increasing the risk of data loss.
Record running data in the Internet of Things device and add data type identification. Write the identified data into the message queue through the Internet of Things server. The message queue is pushed to the target service of the data storage server according to the identification, realizing selective transmission and buffering of data , reduce the load on the data storage server.
Effectively reduces data loss, provides buffering for data transmission through message queues and ensures that different services receive appropriate data types, reducing the load on the data storage server.
Smart Images

Figure CN111371832B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data storage and detection method and equipment. Background Art
[0002] With the continuous development of IoT technology, IoT devices can be connected to data storage devices. IoT devices can collect their own operating data and feed the collected operating data back to the data storage device. In this way, by analyzing the data in the data storage device, the operating status of the IoT device can be understood.
[0003] If a data storage device is connected to multiple IoT devices, the operating data of these IoT devices will be fed back to the data storage device. This will result in a large number of concurrent connections to the data storage device, and the amount of data received at the same time may be large, which may lead to a high load on the data storage device and data loss. Summary of the invention
[0004] The purpose of this application is to provide a data storage and detection method and device that can reduce data loss.
[0005] To achieve the above-mentioned purpose, the present application provides a data storage method on the one hand, which includes: recording the operation data of the Internet of Things device during this operation process, and identifying the data type to which the operation data belongs; adding a type identifier matching the data type to the operation data, and uploading the operation data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operation data into a message queue according to the type identifier; wherein the message queue pushes the operation data to the corresponding target service in the data storage server according to the type identifier, so that the operation data is written into the database through the target service.
[0006] To achieve the above-mentioned purpose, the present application also provides an Internet of Things device on the other hand, which includes: a data recording unit, used to record the operation data of the Internet of Things device during this operation process, and identify the data type to which the operation data belongs; a type identification adding unit, used to add a type identification matching the data type to the operation data; a data uploading unit, used to upload the operation data carrying the type identification to the Internet of Things server, so that the Internet of Things server writes the operation data into a message queue according to the type identification; wherein the message queue pushes the operation data to the corresponding target service in the data storage server according to the type identification, so as to write the operation data into the database through the target service.
[0007] To achieve the above-mentioned purpose, the present application also provides an Internet of Things device on the other hand, which includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions: record the operation data of the Internet of Things device during this operation process, and identify the data type to which the operation data belongs; add a type identifier matching the data type to the operation data, and upload the operation data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operation data into a message queue according to the type identifier; wherein the message queue pushes the operation data to the corresponding target service in the data storage server according to the type identifier, so that the operation data is written into the database through the target service.
[0008] To achieve the above-mentioned purpose, the present application also provides a data storage method on the other hand, which includes: receiving operation data carrying a type identifier sent by an Internet of Things server, and writing the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an Internet of Things device and uploaded to the Internet of Things server by the Internet of Things device; identifying the target type identifier corresponding to the target service in the data storage server, and querying in the message queue whether there is target operation data carrying the target type identifier; if so, providing the target operation data to the target service in the data storage server, so that the target operation data is written into the database through the target service.
[0009] To achieve the above-mentioned purpose, the present application also provides a message queue server on the other hand, which includes: a data writing unit, which is used to receive operation data carrying a type identifier sent by an Internet of Things server, and write the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an Internet of Things device and uploaded to the Internet of Things server by the Internet of Things device; a data query unit, which is used to identify the target type identifier corresponding to the target service in the data storage server, and query whether there is target operation data carrying the target type identifier in the message queue; a data providing unit, which is used to provide the target operation data to the target service in the data storage server if it exists, so as to write the target operation data into the database through the target service.
[0010] To achieve the above-mentioned purpose, the present application also provides a message queue server on the other hand, which includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions: receiving operation data carrying a type identifier sent by an Internet of Things server, and writing the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an Internet of Things device and uploaded to the Internet of Things server by the Internet of Things device; identifying the target type identifier corresponding to the target service in the data storage server, and querying in the message queue whether there is target operation data carrying the target type identifier; if so, providing the target operation data to the target service in the data storage server, so that the target operation data can be written into the database through the target service.
[0011] To achieve the above-mentioned purpose, the present application also provides a data detection method on the other hand, which includes: simulating an Internet of Things device to send a first batch of data to an Internet of Things server, and simulating a terminal device to send a second batch of data to the Internet of Things server; reading a third batch of data received by the Internet of Things server from the simulated Internet of Things device, and reading a fourth batch of data received by the Internet of Things server from the simulated terminal device; comparing the first batch of data with the third batch of data, and recording the data lost in the third batch of data compared to the first batch of data; and comparing the second batch of data with the fourth batch of data, and recording the data lost in the fourth batch of data compared to the second batch of data; wherein, a corresponding alarm message is issued for the recorded lost data.
[0012] To achieve the above-mentioned purpose, the present application further provides a data detection device on the other hand, which includes: a batch data sending unit, which is used to simulate an Internet of Things device to send a first batch of data to an Internet of Things server, and simulate a terminal device to send a second batch of data to the Internet of Things server; a data reading unit, which is used to read a third batch of data received by the Internet of Things server from the simulated Internet of Things device, and read a fourth batch of data received by the Internet of Things server from the simulated terminal device; a data comparison unit, which is used to compare the first batch of data with the third batch of data, and record the data lost in the third batch of data compared to the first batch of data; and compare the second batch of data with the fourth batch of data, and record the data lost in the fourth batch of data compared to the second batch of data; wherein, a corresponding alarm message is issued for the recorded lost data.
[0013] To achieve the above-mentioned purpose, the present application further provides a data detection device on the other hand, which includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions: simulate an Internet of Things device to send a first batch of data to an Internet of Things server, and simulate a terminal device to send a second batch of data to the Internet of Things server; read a third batch of data received by the Internet of Things server from the simulated Internet of Things device, and read a fourth batch of data received by the Internet of Things server from the simulated terminal device; compare the first batch of data with the third batch of data, and record the data lost in the third batch of data compared to the first batch of data; and compare the second batch of data with the fourth batch of data, and record the data lost in the fourth batch of data compared to the second batch of data; wherein, a corresponding alarm message is issued for the recorded lost data.
[0014] As can be seen from the above, the technical solution provided by one or more embodiments of the present application can be set up between the Internet of Things device and the data storage server. Among them, after the Internet of Things device collects the operation data, it can add a corresponding type identifier to the operation data according to the data type of the operation data. The Internet of Things device can upload the operation data carrying the type identifier to the Internet of Things server, and then the Internet of Things server can write the received operation data into the message queue of the message queue server according to the type identifier. Subsequently, the message queue can push the operation data to the corresponding target service in the data storage server according to the type identifier, so that different services in the data storage server can receive different operation data. In this way, the message queue can provide a buffer for data transmission, and can selectively provide operation data to the data storage server, thereby reducing the load of the data storage server, and then can reduce the situation of data loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0016] Figure 1 is a schematic diagram of a system architecture in an embodiment of the present invention;
[0017] Figure 2 is a step diagram of a data storage method in an embodiment of the present invention;
[0018] Figure 3It is a schematic diagram of data storage of a message queue in an embodiment of the present invention;
[0019] Figure 4 It is a schematic diagram of the functional modules of the Internet of Things device in an embodiment of the present invention;
[0020] Figure 5 It is a schematic diagram of the structure of an Internet of Things device in an embodiment of the present invention;
[0021] Figure 6 is a schematic diagram of the internal structure of a cleaning device in an embodiment of the present invention;
[0022] Figure 7 It is a step diagram of a data storage method of a message queue server in an embodiment of the present invention;
[0023] Figure 8 It is a step diagram of a data detection method in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in combination with the specific implementation methods of this application and the corresponding drawings. Obviously, the described implementation methods are only part of the implementation methods of this application, not all of the implementation methods. Based on the implementation methods in this application, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0025] The present application provides a data storage method, which can be applied to Figure 1 In the system architecture shown in Figure 1 It may include an IoT device, an IoT server, a message queue server, and a data storage server. Among them, an IoT device may be an electronic device with a network access function. For example, an IoT device may be a cleaning device such as a vacuum cleaner, a cleaning machine, a sweeping robot, etc. equipped with a wireless communication module. The IoT device may collect various operating data during operation, and may upload the collected operating data to the IoT server. After receiving the operating data sent by the IoT device, the IoT server may pass the operating data to the message queue server, thereby writing the operating data into the message queue of the message queue server. Subsequently, the operating data in the message queue may be provided to the data storage server, so that the data storage server writes the operating data into the database for storage.
[0026] In one embodiment of the present application, a data storage method is provided. Figure 2 , the method may include the following steps.
[0027] S11: Record the operation data of the IoT device during this operation, and identify the data type to which the operation data belongs.
[0028] In this embodiment, during the operation of the IoT device, the embedded operating system can query the operation data of each component in the IoT device according to a specified time period. Taking a vacuum cleaner as an example, the embedded operating system installed in the vacuum cleaner can query the current power data from the battery component of the vacuum cleaner every minute after the vacuum cleaner is turned on. In addition, the embedded operating system can also query the dust sensor in the vacuum cleaner every minute for the current accumulated dust amount. In practical applications, the operation data queried by the embedded operating system may include but is not limited to: battery power, motor power, dust removal amount during a single operation, cumulative total historical dust removal amount, single operation time, cumulative total historical operation time, number of faults, fault type, number of device resets, and number of device power on / off times. Of course, with the continuous advancement of technology and the continuous improvement of demand, the embedded operating system can also obtain more operation data of IoT devices. The above-mentioned operation data is only for better explanation of the technical solution of the present application, and does not mean that the technical solution of the present application can only be applied to the above-mentioned operation data.
[0029] In this embodiment, since the components in the IoT device are relatively complex, the data types of the recorded operating data will also be various. For example, the various operating data of a vacuum cleaner can be divided into battery type, dust removal type, fault type, device reset type, motor power type, etc. In the IoT device, the operating data obtained from different components can have different data types. For example, the operating data obtained from the battery component can correspond to the battery type; the operating data obtained from the dust sensor can correspond to the dust removal type. In this way, by identifying the source of each operating data, the data type to which each operating data belongs can be determined.
[0030] S13: Add a type identifier that matches the data type to the operating data, and upload the operating data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operating data into a message queue according to the type identifier; wherein the message queue pushes the operating data to a corresponding target service in a data storage server according to the type identifier, so that the operating data is written into a database through the target service.
[0031] In this embodiment, different data types can be pre-set with different type identifiers. After identifying the data type to which the operating data belongs, a matching type identifier can be added to the operating data. For example, for battery type operating data, a data identifier can be added. <dc1m>For dust removal type operation data, data identification can be added <dtc>.
[0032] In this embodiment, the operation data carrying the type identification can be uploaded to the Internet of Things server by the Internet of Things device. Among them, in the Internet of Things device, different network access modules can be installed according to the use environment of the Internet of Things device. For example, in vacuum cleaners, sweeping robots, and cleaning machines that need to be moved frequently, wireless communication modules such as Wi-Fi, Bluetooth, ZigBee, and ultra-wideband can be installed. In smart rice cookers and smart refrigerators that remain in a fixed position, in addition to the above-mentioned wireless communication modules, wired network cards can also be installed. In this way, after the Internet of Things device obtains the operation data and adds the corresponding type identification to the operation data, it can upload the operation data carrying the type identification to the Internet of Things server through the network access module.
[0033] In this embodiment, after receiving the operation data carrying the type identifier, the IoT server can pass the operation data to the message queue server. In the message queue server, a message queue can be run. The message queue can be implemented in a variety of ways, for example, the message queue can be RocketMQ, MQTT, Kafka, AMQP, MNS, etc. In actual applications, the corresponding implementation method can be flexibly selected according to the application scenario of the message queue.
[0034] In this implementation, after receiving the operation data carrying the type identifier, the message queue server can write the operation data into the message queue according to the respective type identifier. Figure 3 In the message queue, the operation data can be classified according to the type identifier. Each type of operation data can be sorted according to the upload time of the data. Among them, the upload time can be recorded in the timestamp of the operation data, and the timestamp can be indicated when the IoT device uploads the operation data to the IoT server. Of course, in some other implementations, each type of operation data in the message queue can also be sorted according to the time of writing to the message queue, and this application does not limit this. In each type of operation data in the message queue, the earlier the data is written, the more front-end the queue can be placed. In this way, the message queue can follow the first-in-first-out principle and provide the operation data to the data storage server in sequence according to the sorting results.
[0035] In this embodiment, a plurality of different services may be run in the data storage server, and each service may have its own type identifier. In this way, each service may obtain the operation data that matches its own type identifier from the message queue, rather than obtaining all the operation data in the message queue. For example, a service for analyzing dust removal information may be run in the data storage server, and the type identifier of the service may be <dtc>, then the service can obtain the dust removal type operation data from the message queue. For another example, a service for analyzing battery information is running in the data storage server, and the type identifier of the service is <dc1m>, then the service can obtain the operating data of the battery type from the message queue.
[0036] In this embodiment, each service in the data storage server can pre-register its own key and secret in the message queue server, where the key can be used as the service account name and the secret can be used as the service password. The service in the data storage server can use the registered account name and password to establish a persistent connection with the message queue after being authenticated by the message queue server. Subsequently, the running data in the message queue can be pushed to the corresponding service in the data storage server through the established persistent connection.
[0037] Specifically, for the target service in the data storage server, after the message queue establishes a long connection with the target service, the target type identifier corresponding to the target service can be identified through the negotiation data obtained during the long connection establishment process. Then, the message queue can query whether there is target operation data carrying the target type identifier in the message queue at a certain time period. If so, the target operation data can be provided to the target service through the established long connection. Subsequently, the target service can write the target operation data into the database, thereby completing the storage process of the target operation data.
[0038] In one embodiment, when the message queue provides operation data to the service in the data storage server, if the service in the data storage server can normally receive the operation data, then after receiving the operation data, the service in the data storage server can feedback a confirmation statement to the message queue. When the message queue receives the confirmation statement, it determines that the current operation data has been transmitted, so that the operation data can be deleted from the message queue to avoid repeated transmission. If the message queue cannot receive a confirmation statement from the service of the data storage server within the specified time, it can be considered that the transmission has timed out and the service in the data storage server cannot receive the currently transmitted operation data. At this time, the message queue can cancel the data transmission process this time and keep the currently transmitted operation data in the message queue. Subsequently, when the next data push cycle arrives, it can continue to try to provide the operation data to the service in the data storage server within the data push cycle until the feedback confirmation statement is received.
[0039] In actual applications, IoT devices may be started and stopped multiple times by users unintentionally, resulting in a relatively short startup time each time. In these processes, the data generated is not valid data. If this part of the data is uploaded to the IoT server, invalid data will be generated. In view of this, in one embodiment, after the IoT device is started, the duration of this startup can be counted through the built-in timer, so that it can be determined whether the duration of the IoT device's operation this time reaches the specified duration threshold. If the specified duration threshold is reached, it means that this operation is normal operation and valid data is generated, so that the operation data of the IoT device during this operation can be recorded. If the duration of the IoT device's operation this time has not reached the specified duration threshold and it is shut down, it means that this operation may be a misoperation, so the operation data generated during this operation can be not recorded.
[0040] In a specific application scenario, the aforementioned IoT server, message queue server, data storage server, and database can all be deployed in the cloud. IoT devices can communicate data with cloud servers through network access components, thereby uploading operating data. In addition, the aforementioned IoT server, message queue server, and data storage server can be independent servers or different services deployed on the same or multiple servers. For example, the IoT server and the message queue server can be different services deployed in the same cloud server, or they can be two independent servers.
[0041] In a specific scenario example, taking a vacuum cleaner as an example, a Wi-Fi module, a memory, and an embedded operating system can be installed in the vacuum cleaner. The embedded operating system can obtain various operating data such as battery discharge data, dust removal data, fault data, and device reset data by accessing various components in the vacuum cleaner. The obtained operating data can be stored in the local memory of the vacuum cleaner, and can also be uploaded to the IoT server in the cloud. Taking battery discharge data as an example, when the vacuum cleaner is turned on, the embedded operating system can query the power data from the battery component every minute to obtain the current battery power percentage. In addition, the embedded operating system can also query the number of battery discharges by querying the power data, and the number of battery discharges can be the number of times the vacuum cleaner is turned on. Since the embedded operating system can obtain different battery discharge data at different time nodes, when querying the battery discharge data, a corresponding timestamp can also be generated, which can represent the time node when the battery discharge data is collected. The time node can be represented by the system time of the embedded operating system, and can also be represented by the running time of the vacuum cleaner startup. For example, when the embedded operating system collects battery discharge data at 15:03 system time, the generated timestamp can be 15:03 (of course, the specific date can also be attached). For another example, if the embedded operating system collects battery discharge data when the vacuum cleaner is just turned on, the generated timestamp can be 0, and if the battery discharge data is collected 1 minute after the power-on, the generated timestamp can be 1. In this way, after querying the battery component, the embedded operating system can generate corresponding battery discharge data, and the battery discharge data can include a device usage count field, a collection time field, and a battery power field, wherein the device usage count field can be used to represent the total number of historical startup times of the vacuum cleaner, the collection time field can be used to represent the time node when the battery discharge data is collected, and the battery power field can be used to represent the battery power percentage when the battery discharge data is recorded.
[0042] In a practical example, the battery discharge data generated by the embedded operating system can be expressed as {"dctx":"12","dmx":"0","bp":"4"}, where "dctx":"12" means the battery discharge times are 12 times, "dmx":"0" means the battery discharge data is collected when the vacuum cleaner is just turned on, and "bp":"4" means the power percentage when the battery discharge data is collected is 4%. After the battery discharge data is generated, a type identifier can be added to the battery discharge data. <dc1m>,Afterwards, the vacuum cleaner can upload the battery discharge data carrying the ,type identification to the IoT server through the WiFi module.
[0043] In another specific application scenario, the vacuum cleaner can also record the single dust removal amount generated during this operation and the historical dust removal amount of the vacuum cleaner. Specifically, when the vacuum cleaner is running, the embedded operating system can obtain the current dust removal amount by accessing the dust sensor. When the vacuum cleaner stops running, the embedded operating system can count the single dust removal amount generated during this operation. On the basis of the historical dust removal amount, the single dust removal amount is added to obtain the actual historical dust removal amount of the vacuum cleaner after this operation. A memory can be set in the vacuum cleaner, and the memory can be a component with data storage function. For example, the memory can be a random access memory (RAM), or a U disk, hard disk, etc. connected to the vacuum cleaner. Due to the limited storage space of the memory, in actual applications, only the historical dust removal amount calculated after each operation can be stored in the memory, and the single dust removal amount can be not stored. However, after each operation, the single dust removal amount and the historical dust removal amount of the vacuum cleaner need to be uploaded to the Internet of Things server, so that the single dust removal amount and the historical dust removal amount are finally backed up through the database. In this way, after obtaining the single dust removal amount and the historical dust removal amount, the vacuum cleaner can add a type identifier for the single dust removal amount and the historical dust removal amount. <dtc>, and added <dtc>The single dust removal amount and historical dust removal amount are uploaded to the IoT server.
[0044] For other IoT devices, the above-mentioned single dust removal amount may correspond to the single data amount, and the above-mentioned historical dust removal amount may correspond to the total historical data amount. In this way, the operation data uploaded by the IoT device may include the single data amount generated by the IoT device during this operation and the total historical data amount of the IoT device, wherein the total historical data amount of the IoT device may be stored in the memory of the IoT device.
[0045] In one embodiment, the operation data uploaded by the IoT device can be represented by a numerical value of a specified number of bits. For example, the historical dust removal amount uploaded by the vacuum cleaner can be represented by 32-bit binary data. However, as time continues to accumulate, the numerical value of the operation data may be relatively large, and the numerical value of the specified number of bits may not be able to represent the larger numerical value. At this time, the IoT device can split the recorded operation data into multiple sub-values. For example, after the operation data is converted to binary, 40 bits are required to represent it. At this time, the binary value can be split into two sub-values of 8 bits and 32 bits. In order to correctly splice the complete operation data later, the IoT device can add a corresponding numerical number to each sub-value, wherein the numerical number can be set according to the position of the sub-value in the complete operation data. For example, numerical numbers can be set for each sub-value in sequence starting from 0, and the more forward the sub-value, the smaller the corresponding numerical number can be. Subsequently, the IoT device can upload the sub-value with the added numerical number as operation data to the IoT server. The operation data can eventually be received by the target service in the data storage server. When the target service writes the operation data to the database, the sub-values can be spliced into a complete value according to the numerical number, so that the spliced complete value can be written into the database.
[0046] In one embodiment, the present application also provides an Internet of Things device, see Figure 4 , the IoT device may include the following components.
[0047] A data recording unit, used to record the operation data of the IoT device during this operation and identify the data type to which the operation data belongs;
[0048] A type identification adding unit, used for adding a type identification matching the data type to the operation data;
[0049] A data uploading unit is used to upload the operation data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operation data into a message queue according to the type identifier; wherein the message queue pushes the operation data to a corresponding target service in a data storage server according to the type identifier, so that the operation data is written into a database through the target service.
[0050] See also Figure 5 The present application also provides an Internet of Things device, the Internet of Things device comprising a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions:
[0051] Record the operation data of the IoT device during this operation and identify the data type of the operation data;
[0052] Adding a type identifier matching the data type to the operation data, and uploading the operation data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operation data into a message queue according to the type identifier;
[0053] The message queue pushes the operation data to a corresponding target service in a data storage server according to the type identifier, so that the operation data is written into a database through the target service.
[0054] Specifically, see Figure 6 , the above-mentioned IoT device can be a cleaning device such as a vacuum cleaner, a cleaning machine, or a sweeping robot. The cleaning device may include one or more processors 100, one or more memories 101 storing computer programs, and one or more sensors 102. One or more sensors 102 are used to collect data related to the cleaning device itself and data related to its surrounding environment during the movement of the cleaning device. In addition, the cleaning device may also include necessary components such as an audio and video component 103, a power supply component 104, etc. One or more processors 100 are used to execute the computer program stored in the memory 101. When the computer program is executed, the above-mentioned data storage method can be implemented.
[0055] In this embodiment, the memory in the IoT device may include a physical device for storing information, which is usually a device that digitizes the information and then stores it in a medium using electrical, magnetic or optical methods. The memory may include: a device that stores information using electrical energy, such as RAM, ROM, etc.; a device that stores information using magnetic energy, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device that stores information using optical means, such as a CD or a DVD. Of course, there are other types of memory, such as quantum memory, graphene memory, etc.
[0056] In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc.
[0057] See also Figure 7 , the present application also provides a data storage method, which may include the following steps.
[0058] S21: receiving operation data carrying a type identifier sent by the IoT server, and writing the operation data into a message queue according to the type identifier; wherein the operation data is recorded by the IoT device and uploaded to the IoT server by the IoT device;
[0059] S23: Identify the target type identifier corresponding to the target service in the data storage server, and query whether there is target operation data carrying the target type identifier in the message queue;
[0060] S25: If so, providing the target operation data to the target service in the data storage server, so that the target operation data is written into a database through the target service.
[0061] In a specific application scenario, after the message queue server receives the operating data carrying the type identifier sent by the Internet of Things server, it can classify and store the operating data in the message queue. After the target service in the data storage server passes the authentication of the message queue through the preset authentication method, it can establish a long connection with the message queue. Subsequently, the message queue can provide the corresponding target operating data to the target service through the established long connection according to the type identifier of the target service. Among them, the preset authentication method can be the above-mentioned method of registering an account name and password in the message queue server. Of course, in actual applications, more authentication methods can be included, and this application does not limit this.
[0062] In a specific application scenario, taking a vacuum cleaner as an example, two memories can be installed in the vacuum cleaner, and the historical dust removal amount of the vacuum cleaner can be written into these two memories. At the same time, the historical dust removal amount in only one memory can be uploaded to the IoT server, and the other memory can be used as a backup server. However, in actual applications, a certain memory may fail during operation, resulting in the latest stored historical dust removal amount not being uploaded to the IoT server normally, but uploading the last stored historical dust removal amount to the IoT server. In this way, the historical dust removal amount currently received by the IoT server may be less than or equal to the historical dust removal amount received last time. In this case, it can be determined that one of the memories in the vacuum cleaner has failed, and the historical dust removal amount currently received is inaccurate and needs to be corrected. Specifically, in addition to receiving the historical dust removal amount, the IoT server can also receive the single dust removal amount during the current operation of the vacuum cleaner. Since the single dust removal amount will not be stored in the memory of the vacuum cleaner, but directly uploaded to the IoT server, the value of the single dust removal amount is usually accurate. In view of this, the current actual historical dust removal amount can be calculated based on the single dust removal amount currently received and the historical dust removal amount received last time. Specifically, the current actual historical dust removal amount may be the sum of the currently received single dust removal amount and the last received historical dust removal amount. In this way, after the actual historical dust removal amount is calculated, the actual historical dust removal amount may be used to replace the currently received historical dust removal amount, thereby completing the data correction, so that the operation data finally stored in the database is accurate operation data.
[0063] The single dust removal amount of the above vacuum cleaner can correspond to the single data amount of other IoT devices, and the historical dust removal amount can correspond to the total historical data amount of other IoT devices. In this way, if the total historical data amount currently received is less than or equal to the total historical data amount received last time, the current actual total historical data amount can be calculated based on the single data amount currently received and the total historical data amount received last time, and the actual total historical data amount can be used to replace the current total historical data amount received.
[0064] The present application also provides a message queue server, the message queue server comprising:
[0065] A data writing unit, used for receiving operation data carrying a type identifier sent by an IoT server, and writing the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an IoT device and uploaded to the IoT server by the IoT device;
[0066] A data query unit, used to identify a target type identifier corresponding to a target service in a data storage server, and query whether there is target operation data carrying the target type identifier in the message queue;
[0067] A data providing unit is used to provide the target operation data to the target service in the data storage server, if any, so as to write the target operation data into a database through the target service.
[0068] The present application also provides a message queue server, the message queue server comprising a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions:
[0069] Receive operation data carrying a type identifier sent by an IoT server, and write the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an IoT device and uploaded to the IoT server by the IoT device;
[0070] Identify the target type identifier corresponding to the target service in the data storage server, and query whether there is target operation data carrying the target type identifier in the message queue;
[0071] If so, the target operation data is provided to the target service in the data storage server, so that the target operation data is written into a database through the target service.
[0072] In one embodiment, the IoT device and the user's terminal device can transmit data and communicate instructions through the IoT server. However, since the current IoT device and the user's terminal device usually transmit data through the MQTT service, and the MQTT service only transmits data once, if it is affected by network fluctuations, it is likely to cause data loss. In view of this, in this embodiment, a data detection method can be provided to detect possible data loss during data transmission. For details, please refer to Figure 8 , the data detection method may include the following steps.
[0073] S31: Simulate an IoT device to send a first batch of data to an IoT server, and simulate a terminal device to send a second batch of data to the IoT server.
[0074] S33: reading the third batch of data received by the Internet of Things server from the simulated Internet of Things device, and reading the fourth batch of data received by the Internet of Things server from the simulated terminal device.
[0075] S35: Compare the first batch of data with the third batch of data, and record the data lost in the third batch of data compared with the first batch of data; and compare the second batch of data with the fourth batch of data, and record the data lost in the fourth batch of data compared with the second batch of data; wherein, corresponding alarm information is issued for the recorded lost data.
[0076] In this embodiment, in order to test the data transmission performance of the current system architecture, the Internet of Things device and the terminal device can be simulated by a test tool such as jmeter to send batch data. Specifically, when simulating an Internet of Things device, a first batch of data can be sent to the Internet of Things server, and the first batch of data may be received in whole or in part by the Internet of Things server. At this time, the third batch of data received by the Internet of Things server from the simulated Internet of Things device can be read by the test tool, and the first batch of data and the third batch of data can be compared, so that the missing data in the third batch of data can be found. This part of the missing data can be recorded, and a corresponding alarm message can be generated. By consulting the alarm message and the recorded missing data, the cause of the data loss can be analyzed, so that the existing system architecture can be improved.
[0077] Similarly, when simulating a terminal device, a second batch of data may be sent to the IoT server, and the second batch of data may be received in whole or in part by the IoT server. At this time, the test tool may read the fourth batch of data received by the IoT server from the simulated terminal device, and the second batch of data and the fourth batch of data may be compared, so as to find the missing data in the fourth batch of data. This part of the missing data may be recorded, and a corresponding alarm message may be generated.
[0078] In one embodiment, the transmission status of the data instruction between the IoT server and the terminal device can also be detected. When the IoT server sends a data instruction to the terminal device, the first data instruction sent from the IoT server can be recorded, and the second data instruction received by the terminal device from the IoT server can be read. Then, the first data instruction and the second data instruction can be compared, and the data instruction lost in the second data instruction compared to the first data instruction can be recorded. For the recorded lost data instruction, a corresponding alarm message can also be issued. In this way, for the IoT device, the IoT server, and the terminal device, no matter which party loses data, the lost data can be recorded and the corresponding alarm message can be generated, which helps to improve the existing system architecture to alleviate the data loss.
[0079] In one embodiment, the Internet of Things server can also be stress tested to determine the maximum amount of data that the Internet of Things server can withstand when dealing with the reception of batch data. Specifically, the test tool can generate multiple batches of batch detection data with different data volumes, and the data volumes of these batch detection data can be gradually increased. Then, these batch detection data with different data volumes can be sent to the Internet of Things server in order from small to large data volumes, and the data loss rate of the Internet of Things server when receiving these batch detection data can be recorded. Subsequently, the recorded data loss rate can be compared with the preset loss rate threshold to determine the maximum amount of data that the Internet of Things server can withstand. Specifically, the data loss rate greater than or equal to the preset loss rate threshold can be counted in the recorded data loss rate threshold. Among them, the preset loss rate threshold can be used as the maximum loss rate that can be tolerated during data transmission. If it reaches or exceeds the preset loss rate threshold, it means that the data transmission state is not ideal. The preset loss rate threshold can be flexibly set according to actual needs. After the data loss rate greater than or equal to the preset loss rate threshold is obtained by statistics, the minimum target data loss rate in the statistical data loss rate can be identified. Then, the batch detection data corresponding to the target data loss rate can be determined, and the data volume of the determined batch detection data is used as the maximum data volume of the IoT server. In this way, when the maximum data volume of the IoT server is determined, it is possible to avoid sending data exceeding the maximum data volume to the IoT server in the same batch, thereby reducing data loss.
[0080] The present application also provides a data detection device, the device comprising:
[0081] A batch data sending unit, used to simulate an Internet of Things device sending a first batch of data to an Internet of Things server, and simulate a terminal device sending a second batch of data to the Internet of Things server;
[0082] A data reading unit, configured to read a third batch of data received by the IoT server from the simulated IoT device, and to read a fourth batch of data received by the IoT server from the simulated terminal device;
[0083] A data comparison unit is used to compare the first batch of data with the third batch of data, and record the data lost in the third batch of data compared with the first batch of data; and to compare the second batch of data with the fourth batch of data, and record the data lost in the fourth batch of data compared with the second batch of data; wherein a corresponding alarm message is issued for the recorded lost data.
[0084] The present application also provides a data detection device, the data detection device comprising a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions:
[0085] Simulate an IoT device to send a first batch of data to an IoT server, and simulate a terminal device to send a second batch of data to the IoT server;
[0086] Reading a third batch of data received by the IoT server from the simulated IoT device, and reading a fourth batch of data received by the IoT server from the simulated terminal device;
[0087] The first batch of data is compared with the third batch of data, and data lost in the third batch of data compared with the first batch of data is recorded; and the second batch of data is compared with the fourth batch of data, and data lost in the fourth batch of data compared with the second batch of data is recorded; wherein corresponding alarm information is issued for the recorded lost data.
[0088] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0089] As can be seen from the above, the technical solution provided by one or more embodiments of the present application can be set up between the Internet of Things device and the data storage server. Among them, after the Internet of Things device collects the operation data, it can add a corresponding type identifier to the operation data according to the data type of the operation data. The Internet of Things device can upload the operation data carrying the type identifier to the Internet of Things server, and then the Internet of Things server can write the received operation data into the message queue of the message queue server according to the type identifier. Subsequently, the message queue can push the operation data to the corresponding target service in the data storage server according to the type identifier, so that different services in the data storage server can receive different operation data. In this way, the message queue can provide a buffer for data transmission, and can selectively provide operation data to the data storage server, thereby reducing the load of the data storage server, and then can reduce the situation of data loss.
[0090] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0095] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0096] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transit media), such as modulated data signals and carrier waves.
[0097] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.
[0098] The above is only the implementation mode of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.< / dtc> < / dtc> < / dtc> < / dtc>
Claims
1. A data storage method, It is characterized in that The method comprises: The IoT device records the operation data during this operation and identifies the data type to which the operation data belongs; the data type is divided according to the component type in the IoT device; The IoT device adds a type identifier that matches the data type to the operation data, and uploads the operation data carrying the type identifier to the IoT server, so that the IoT server writes the operation data into a message queue according to the type identifier; The message queue pushes the operation data to the corresponding target service in the data storage server through a long connection according to the type identifier, so that the operation data is written into the database through the target service.
2. The method according to claim 1, It is characterized in that The operation data of IoT devices recorded during this operation include: Determine whether the duration of the current operation of the IoT device reaches a specified duration threshold. If the specified duration threshold is reached, record the operation data of the IoT device during the current operation.
3. The method according to claim 1, It is characterized in that The operation data includes battery discharge data, and the battery discharge data includes a device usage count field, a collection time field, and a battery power field, wherein the device usage count field is used to represent the total number of historical startup times of the IoT device, the collection time field is used to represent the time node when the battery discharge data is collected, and the battery power field is used to represent the battery power percentage when the battery discharge data is recorded.
4. The method according to claim 1, It is characterized in that The operation data includes a single amount of data generated by the IoT device during this operation and a total amount of historical data of the IoT device, wherein the total amount of historical data of the IoT device is stored in a memory of the IoT device.
5. The method according to claim 1, It is characterized in that If the operating data cannot be represented by a numerical value with a specified number of bits, the method further includes: Splitting the recorded operation data into multiple sub-values, adding a corresponding value number to each of the sub-values, and uploading the sub-values with the added value numbers as operation data to the Internet of Things server; When the target service writes the operating data into a database, each of the sub-values is concatenated into a complete value according to the value number, and the complete value is written into the database.
6. The method according to claim 1, It is characterized in that The operating data includes at least one of battery power, motor power, dust removal amount during a single operation, cumulative historical dust removal amount, single operation time, cumulative historical operation time, number of faults, type of fault, number of device resets, and number of device power on and off times.
7. An IoT device, It is characterized in that The IoT devices include: A data recording unit, used to record the operation data of the IoT device during this operation and identify the data type to which the operation data belongs; the data type is divided according to the component type in the IoT device; A type identification adding unit, used for adding a type identification matching the data type to the operation data; A data uploading unit is used to upload the operating data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operating data into a message queue according to the type identifier; wherein the message queue pushes the operating data to the corresponding target service in the data storage server through a long connection according to the type identifier, so that the operating data is written into the database through the target service.
8. An IoT device, It is characterized in that The IoT device includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the computer program is used to implement the following functions: Recording the operation data of the IoT device during this operation, and identifying the data type to which the operation data belongs; the data type is divided according to the component type in the IoT device; Adding a type identifier matching the data type to the operation data, and uploading the operation data carrying the type identifier to the Internet of Things server, so that the Internet of Things server writes the operation data into a message queue according to the type identifier; The message queue pushes the operation data to the corresponding target service in the data storage server through a long connection according to the type identifier, so that the operation data is written into the database through the target service.
9. A data storage method, It is characterized in that The method comprises: Receive operation data carrying a type identifier sent by an IoT server, and write the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an IoT device and uploaded to the IoT server by the IoT device; the type identifier matches the data type to which the operation data belongs, and the data type is divided according to the component type in the IoT device; Identify the target type identifier corresponding to the target service in the data storage server, and query whether there is target operation data carrying the target type identifier in the message queue; If so, the target operation data is provided to the target service in the data storage server through a long connection, so that the target operation data is written into the database through the target service.
10. The method according to claim 9, It is characterized in that The method further comprises: If the target service cannot receive the target operation data, retain the target operation data in the message queue, and provide the target operation data to the target service again in the next data push cycle; If the target operation data is received by the target service, the target operation data is deleted from the message queue.
11. The method according to claim 9, It is characterized in that The operation data stored in the message queue is classified according to the type identifier; wherein each type of operation data is sorted according to the upload time, and the operation data of the same type is provided to the data storage server in sequence according to the sorting result.
12. The method according to claim 9, It is characterized in that After the target service passes the authentication of the message queue through a preset authentication method, the long connection is established with the message queue.
13. The method according to claim 9, It is characterized in that The operation data includes the single data volume generated by the IoT device during this operation and the total historical data volume of the IoT device; The method further comprises: If the total amount of historical data currently received is less than or equal to the total amount of historical data received last time, calculate the current actual total amount of historical data based on the current amount of single data received and the total amount of historical data received last time, and use the actual total amount of historical data to replace the current total amount of historical data received.
14. A message queue server, It is characterized in that The message queue server comprises: A data writing unit, used for receiving operation data carrying a type identifier sent by an IoT server, and writing the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an IoT device and uploaded to the IoT server by the IoT device; the type identifier matches the data type to which the operation data belongs, and the data type is divided according to the component type in the IoT device; A data query unit, used to identify a target type identifier corresponding to a target service in a data storage server, and query whether there is target operation data carrying the target type identifier in the message queue; A data providing unit is used to provide the target operation data to the target service in the data storage server through a long connection, if any, so as to write the target operation data into a database through the target service.
15. A message queue server, It is characterized in that The message queue server includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions: Receive operation data carrying a type identifier sent by an IoT server, and write the operation data into a message queue according to the type identifier; wherein the operation data is recorded by an IoT device and uploaded to the IoT server by the IoT device; the type identifier matches the data type to which the operation data belongs, and the data type is divided according to the component type in the IoT device; Identify the target type identifier corresponding to the target service in the data storage server, and query whether there is target operation data carrying the target type identifier in the message queue; If so, the target operation data is provided to the target service in the data storage server through a long connection, so that the target operation data is written into the database through the target service.
16. A data detection method, It is characterized in that The method comprises: Simulate an IoT device to send a first batch of data to an IoT server, and simulate a terminal device to send a second batch of data to the IoT server; Reading a third batch of data received by the IoT server from the simulated IoT device, and reading a fourth batch of data received by the IoT server from the simulated terminal device; comparing the first batch of data with the third batch of data, and recording the data lost in the third batch of data compared with the first batch of data; and comparing the second batch of data with the fourth batch of data, and recording the data lost in the fourth batch of data compared with the second batch of data; wherein corresponding alarm information is issued for the recorded lost data; The IoT server is used to receive the operation data carrying the type identifier sent by the IoT device, and write the operation data into the message queue according to the type identifier; the type identifier matches the data type to which the operation data belongs, and the data type is divided according to the component type in the IoT device; The message queue is used to push the operation data to the corresponding target service in the data storage server through a long connection according to the type identifier, so as to write the operation data into the database through the target service.
17. The method according to claim 16, It is characterized in that The method further comprises: When the Internet of Things server sends a data instruction to the terminal device, a first data instruction sent from the Internet of Things server is recorded, and a second data instruction received by the terminal device from the Internet of Things server is read; The first data instruction and the second data instruction are compared, and the data instruction lost in the second data instruction compared with the first data instruction is recorded, and corresponding alarm information is issued for the recorded lost data instruction.
18. The method according to claim 16, It is characterized in that The method further comprises: Sending batch detection data with different data volumes to the Internet of Things server in sequence, and recording the data loss rate of the Internet of Things server when receiving the batch detection data; The recorded data loss rate is compared with a preset loss rate threshold to determine the maximum amount of data that the IoT server can bear.
19. The method according to claim 18, It is characterized in that Determining the maximum amount of data that the IoT server can bear includes: Among the recorded data loss rate thresholds, counting data loss rates greater than or equal to the preset loss rate threshold, and identifying the smallest target data loss rate among the counted data loss rates; The batch detection data corresponding to the target data loss rate is determined, and the data volume of the determined batch detection data is used as the maximum data volume to be tolerated by the Internet of Things server.
20. A data detection device, It is characterized in that The device comprises: A batch data sending unit, used to simulate an Internet of Things device sending a first batch of data to an Internet of Things server, and simulate a terminal device sending a second batch of data to the Internet of Things server; A data reading unit, configured to read a third batch of data received by the IoT server from the simulated IoT device, and to read a fourth batch of data received by the IoT server from the simulated terminal device; a data comparison unit, configured to compare the first batch of data with the third batch of data, and record the data lost in the third batch of data compared with the first batch of data; and to compare the second batch of data with the fourth batch of data, and record the data lost in the fourth batch of data compared with the second batch of data; wherein a corresponding alarm message is issued for the recorded lost data; The IoT server is used to receive the operation data carrying the type identifier sent by the IoT device, and write the operation data into the message queue according to the type identifier; the type identifier matches the data type to which the operation data belongs, and the data type is divided according to the component type in the IoT device; The message queue is used to push the operation data to the corresponding target service in the data storage server through a long connection according to the type identifier, so as to write the operation data into the database through the target service.
21. A data detection device, It is characterized in that The data detection device includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions: Simulate an IoT device to send a first batch of data to an IoT server, and simulate a terminal device to send a second batch of data to the IoT server; Reading a third batch of data received by the IoT server from the simulated IoT device, and reading a fourth batch of data received by the IoT server from the simulated terminal device; comparing the first batch of data with the third batch of data, and recording data lost in the third batch of data compared to the first batch of data; and comparing the second batch of data with the fourth batch of data, and recording the data lost in the fourth batch of data compared with the second batch of data; wherein corresponding alarm information is issued for the recorded lost data; The IoT server is used to receive the operation data carrying the type identifier sent by the IoT device, and write the operation data into the message queue according to the type identifier; the type identifier matches the data type to which the operation data belongs, and the data type is divided according to the component type in the IoT device; The message queue is used to push the operation data to the corresponding target service in the data storage server through a long connection according to the type identifier, so as to write the operation data into the database through the target service.
