A data analysis method and device based on an internet of things device and the internet of things device

Abstract

The embodiment of the application discloses a data analysis method and device based on an Internet of Things equipment and the Internet of Things equipment; the embodiment of the application can detect instructions generated by an application program of the Internet of Things equipment; when the instruction type of the instructions is an object operation type, a first signal and a second signal are generated; the first signal is used to trigger a data collection module of the Internet of Things equipment to collect buried point data corresponding to the instructions, wherein the buried point data represents operations performed by an object on the application program of the Internet of Things equipment; the second signal is used to trigger an image collection module of the Internet of Things equipment to collect object image data of the object; the buried point data is subjected to data expansion processing according to the object image data, and expanded buried point data is obtained; the expanded buried point data is reported, so that the application program of the Internet of Things equipment is subjected to object operation analysis based on the expanded buried point data, and the accuracy and reliability of data analysis of the Internet of Things equipment can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a data analysis method, apparatus, and IoT device based on an Internet of Things (IoT) device. Background Technology

[0002] The Internet of Things (IoT), or "Internet of Everything," is a network extending and expanding upon the internet. It combines various information sensing devices with the network to form a vast network, enabling interconnection and interoperability between people, machines, and things anytime, anywhere. In IoT scenarios, when IoT devices malfunction, feedback typically relies on the user's problem description. However, differences in user perception can lead to incomplete and inaccurate feedback, hindering data analysis of IoT devices. Summary of the Invention

[0003] This application provides a data analysis method, apparatus, IoT device, and storage medium based on IoT devices, which can improve the accuracy and reliability of data analysis on IoT devices.

[0004] This application provides a data analysis method based on Internet of Things (IoT) devices, including:

[0005] Detect instructions generated by applications in IoT devices;

[0006] When the instruction type of the instruction is detected to be an object operation type, a first signal and a second signal are generated;

[0007] The first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device;

[0008] The second signal is used to trigger the image acquisition module of the IoT device to acquire object image data of the object;

[0009] Based on the object image data, the embedded data is augmented to obtain augmented embedded data;

[0010] The expanded data points are reported so that the application of the IoT device can perform object operation analysis based on the expanded data points.

[0011] Accordingly, embodiments of this application also provide a data analysis device based on an Internet of Things (IoT) device, comprising:

[0012] The detection unit is used to detect the instructions generated by the application of the Internet of Things (IoT) device.

[0013] The generation unit is used to generate a first signal and a second signal when the instruction type of the instruction is detected to be an object operation type;

[0014] A data acquisition unit is used to trigger the data acquisition module of the Internet of Things device to acquire the embedded data corresponding to the instruction using the first signal, wherein the embedded data represents the operation performed by the object on the application of the Internet of Things device;

[0015] The image acquisition unit is used to trigger the image acquisition module of the IoT device to acquire object image data of the object using the second signal;

[0016] An expansion unit is used to perform data expansion processing on the embedded data based on the object image data to obtain expanded embedded data;

[0017] The reporting unit is used to report the expanded embedded data so that the application of the Internet of Things device can be used for object operation analysis based on the expanded embedded data.

[0018] In one embodiment, the data acquisition unit may include:

[0019] The data acquisition subunit is used to trigger the data acquisition module of the IoT device to acquire the embedded data stream using the first signal, and add the embedded data stream to the embedded data queue;

[0020] The first determining subunit is used to determine the queue position identifier of the embedded data stream in the embedded data queue;

[0021] The second determining subunit is used to determine the operation information corresponding to the instruction based on the embedded data stream;

[0022] The first generation subunit is used to generate the operation identifier corresponding to the embedded data stream;

[0023] The integration subunit is used to integrate the operation identifier, the operation information, and the queue position identifier to obtain the embedded data.

[0024] In one embodiment, the expansion unit may include:

[0025] The first addition subunit is used to add an image field to the embedded data to obtain the embedded data after addition.

[0026] The second adding subunit is used to add the object image data to the added tracking data based on the image field, so as to obtain the expanded tracking data.

[0027] In one embodiment, the image acquisition unit may include:

[0028] An image acquisition subunit is used to trigger the image acquisition module of the Internet of Things device to acquire object video using the second signal, the object video including multiple object video frames;

[0029] The comparison subunit is used to compare the image quality of the video frames of the object and obtain the comparison result;

[0030] A filtering subunit is used to filter out a reference object image of the object from the plurality of video frames based on the comparison results;

[0031] The attribute transformation subunit is used to perform attribute transformation on the reference object image to obtain the object image data.

[0032] In one embodiment, the attribute conversion subunit may include:

[0033] The recognition module is used to recognize the reference object image and obtain the image attribute information corresponding to the reference object image;

[0034] A compression module is used to compress the reference object image based on the image attribute information to obtain a compressed reference object image;

[0035] The serialization module is used to serialize the compressed reference object image to obtain the object image data.

[0036] In one embodiment, the reporting unit may include:

[0037] The third addition subunit is used to add the expanded tracking data to the tracking queue;

[0038] The data encryption subunit is used to encrypt the data in the data collection queue to obtain the encrypted data collection queue.

[0039] The reporting subunit is used to report the encrypted tracking data queue so that the application of the Internet of Things device can be analyzed for object operations based on the expanded tracking data of the encrypted tracking data queue.

[0040] In one embodiment, the third adding subunit may include:

[0041] The generation module is used to generate an instruction type identifier based on the instruction type corresponding to the instruction;

[0042] The first adding module is used to add the instruction type identifier to the expanded embedding data to obtain operation embedding data;

[0043] The second adding module is used to add the operation tracking data to the tracking queue.

[0044] In one embodiment, the detection unit may include:

[0045] The detection subunit is used to call the event detector to detect the instructions generated by the application of the IoT device.

[0046] In one embodiment, the generation unit may include:

[0047] A generation subunit is configured to generate a first signal and a second signal when the event detector detects that the instruction type of the instruction is an object operation type.

[0048] In one embodiment, the data analysis apparatus may further include:

[0049] The parsing unit is used to parse the detected instructions to obtain the instruction type corresponding to the instructions;

[0050] The matching unit is used to match the instruction type with the preset object operation type to obtain a matching result;

[0051] The determining unit is used to determine whether the instruction type of the instruction is an object operation type based on the matching result.

[0052] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an IoT device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the IoT device to perform the methods provided in the various alternative embodiments described above.

[0053] Accordingly, this application embodiment also provides a storage medium storing instructions, which, when executed by a processor, implement any of the data analysis methods based on IoT devices provided in this application embodiment.

[0054] This application embodiment can detect instructions generated by the application of an IoT device; when the instruction type of the detected instruction is an object operation type, a first signal and a second signal are generated; the first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device; the second signal is used to trigger the image acquisition module of the IoT device to collect the object image data of the object; based on the object image data, the embedded data is subjected to data augmentation processing to obtain augmented embedded data; the augmented embedded data is reported so that object operation analysis of the IoT device application based on the augmented embedded data can improve the accuracy and reliability of IoT device data analysis. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram illustrating a scenario of the data analysis method based on IoT devices provided in an embodiment of this application.

[0057] Figure 2 This is a flowchart illustrating the data analysis method based on IoT devices provided in an embodiment of this application;

[0058] Figure 3 This is another scenario illustration of the data analysis method based on IoT devices provided in the embodiments of this application;

[0059] Figure 4 This is another scenario illustration of the data analysis method based on IoT devices provided in the embodiments of this application;

[0060] Figure 5 This is another scenario illustration of the data analysis method based on IoT devices provided in the embodiments of this application;

[0061] Figure 6 This is another flowchart illustrating the data analysis method based on IoT devices provided in the embodiments of this application;

[0062] Figure 7 This is a schematic diagram of the structure of the data analysis device based on an Internet of Things (IoT) device provided in the embodiments of this application;

[0063] Figure 8 This is a schematic diagram of the structure of the Internet of Things device provided in the embodiments of this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. However, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0065] This application proposes a data analysis method based on an Internet of Things (IoT) device. This method can be executed by a data analysis device integrated within the IoT device. The IoT device can include at least one of a terminal and a server. In other words, the data analysis method proposed in this application can be executed by a terminal, a server, or jointly by a terminal and a server capable of communicating with each other.

[0066] The terminal may include, but is not limited to, smartphones, tablets, laptops, personal computers (PCs), smart home appliances, wearable electronic devices, VR / AR devices, in-vehicle terminals, intelligent voice interaction devices, etc.

[0067] A server can be an interconnecting server between multiple heterogeneous systems or a backend server. It can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc.

[0068] It should be noted that the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0069] In one embodiment, such as Figure 1The data analysis device based on the Internet of Things (IoT) device can be integrated into the IoT device to implement the data analysis method based on the IoT device proposed in this application embodiment. Specifically, the IoT device 11 can detect the instructions generated by the application of the IoT device 11; when the instruction type of the detected instruction is an object operation type, a first signal and a second signal are generated; the first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device 11; the second signal is used to trigger the image acquisition module of the IoT device 11 to collect the object image data of the object; based on the object image data, the embedded data is subjected to data augmentation processing to obtain augmented embedded data; the augmented embedded data is reported to the server 10 so that the application of the IoT device 11 can be analyzed for object operation based on the augmented embedded data.

[0070] The following will provide a detailed description of each example. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0071] This application embodiment will be described from the perspective of a data analysis device based on an Internet of Things (IoT) device. This data analysis device based on an IoT device can be integrated into an IoT device, which can be a server or a terminal or other device.

[0072] like Figure 2 The present invention provides a data analysis method based on Internet of Things (IoT) devices, the specific process of which includes:

[0073] 101. Detect the instructions generated by the application of IoT devices.

[0074] Applications can include various mobile phone software (APP), tablet software, laptop software, personal computer (PC) software, and so on.

[0075] For example, applications can include browsers, video playback software, game software, and so on.

[0076] For example, applications can also include mini-programs. A mini-program is an application that can be used without downloading or installing it. To provide users with more diverse business services, developers can create corresponding mini-programs for terminal applications (such as instant messaging applications, shopping applications, email applications, etc.). These mini-programs can be embedded as sub-applications within the terminal application, and by running the sub-application (i.e., the corresponding mini-program) within the application, corresponding business services can be provided to the user.

[0077] Internet of Things (IoT) devices include those that can connect to a network and have the ability to transmit data. For example, IoT devices can include smartphones, laptops, tablets, smart TVs, smart appliances, and wearable devices, among others.

[0078] In one embodiment, IoT devices typically provide services to users through an application, and users can use the services provided by the application through certain operations on the application interface.

[0079] For example, when the IoT device is a palm-scanning device, it can include a palm-scanning recognition application. Users can use this application to perform functions such as palm-scanning payments and palm-scanning door opening.

[0080] For example, when the IoT device is a smart desk lamp, the smart desk lamp may include a desk lamp control application. Users can use the smart desk lamp to realize functions such as automatically turning the desk lamp on and off and automatically adjusting the brightness.

[0081] In one embodiment, the Internet of Things (IoT) device may include a variety of electronic components. For example, the IoT device may include an event detector, a data acquisition module, and an image acquisition module.

[0082] The event detector can be used to detect commands generated by applications on IoT devices. When the event detector detects that the command type is an object operation type, it generates a first signal and a second signal. This event detector can be considered equivalent to a time notification in the IoT device system. By defining predefined event-triggered event detectors, they can respond and execute subsequent logic.

[0083] The data acquisition module can be used to collect data generated by IoT devices. In one embodiment, the data acquisition module may include various embedded data acquisition algorithms, and the data acquisition module can collect embedded data by calling these algorithms.

[0084] The image acquisition module can be used to capture images of the user. For example, this image acquisition module can be a camera module, etc.

[0085] In one embodiment, an IoT device can detect instructions generated by the IoT device's application using an event detector.

[0086] 102. When the instruction type of the instruction is detected to be an object operation type, generate the first signal and the second signal.

[0087] In one embodiment, when the event detector of the IoT device detects that the instruction type of the instruction is an object operation type, a first signal and a second signal can be generated.

[0088] When the instruction type is an object operation type, it means that the instruction is triggered based on the user's operation.

[0089] The first signal can be used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the command. This first signal can be an electrical signal.

[0090] The second signal can be used to trigger the image acquisition module of the IoT device to acquire object image data. This second signal can be an electrical signal.

[0091] In this embodiment of the application, in order to improve the accuracy and reliability of data analysis of IoT devices, in addition to collecting the embedded data corresponding to user operations, object image data of the user can also be collected. Through object image data, it is helpful to reconstruct the scene when the user uses the IoT device, making the data analysis of IoT devices more accurate and reliable.

[0092] Therefore, when the instruction type of the detected instruction is an object operation type, the event detector of the IoT device can generate a first signal and a second signal.

[0093] In one embodiment, this application can divide the data generated in an IoT device into two types: data generated internally by the IoT device and data generated by the IoT device based on user triggers. To collect user-triggered data, instructions generated by the IoT device's application can be detected. When an instruction generated based on a user's operation is detected, the data corresponding to the user's operation can be collected.

[0094] In one embodiment, when detecting instructions generated by an application on an IoT device, the type of the instruction can be determined after the instruction is detected. When the instruction type is determined to be an object operation type, a first signal and a second signal can be generated.

[0095] Specifically, before the step "when the instruction type of the instruction is detected to be an object operation type, generate the first signal and the second signal", the following may also be included:

[0096] The detected instructions are parsed to obtain the corresponding instruction type;

[0097] The command type is matched with the preset object operation type to obtain the matching result;

[0098] The matching results determine whether the instruction type is an object operation type.

[0099] In one embodiment, the detected instructions can be parsed to obtain the instruction type corresponding to the instruction. For example, the detected instructions can be iterated to obtain the instruction type corresponding to each instruction.

[0100] Then, the instruction type can be matched with the preset object operation type to obtain the matching result. Based on the matching result, it is then determined whether the instruction type is an object operation type. That is, it is determined whether the instruction type and the preset object operation type are the same. If they are the same, it means the instruction type is an object operation type. If they are different, it means the instruction type is not an object operation type.

[0101] 103. The first signal triggers the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction. The embedded data represents the operation performed by the object on the IoT device's application.

[0102] In one embodiment, a first signal can be used to trigger the data acquisition module of the IoT device to collect embedded data corresponding to the instruction. This embedded data represents the operation performed by the user on the IoT device's application. For example, the embedded data may include whether the user's operation on the application is a click or a swipe, which user flow it is, which step of the operation it is, which position on the interface it is, and the current timestamp, etc. For example, suppose the IoT device is a palm-swipe device. When a user uses the palm-swipe device to make a palm-swipe payment, the embedded data can indicate that the user triggered a palm-swipe payment operation. Furthermore, the embedded data can also indicate the time the user triggered the operation, etc.

[0103] In one embodiment, the data acquisition module may include various tracking data acquisition algorithms, which can be used to collect tracking data. For example, tracking data acquisition algorithms may include front-end and back-end tracking algorithms, full tracking algorithms, or visual tracking algorithms, etc.

[0104] The main implementation of front-end and back-end event tracking involves embedding a piece of data collection code into the application's functional code. When the user interacts with the application and the event tracking triggers, the information to be collected is reported. The data reporting format is generally in the form of {key, value}.

[0105] Full tracking is also a type of front-end tracking, but it does not support custom parameters. Simply put, it encapsulates all the necessary attribute acquisition methods into a Software Development Kit (SDK), and embeds this SDK at any location where tracking is needed to collect tracking information.

[0106] Visual event tracking means that developers, in addition to integrating the data collection SDK, do not need to write additional event tracking code. Instead, business personnel use the analytics platform's selection function to "select" the controls that need to be captured for user actions and assign event names. After the selection is completed, these configurations are synchronized to each user's terminal, and the data collection SDK automatically collects and sends user action data according to the selected configurations.

[0107] In one embodiment, to standardize the format of the collected embedded data, the embedded data corresponding to the instruction can be collected according to the following steps. Specifically, the step "using the first signal to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction" may include:

[0108] The first signal triggers the data acquisition module of the IoT device to collect the embedded data stream and add the embedded data stream to the embedded data queue;

[0109] Determine the queue position identifier of the data stream within the data queue;

[0110] Determine the operation information corresponding to the instruction based on the embedded data stream;

[0111] Generate operation identifiers corresponding to the embedded data stream;

[0112] The operation identifier, operation information, and queue position identifier are integrated to obtain the embedded data.

[0113] In one embodiment, a first signal can be used to trigger the data acquisition module of an IoT device to collect embedded data streams.

[0114] In this context, a data stream can be an ordered sequence of bytes with a start and an end point. A tracked data stream can be a data sequence corresponding to user actions. In one embodiment, a tracked data stream can also represent an operation performed by an object on an IoT device's application. For example, a tracked data stream can include whether the user's action on the application is a click or a swipe, which user flow it belongs to, which step of the operation it is, which position on the interface it is, and the current timestamp, etc. The difference between a tracked data stream and tracked data is that a tracked data stream is the raw data corresponding to user actions, while tracked data is data obtained after integration and processing. For example, a tracked data stream is a data sequence, while tracked data can be in the form of {key-value}.

[0115] In one embodiment, after the data stream of the event tracking points is collected, it can be added to a data tracking point queue. The data tracking point queue can be used to store all collected data streams. For example, all data streams collected by the data acquisition module can be stored in the data tracking point queue.

[0116] Then, the queue position identifier of the event tracking data stream in the event tracking data queue can be determined. For example, the queue position identifier can indicate the storage location of the event tracking data stream in the event tracking data queue. For example, when the queue position identifier of the event tracking data stream is 17, it can be said that the storage location of the event tracking data stream in the event tracking data queue is the 17th position.

[0117] In one embodiment, the operation information corresponding to the instruction can be determined based on the data stream from the embedded points. This operation information can describe what action the user performed on the application. For example, the operation information could be "user.click.homepage.button," indicating that the user clicked the home button. Another example is "user.slice.lockout.button," indicating that the user slid the unlock button, and so on.

[0118] In one embodiment, the event tracking data stream may include an operation information field corresponding to a user operation. The operation information field can be obtained by traversing the event tracking data stream, and then used as the operation information corresponding to the instruction.

[0119] In one embodiment, an operation identifier corresponding to the event tracking data stream can be generated. This operation identifier can be a unique identifier for the event tracking data stream. This operation identifier distinguishes different event tracking data streams, thereby different event tracking data. For example, the operation identifier can be a session identifier (sessionId). The sessionId is used to identify a user's single operation process. For instance, in an offline facial recognition payment scenario, one sessionId corresponds to one user's payment process; it is a 16-bit random string, and a new process will be reconstructed.

[0120] In one embodiment, the operation identifier, operation information, and queue position identifier can be integrated to obtain the event tracking data. For example, the operation identifier, operation information, and queue position identifier can be filled into a preset event tracking data field according to a preset format to obtain the event tracking data.

[0121] For example, event tracking data could be {sessionId: Hloa1acxuw3U, action: user.click.homepage.button, seq: 0}. Here, "sessionId: Hloa1acxuw3U" indicates the operation identifier corresponding to the event tracking data. Generally, each event tracking data has a different operation identifier, allowing for differentiation between event tracking data. "action: user.click.homepage.button" refers to the operation information, indicating the user's action on the IoT device. "seq: 0" indicates the queue position information of the event tracking data stream. For example, "seq: 0" could mean the 0th position of the event tracking data stream in the event tracking data queue. When developers perform data analysis based on event tracking data, if they want to obtain the original event tracking data stream, they can trace back to the corresponding event tracking data stream through the queue position identifier.

[0122] In one embodiment, the event tracking data includes an operation identifier, operation information, and a queue position identifier. The operation information allows developers to understand what operations the user performed on the IoT device. The queue position identifier allows developers to trace the corresponding event tracking data stream. Then, developers can use the event tracking data stream to perform a more thorough analysis of user operations, better identify problems with the IoT device, and improve the accuracy and reliability of IoT device data analysis.

[0123] 104. Use the second signal to trigger the image acquisition module of the IoT device to acquire object image data.

[0124] In one embodiment, in an industrial setting, offline IoT device problem feedback relies on a user's problem description. For example, when an IoT device malfunctions or encounters a problem, it displays a problem feedback interface, allowing users to report the issue. However, differences in user perception can lead to unclear problem descriptions, uncertain anomaly times, and unclear operational steps, making it difficult for developers to effectively analyze and pinpoint the problems with the IoT devices.

[0125] For example, assuming an IoT device is a palm-scanning device, we can use an application scenario that uses palm recognition to obtain user identity information for payment and consumption as an example.

[0126] Users can trigger the camera's detection event by displaying their palm above the camera on the palm-scanning device, and begin image acquisition. After successful palmprint selection, the application layer obtains the user's palmprint feature data and sends a network request to the server. The server's palmprint retrieval model (e.g., a machine learning model) matches the user's palmprint with the user data in the database, finds the corresponding user identity information, and then returns the user's identity information to the palm-scanning device via a network packet.

[0127] After receiving the backend command, the swipe device obtains the user's identity information and begins page redirection. All system actions up to this point (such as camera capture, recognition, and algorithm optimization) are considered to be internal system state transitions, while the interface transition is triggered by the user's swipe action, and this change in interface transition can be considered an external system state transition.

[0128] To accurately pinpoint the external state transitions of the system, an image acquisition module can collect object image data. After acquiring the object image data, it can be correlated with embedded data. The embedded data describes the user's operation process, while the object image data identifies who triggered the operation. When developers cannot accurately analyze IoT device faults using embedded data, they can determine the user corresponding to the operation based on the object image data. Then, developers can communicate with the user to understand the user's experience using the IoT device, thereby better locating and analyzing any problems with the IoT device.

[0129] For example, such as Figure 3 As shown, the swipe interface of a swipe device may include a hidden control. When a user accidentally triggers this hidden control, the swipe device will display a hidden settings page. At this point, because the user is unfamiliar with the swipe device's operation, they may be unable to close the hidden settings page, causing the swipe device to malfunction. After obtaining the event tracking data, developers may only know that the swipe device is in a hidden settings page state, but they don't know the actual reason why the swipe device is unusable. However, by using object image data, developers can communicate with users to reconstruct the usage scenario of the swipe device and pinpoint the problem.

[0130] Therefore, the second signal can be used to trigger the image acquisition module of the IoT device to acquire object image data.

[0131] The object image data can include images that record the object's characteristic information, which can identify the object. For example, the object image data could be an image of the object's face. Or, for example, it could be an image of the object's palm, and so on.

[0132] In one embodiment, there are several methods to use a second signal to trigger the image acquisition module of an IoT device to acquire object image data of an object.

[0133] For example, a second signal can be used to trigger the image acquisition module of an IoT device to directly acquire object image data. For instance, when a user operates an IoT device, the image acquisition module can acquire an image of the object.

[0134] In one embodiment, to improve the accuracy of image acquisition, the image acquisition module can also acquire object video, and then select the video frame with the best image quality as the object image. Specifically, the step "using the second signal to trigger the image acquisition module of the IoT device to acquire object image data" may include:

[0135] The second signal triggers the image acquisition module of the IoT device to acquire object video, which includes multiple object video frames.

[0136] The image quality of the target video frames is compared to obtain the comparison results.

[0137] Based on the comparison results, a reference object image of the object is selected from multiple video frames;

[0138] The reference object image is transformed to obtain the object image data.

[0139] The reference image can refer to the image with the best image quality in the video frame.

[0140] Image quality can include factors such as the angle, size, centering, and color image sharpness of the face / hand in the video frame.

[0141] In one embodiment, a second signal can be used to trigger the image acquisition module of an IoT device to acquire object video, which includes multiple object video frames.

[0142] For example, an image acquisition module can be used to capture a video of a subject's face. As another example, an image acquisition module can be used to capture a video of a subject's palm.

[0143] Then, the image quality of the object's video frames can be compared to obtain the comparison results. By comparing the image quality of the object's video frames, the video frame with the best image quality can be identified. The best image quality can be measured from multiple dimensions. For example, image quality can be measured by combining factors such as image resolution, the completeness of the object's face, and image brightness.

[0144] For example, reference object images of an object can be filtered from multiple video frames of a facial video based on the angle, size, centering, and color image sharpness of the face / hand in each video frame.

[0145] In one embodiment, to facilitate image storage, the reference object image can be subjected to attribute transformation to obtain object image data. Here, the reference object image is in the form of an image, while the object image data is in the form of data.

[0146] Uploading images directly as raw data would consume a significant amount of energy. A better approach is to compress the reference object image to obtain a compressed reference object image. Furthermore, user-triggered application state changes on IoT devices are likely frequent operations, meaning the device might collect multiple images from the user. Uploading these images directly without compression would result in low reporting efficiency. To improve reporting efficiency, images can be compressed, and then the compressed object image can be serialized to obtain the object image data.

[0147] Specifically, the step "perform attribute transformation on the reference object image to obtain object image data" may include:

[0148] The reference object image is identified to obtain the image attribute information corresponding to the reference object image;

[0149] Based on image attribute information, the reference object image is compressed to obtain the compressed reference object image;

[0150] The compressed reference object image is serialized to obtain the object image data.

[0151] Image attribute information may include image resolution, number of pixels, size, brightness, saturation bit depth, and color channels, etc.

[0152] In one embodiment, a reference object image can be identified to obtain image attribute information corresponding to the reference object image. Then, based on the image attribute information, the reference object image can be compressed to obtain a compressed reference object image.

[0153] There are several ways to compress a reference object image based on its image attribute information to obtain a compressed reference object image.

[0154] For example, the open-source framework Luban / Compressor can be used to compress images based on their image attributes. For instance, objects can be compressed based on their resolution.

[0155] For example, assuming the reference object image has a size of 720*1280 and a file size of 390k, by compressing the reference object image, the compressed reference object image can be obtained with a size of 720*1280 and a file size of 87k.

[0156] For example, assuming the reference object image has a size of 3096*4128 and a file size of 3.12M, by compressing the reference object image, the compressed reference object image can be obtained with a size of 1548*2064 and a file size of 141k.

[0157] In one embodiment, the compressed reference object image can be serialized to obtain object image data. Serialization refers to converting a data structure or object into a format that can be stored and transmitted (e.g., over a network), while ensuring that the serialized result can be reconstructed from the original data structure or object later (potentially in another computing environment). For example, the compressed reference object image can be serialized based on the Protobuf protocol to obtain object image data. The Protobuf protocol is an efficient and lightweight structured data storage method that can be used for (data) communication protocols, data storage, etc.

[0158] In one embodiment, such as Figure 4 As shown, on IoT devices with built-in image acquisition modules, services are provided to users through an application. Users can use application services (such as facial recognition payment, palm recognition payment, palm recognition door opening, etc.) by performing some operations on the application interface.

[0159] When a user interacts with the application interface, it triggers the application's event detectors (such as click events and swipe detection events in Android). Upon receiving the user's interaction, the system generates two signals simultaneously. One signal notifies the data acquisition module to generate event tracking data (including information about the user's interaction, such as whether it was a click or a swipe, which user flow it belongs to, which step of the operation it is, its location on the screen, the current timestamp, etc.). The other signal reaches the image acquisition module, which then begins image recognition and ultimately acquires a photograph of the user at the scene.

[0160] 105. Based on the object image data, perform data augmentation on the embedded data to obtain augmented embedded data.

[0161] In one embodiment, in order to associate the embedded data and the object image data, the embedded data can be augmented based on the object image data to obtain augmented embedded data.

[0162] Specifically, the step "based on the object image data, perform data augmentation on the embedded data to obtain augmented embedded data" may include:

[0163] Add an image field to the event tracking data to obtain the event tracking data after the addition;

[0164] Based on the image field, the object image data is added to the added tracking data to obtain the expanded tracking data.

[0165] In one embodiment, an image field can be added to the event tracking data to obtain the event tracking data after the addition. For example, an image field can be added to the event tracking data according to a preset location to obtain the event tracking data after the addition. For example, the image field can be "live_photos", and the content that has been added to the event tracking data can be identified through the image field.

[0166] Then, based on the image field, the object image data can be added to the added tracking data to obtain the expanded tracking data.

[0167] For example, the original event tracking data is {sessionId: Hloa1acxuw3U, action: user.click.homepage.button, seq: 0}. By expanding the event tracking data, the expanded event tracking data can be obtained as {sessionId: Hloa1acxuw3U, action: user.click.homepage.button, seq: 0, live_photos: live photos}.

[0168] In one embodiment, such as Figure 4 As shown, the image acquisition module uploads user photos to the business system, which then associates these user photos with the event tracking data for this operation, treating it as an additional field: live_photos. Based on this field, the event tracking data is expanded to obtain the expanded event tracking data.

[0169] Then, the expanded event tracking data can be updated to the current event tracking queue.

[0170] At this time, there are two types of event tracking data in the event tracking queue: ordinary event tracking data and extended event tracking data. Ordinary event tracking data generally marks non-user operation behavior (such as system-triggered behavior, for example: the time spent on interface jump), while extended event tracking data contains the data stream of this user's facial recognition (which can be hexadecimal or binary stream converted by the protobuf protocol).

[0171] 106. Report the expanded data points so that the IoT device application can perform object operation analysis based on the expanded data points.

[0172] In one embodiment, after obtaining the expanded tracking data, the expanded tracking data can be reported, enabling the application of the IoT device to perform object operation analysis based on the expanded tracking data.

[0173] Among them, object operation analysis of IoT device applications based on expanded embedded data can refer to the ability to reconstruct the scenario information of objects operating IoT devices by analyzing the expanded embedded data.

[0174] For example, after receiving expanded tracking data, IoT devices can report this data to the server. Developers can then analyze this expanded tracking data to determine who performed which step, what environmental changes occurred during each step, and so on, which can assist developers in problem localization and even troubleshooting.

[0175] In one embodiment, such as Figure 4 As shown, IoT devices can add the expanded tracking data to the tracking queue, and then report the tracking queue to achieve the reporting of the expanded tracking data.

[0176] The event tracking queue can be used to store event tracking data that needs to be reported. For example, data generated by IoT devices can include two types: data generated internally by the IoT device and data generated based on user triggers. During IoT data collection, in addition to collecting user-triggered data, data generated internally by the IoT device is also collected and reported. This allows developers to perform comprehensive analysis of the IoT device based on the internal data and the event tracking data corresponding to user actions, improving the accuracy of problem analysis. Therefore, the event tracking queue can include not only expanded event tracking data but also data generated internally by the IoT device. After queuing the event tracking queue, developers can use the data in the queue to analyze problems with the IoT device.

[0177] In one embodiment, to facilitate data analysis by developers, before adding the expanded event tracking data to the event tracking queue for reporting, an instruction type identifier can be generated based on the instruction type corresponding to the instruction. Then, the instruction type identifier is added to the expanded event tracking data to obtain operation event tracking data, and finally, the operation event tracking data is added to the event tracking queue.

[0178] Specifically, the step "add the expanded tracking data to the tracking queue" can include:

[0179] Generate an instruction type identifier based on the instruction type corresponding to the instruction;

[0180] Add the instruction type identifier to the expanded event tracking data to obtain the operation event tracking data;

[0181] Add the operation tracking data to the tracking queue.

[0182] The instruction type identifier can be used to indicate whether the data stream is generated internally by the IoT device or generated by the IoT device based on user triggers. For example, when the instruction's operation identifier is "origin: 0", it indicates that the data stream is generated internally by the IoT device. Conversely, when the instruction's operation identifier is "origin: 1", it indicates that the data stream is generated by the IoT device based on user triggers.

[0183] For example, the expanded event tracking data is {sessionId: Hloa1acxuw3U, action: user.click.homepage.button, seq: 0, live_photos: on-site photos}. After adding the instruction type identifier "sky price" to the expanded event tracking data, the event tracking operation data becomes {sessionId: Hloa1acxuw3U, action: user.click.homepage.button, seq: 0, live_photos: on-site photos, origin: 1}. Then, the event tracking operation data can be added to the event tracking queue.

[0184] Tracking data typically refers to a data sequence recorded in the code by developers to document changes in system behavior. From the perspective of the generation scenario, we divide tracking data into two types: system-triggered code tracking and user-triggered code tracking through the interface. We define a data sequence as consisting of multiple {key, value} key-value pairs, and we will add an extra field with the key being origin. The value of origin for system-generated tracking events is 0, and the value for user-triggered events is set to 1, so that the data can be distinguished based on this field when retrieving data in the backend system.

[0185] In one embodiment, to improve the security of data reporting, after adding the expanded tracking data to the tracking data, the data in the tracking queue can be encrypted to obtain an encrypted tracking queue. Then, the encrypted tracking queue is reported.

[0186] Specifically, the step "reporting the expanded tracking data so that the application for IoT devices can perform object operation analysis based on the expanded tracking data" may include:

[0187] Add the expanded event tracking data to the event tracking queue;

[0188] Encrypt the data in the data tracking queue to obtain the encrypted data tracking queue;

[0189] The encrypted tracking data queue is reported to enable the IoT device application to perform object operation analysis based on the expanded tracking data of the encrypted tracking data queue.

[0190] There are several ways to encrypt the data in the event tracking queue to obtain an encrypted event tracking queue.

[0191] For example, data in the data tracking queue can be encrypted using methods such as the Data Encryption Algorithm (DEA), the Secure Hash Algorithm (SHA), or the Advanced Encryption Standard (AES) to obtain the encrypted data tracking queue.

[0192] Then, the encrypted tracking data queue can be reported to enable object operation analysis of IoT device applications based on the expanded tracking data of the encrypted tracking data queue.

[0193] In one embodiment, such as Figure 5 As shown, the generated data tracking queue is compressed and encrypted at the application level, and then transmitted to the server through the reporting framework and network framework. After decompression and decryption, the server parses the data according to the corresponding fields and stores it in the database.

[0194] When developers need to locate problems, they can obtain the product serial number of the IoT device and a general description of the conditions (such as the time range, whether it is a click operation or a swipe operation), and then filter out the batch of event tracking queues from the database. The event tracking queue contains all the operation steps and on-site environment information of the device within the set time range.

[0195] Based on the corresponding data, developers can determine who performed which step (e.g., the device entered the settings page causing the camera to disconnect and become unrecoverable, who actively closed the application or caused the application to malfunction, who returned the interface to the home page), and what environmental changes occurred during each step (e.g., the impact of light intensity on recognition results during user operations, etc.). This can assist R&D in locating problems locally and even replaying problems.

[0196] In this embodiment, instructions generated by the application of an IoT device can be detected. When the instruction type is detected to be an object operation type, a first signal and a second signal are generated. The first signal triggers the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device. The second signal triggers the image acquisition module of the IoT device to collect the object image data. Based on the object image data, the embedded data is augmented to obtain augmented embedded data. The augmented embedded data is reported so that object operation analysis of the IoT device application can be performed based on the augmented embedded data. Through this embodiment, the problem of inaccurate user identification during problem localization caused by multiple users switching operations in offline IoT devices can be effectively solved. For example, when analyzing problems of IoT devices, if developers cannot analyze the problems of IoT devices through ordinary embedded data, they can determine the identity of the user operating the IoT device through operation embedded data. Then, developers can contact the operating user to reconstruct the scenario in which the IoT device malfunctioned, thereby improving the accuracy and reliability of IoT device data analysis.

[0197] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0198] This application will use the example of integrating a data analysis method based on an IoT device into an IoT device to illustrate the method of this application.

[0199] In one embodiment, such as Figure 6 As shown, a data analysis method based on IoT devices has the following specific process:

[0200] 201. IoT devices detect the instructions generated by the IoT device's application.

[0201] On IoT devices with built-in image acquisition modules, services are provided to users through an application. Users can use application services (such as facial recognition payment, palm recognition payment, palm recognition door opening, etc.) by performing certain operations on the application interface.

[0202] 202. When the IoT device detects that the instruction type of the instruction is an object operation type, it generates a first signal and a second signal.

[0203] When a user interacts with the application interface, it triggers the application's event detectors (such as click events and swipe detection events in Android). Upon receiving the user's interaction, the system generates two signals simultaneously. One signal notifies the data acquisition module to generate event tracking data (including information about the user's interaction, such as whether it was a click or a swipe, which user flow it belongs to, which step of the operation it is, its location on the screen, the current timestamp, etc.). The other signal reaches the image acquisition module, which then begins image recognition and ultimately acquires a photograph of the user at the scene.

[0204] 203. The IoT device uses the first signal to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction. The embedded data represents the operation performed by the object on the application of the IoT device.

[0205] 204. The Internet of Things (IoT) device uses a second signal to trigger the image acquisition module of the IoT device to acquire object image data of the object.

[0206] 205. Based on the object image data, the IoT device performs data augmentation processing on the embedded data to obtain augmented embedded data.

[0207] The image acquisition module uploads user photos to the business system, which then associates these user photos with the event tracking data for this operation, treating it as an additional field: live_photos. Based on this field, the event tracking data is expanded to obtain the expanded event tracking data.

[0208] Then, the expanded event tracking data can be updated to the current event tracking queue.

[0209] At this time, there are two types of event tracking data in the event tracking queue: ordinary event tracking data and extended event tracking data. Ordinary event tracking data generally marks non-user operation behavior (such as system-triggered behavior, for example: the time spent on interface jump), while extended event tracking data contains the data stream of this user's facial recognition (which can be hexadecimal or binary stream converted by the protobuf protocol).

[0210] 206. IoT devices report the expanded embedded data so that the IoT device's application can perform object operation analysis based on the expanded embedded data.

[0211] Finally, at the application level, the generated data tracking queue is compressed and encrypted, and then transmitted to the server through the reporting framework and network framework. After decompression and decryption, the server parses the data according to the corresponding fields and stores it in the database.

[0212] When developers need to locate problems, they can obtain the product serial number of the IoT device and a general description of the conditions (such as the time range, whether it is a click operation or a swipe operation), and then filter out the batch of event tracking queues from the database. The event tracking queue contains all the operation steps and on-site environment information of the device within the set time range.

[0213] Based on the corresponding data, developers can identify who performed which step, what environmental changes occurred during each step, and so on. This can assist developers in locating problems locally and even replaying them.

[0214] In this embodiment, the IoT device detects instructions generated by its application. When the IoT device detects that the instruction type is an object operation type, it generates a first signal and a second signal. The IoT device uses the first signal to trigger its data acquisition module to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the IoT device's application. The IoT device uses the second signal to trigger its image acquisition module to collect the object image data. Based on the object image data, the IoT device performs data augmentation processing on the embedded data to obtain augmented embedded data. The IoT device reports the augmented embedded data so that object operation analysis can be performed on the IoT device's application based on the augmented embedded data. Through this embodiment, the accuracy and reliability of IoT device data analysis can be improved.

[0215] To better implement the data analysis method based on IoT devices provided in this application, one embodiment also provides a data analysis device based on IoT devices, which can be integrated into the IoT device. The meanings of the terms used are the same as in the data analysis method based on IoT devices described above, and specific implementation details can be found in the description of the method embodiments.

[0216] In one embodiment, a data analysis device based on an Internet of Things (IoT) device is provided. This IoT-based data analysis device can be specifically integrated into the IoT device, such as... Figure 7 As shown, the data analysis device based on IoT devices includes: a detection unit 301, a generation unit 302, a data acquisition unit 303, an image acquisition unit 304, an expansion unit 305, and a reporting unit 306, as detailed below:

[0217] The detection unit 301 is used to detect the instructions generated by the application of the Internet of Things device;

[0218] The generation unit 302 is used to generate a first signal and a second signal when the instruction type of the instruction is detected to be an object operation type;

[0219] Data acquisition unit 303 is used to trigger the data acquisition module of the Internet of Things device to acquire the embedded data corresponding to the instruction using the first signal, wherein the embedded data represents the operation performed by the object on the application of the Internet of Things device;

[0220] Image acquisition unit 304 is used to trigger the image acquisition module of the Internet of Things device to acquire object image data of the object using the second signal;

[0221] The expansion unit 305 is used to perform data expansion processing on the embedded data according to the object image data to obtain expanded embedded data;

[0222] The reporting unit 306 is used to report the expanded embedded data so that the application of the Internet of Things device can be analyzed for object operation based on the expanded embedded data.

[0223] In one embodiment, the data acquisition unit 303 may include:

[0224] The data acquisition subunit is used to trigger the data acquisition module of the IoT device to acquire the embedded data stream using the first signal, and add the embedded data stream to the embedded data queue;

[0225] The first determining subunit is used to determine the queue position identifier of the embedded data stream in the embedded data queue;

[0226] The second determining subunit is used to determine the operation information corresponding to the instruction based on the embedded data stream;

[0227] The first generation subunit is used to generate the operation identifier corresponding to the embedded data stream;

[0228] The integration subunit is used to integrate the operation identifier, the operation information, and the queue position identifier to obtain the embedded data.

[0229] In one embodiment, the expansion unit 305 may include:

[0230] The first addition subunit is used to add an image field to the embedded data to obtain the embedded data after addition.

[0231] The second adding subunit is used to add the object image data to the added tracking data based on the image field, so as to obtain the expanded tracking data.

[0232] In one embodiment, the image acquisition unit 304 may include:

[0233] An image acquisition subunit is used to trigger the image acquisition module of the Internet of Things device to acquire object video using the second signal, the object video including multiple object video frames;

[0234] The comparison subunit is used to compare the image quality of the video frames of the object and obtain the comparison result;

[0235] A filtering subunit is used to filter out a reference object image of the object from the plurality of video frames based on the comparison results;

[0236] The attribute transformation subunit is used to perform attribute transformation on the reference object image to obtain the object image data.

[0237] In one embodiment, the attribute conversion subunit may include:

[0238] The recognition module is used to recognize the reference object image and obtain the image attribute information corresponding to the reference object image;

[0239] A compression module is used to compress the reference object image based on the image attribute information to obtain a compressed reference object image;

[0240] The serialization module is used to serialize the compressed reference object image to obtain the object image data.

[0241] In one embodiment, the reporting unit 306 may include:

[0242] The third addition subunit is used to add the expanded tracking data to the tracking queue;

[0243] The data encryption subunit is used to encrypt the data in the data collection queue to obtain the encrypted data collection queue.

[0244] The reporting subunit is used to report the encrypted tracking data queue so that the application of the Internet of Things device can be analyzed for object operations based on the expanded tracking data of the encrypted tracking data queue.

[0245] In one embodiment, the third adding subunit may include:

[0246] The generation module is used to generate an instruction type identifier based on the instruction type corresponding to the instruction;

[0247] The first adding module is used to add the instruction type identifier to the expanded embedding data to obtain operation embedding data;

[0248] The second adding module is used to add the operation tracking data to the tracking queue.

[0249] In one embodiment, the detection unit 301 may include:

[0250] The detection subunit is used to call the event detector to detect the instructions generated by the application of the IoT device.

[0251] In one embodiment, the generation unit 302 may include:

[0252] A generation subunit is configured to generate a first signal and a second signal when the event detector detects that the instruction type of the instruction is an object operation type.

[0253] In one embodiment, the data analysis apparatus may further include:

[0254] The parsing unit is used to parse the detected instructions to obtain the instruction type corresponding to the instructions;

[0255] The matching unit is used to match the instruction type with the preset object operation type to obtain a matching result;

[0256] The determining unit is used to determine whether the instruction type of the instruction is an object operation type based on the matching result.

[0257] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0258] The data analysis device based on IoT devices described above can improve the accuracy and reliability of IoT device data analysis.

[0259] This application also provides an Internet of Things (IoT) device, which may include a terminal or a server. For example, the IoT device can serve as a data analysis terminal based on IoT devices, such as a mobile phone, tablet computer, etc.; or, for example, the IoT device can serve as a server, such as a data analysis server based on IoT devices. Figure 8 As shown, it illustrates the structural diagram of the terminal involved in the embodiments of this application, specifically:

[0260] The IoT device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 8 The IoT device structure shown does not constitute a limitation on IoT devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0261] The processor 401 is the control center of the IoT device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402 to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0262] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of IoT devices, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0263] The IoT device also includes a power supply 403 that powers the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0264] The IoT device may also include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0265] Although not shown, IoT devices may also include display units, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the IoT device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0266] Detect instructions generated by applications in IoT devices;

[0267] When the instruction type of the instruction is detected to be an object operation type, a first signal and a second signal are generated;

[0268] The first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device;

[0269] The second signal is used to trigger the image acquisition module of the IoT device to acquire object image data of the object;

[0270] Based on the object image data, the embedded data is augmented to obtain augmented embedded data;

[0271] The expanded data points are reported so that the application of the IoT device can perform object operation analysis based on the expanded data points.

[0272] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0273] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an Internet of Things (IoT) device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the IoT device to perform the methods provided in the various optional implementations of the above embodiments.

[0274] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0275] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the data analysis methods based on IoT devices provided in embodiments of this application. For example, the computer program can execute the following steps:

[0276] Detect instructions generated by applications in IoT devices;

[0277] When the instruction type of the instruction is detected to be an object operation type, a first signal and a second signal are generated;

[0278] The first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device;

[0279] The second signal is used to trigger the image acquisition module of the IoT device to acquire object image data of the object;

[0280] Based on the object image data, the embedded data is augmented to obtain augmented embedded data;

[0281] The expanded data points are reported to enable object operation analysis of the IoT device's application based on these data. Specific implementation details of each of these operations can be found in the preceding embodiments and will not be repeated here.

[0282] Since the computer program stored in the storage medium can execute the steps in any of the data analysis methods based on IoT devices provided in the embodiments of this application, the beneficial effects that any of the data analysis methods based on IoT devices provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0283] The foregoing has provided a detailed description of a data analysis method, apparatus, IoT device, and storage medium based on an IoT device according to embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data analysis method based on an Internet of Things device, characterized by, include: Detect instructions generated by applications in IoT devices; When the instruction type of the instruction is detected to be an object operation type, a first signal and a second signal are generated; The first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data corresponding to the instruction, wherein the embedded data represents the operation performed by the object on the application of the IoT device; The second signal is used to trigger the image acquisition module of the IoT device to acquire object image data of the object; Based on the object image data, the embedded data is augmented to obtain augmented embedded data; The expanded data points are reported so that the application of the IoT device can perform object operation analysis based on the expanded data points.

2. The method of claim 1, wherein, The step of triggering the data acquisition module of the IoT device to acquire the embedded data corresponding to the instruction using the first signal includes: The first signal is used to trigger the data acquisition module of the IoT device to collect the embedded data stream and add the embedded data stream to the embedded data queue; Determine the queue position identifier of the embedded data stream in the embedded data queue; The operation information corresponding to the instruction is determined based on the embedded data stream; Generate the operation identifier corresponding to the embedded data stream; The operation identifier, the operation information, and the queue position identifier are integrated to obtain the embedded data.

3. The method according to claim 1 or 2, characterized in that, The step of augmenting the embedded data based on the object image data to obtain augmented embedded data includes: Add an image field to the embedded data to obtain the embedded data after addition; Based on the image field, the object image data is added to the added tracking data to obtain the expanded tracking data.

4. The method of claim 1, wherein, The step of triggering the image acquisition module of the IoT device to acquire object image data of the object using the second signal includes: The second signal is used to trigger the image acquisition module of the IoT device to acquire object video, the object video including multiple object video frames; The image quality of the video frames of the object is compared to obtain the comparison results. Based on the comparison results, a reference object image of the object is selected from the multiple object video frames; The reference object image is subjected to attribute transformation to obtain the object image data.

5. The method according to claim 4, characterized in that, The step of performing attribute transformation on the reference object image to obtain the object image data includes: The reference object image is identified to obtain the image attribute information corresponding to the reference object image; Based on the image attribute information, the reference object image is compressed to obtain a compressed reference object image; The compressed reference object image is serialized to obtain the object image data.

6. The method according to claim 1, characterized in that, The reporting of the expanded data points, enabling object operation analysis of the IoT device's application based on the expanded data points, includes: Add the expanded data points to the data point queue; The data in the data tracking queue is encrypted to obtain the encrypted data tracking queue; The encrypted tracking data queue is reported so that the application of the IoT device can be analyzed for object operations based on the expanded tracking data of the encrypted tracking data queue.

7. The method according to claim 6, characterized in that, Adding the expanded tracking data to the tracking queue includes: Based on the instruction type corresponding to the instruction, an instruction type identifier is generated; Add the instruction type identifier to the expanded data points to obtain the operation data points; Add the operation tracking data to the tracking queue.

8. The method according to claim 1, characterized in that, The detection of instructions generated by the application of the Internet of Things device includes: The event detector is invoked to detect commands generated by the application on the IoT device; When the instruction type of the instruction is detected to be an object operation type, generating a first signal and a second signal includes: When the event detector detects that the instruction type of the instruction is an object operation type, the event detector generates a first signal and a second signal.

9. The method according to claim 1 or 8, characterized in that, Before generating the first signal and the second signal when the instruction type of the instruction is detected to be an object operation type, the method further includes: The detected instruction is parsed to obtain the instruction type corresponding to the instruction; The instruction type is matched with the preset object operation type to obtain the matching result; Based on the matching results, determine whether the instruction type of the instruction is an object operation type.

10. A data analysis device based on Internet of Things (IoT) devices, characterized in that, include: The detection unit is used to detect the instructions generated by the application of the Internet of Things (IoT) device. The generation unit is used to generate a first signal and a second signal when the instruction type of the instruction is detected to be an object operation type; A data acquisition unit is used to trigger the data acquisition module of the Internet of Things device to acquire the embedded data corresponding to the instruction using the first signal, wherein the embedded data represents the operation performed by the object on the application of the Internet of Things device; The image acquisition unit is used to trigger the image acquisition module of the IoT device to acquire object image data of the object using the second signal; An expansion unit is used to perform data expansion processing on the embedded data based on the object image data to obtain expanded embedded data; The reporting unit is used to report the expanded embedded data so that the application of the Internet of Things device can be used for object operation analysis based on the expanded embedded data.

11. The apparatus according to claim 10, characterized in that, The data acquisition unit includes: The data acquisition subunit is used to trigger the data acquisition module of the IoT device to acquire the embedded data stream using the first signal, and add the embedded data stream to the embedded data queue; The first determining subunit is used to determine the queue position identifier of the embedded data stream in the embedded data queue; The second determining subunit is used to determine the operation information corresponding to the instruction based on the embedded data stream; The first generation subunit is used to generate the operation identifier corresponding to the embedded data stream; The integration subunit is used to integrate the operation identifier, the operation information, and the queue position identifier to obtain the embedded data.

12. The apparatus according to claim 10 or 11, characterized in that, The expansion unit includes: The first addition subunit is used to add an image field to the embedded data to obtain the added embedded data. The second adding subunit is used to add the object image data to the added tracking data based on the image field, so as to obtain the expanded tracking data.

13. The apparatus according to claim 10, characterized in that, The image acquisition unit includes: An image acquisition subunit is used to trigger the image acquisition module of the Internet of Things device to acquire object video using the second signal, the object video including multiple object video frames; The comparison subunit is used to compare the image quality of the video frames of the object and obtain the comparison result; A filtering subunit is used to filter out a reference object image of the object from the plurality of object video frames based on the comparison results; The attribute transformation subunit is used to perform attribute transformation on the reference object image to obtain the object image data.

14. The apparatus according to claim 13, characterized in that, The attribute conversion subunit includes: The recognition module is used to recognize the reference object image and obtain the image attribute information corresponding to the reference object image; A compression module is used to compress the reference object image based on the image attribute information to obtain a compressed reference object image. The serialization module is used to serialize the compressed reference object image to obtain the object image data.

15. The apparatus according to claim 10, characterized in that, The reporting unit includes: The third addition subunit is used to add the expanded tracking data to the tracking queue; The data encryption subunit is used to encrypt the data in the data collection queue to obtain the encrypted data collection queue. The reporting subunit is used to report the encrypted tracking data queue so that the application of the Internet of Things device can be analyzed for object operations based on the expanded tracking data of the encrypted tracking data queue.

16. The apparatus according to claim 15, characterized in that, The third added subunit includes: The generation module is used to generate an instruction type identifier based on the instruction type corresponding to the instruction; The first adding module is used to add the instruction type identifier to the expanded embedding data to obtain operation embedding data; The second adding module is used to add the operation tracking data to the tracking queue.

17. The apparatus according to claim 10, characterized in that, The detection unit includes: The detection subunit is used to call the event detector to detect the instructions generated by the application of the IoT device; The generation unit includes: A generation subunit is configured to generate a first signal and a second signal when the event detector detects that the instruction type of the instruction is an object operation type.

18. The apparatus according to claim 10 or 17, characterized in that, The data analysis device further includes: The parsing unit is used to parse the detected instructions to obtain the instruction type corresponding to the instructions; The matching unit is used to match the instruction type with the preset object operation type to obtain a matching result; The determining unit is used to determine whether the instruction type of the instruction is an object operation type based on the matching result.

19. An Internet of Things (IoT) device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the operations in the data analysis method based on an Internet of Things device according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data analysis method based on an Internet of Things device according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps in the data analysis method based on the Internet of Things device as described in any one of claims 1 to 9.

Images