Embedded collector monitoring method based on multi-modal data synchronous collection

Abstract

The invention discloses an embedded collector monitoring method based on multi-modal data synchronous acquisition, which relates to the technical field of embedded acquisition, realizes synchronous acquisition of various types of data by constructing a multi-modal data acquisition system, and inputs the acquired data into an embedded collector monitoring platform. A monitoring network is constructed to bind data communication between an embedded collector monitoring platform and an embedded collector, a monitoring instruction is transmitted by using the data communication, operation monitoring, including data accuracy monitoring, equipment state monitoring and fault early warning, of the embedded collector is realized, real-time feedback information during operation of the embedded collector is acquired, and the real-time feedback information is transmitted to the embedded collector. And fault diagnosis and operation management are carried out, so that efficient monitoring and management of the embedded collector are realized.

Description

Technical Field

[0001] The present invention relates to the field of embedded acquisition technology, and in particular to an embedded collector monitoring method based on multi-modal data synchronous acquisition. Background Art

[0002] Embedded data collectors are widely used in the field of modern information technology and can be used in industrial production monitoring, environmental data collection, smart home control, etc. With the continuous expansion of application scenarios, the functional requirements for embedded data collectors are also increasing, and it is necessary to collect multiple types of data, such as images, audio, sensor data, etc., to meet complex business needs.

[0003] However, in the current use of embedded data collectors, there are problems such as asynchronous multimodal data collection, difficulty in ensuring data accuracy, untimely monitoring of equipment operation status, and inefficient fault handling. These problems not only affect the quality and availability of data, but may also cause failures in the operation of the collector, affecting the stability and reliability of the entire system. Therefore, an efficient monitoring method is needed to solve these problems. Summary of the invention

[0004] In order to solve the above problems, the object of the present invention is to provide an embedded collector monitoring method based on synchronous acquisition of multi-modal data.

[0005] The purpose of the present invention can be achieved by the following technical solution: An embedded collector monitoring method based on multi-modal data synchronous acquisition comprises the following steps: Step S1: construct a multimodal data collection system, obtain multimodal data and input it into the embedded collector monitoring platform, then build a monitoring network, and bind the data communication between the embedded collector monitoring platform and each embedded collector through the monitoring network; Step S2: Transmitting monitoring instructions to the embedded collector through data communication, and then performing operation monitoring on the embedded collector, wherein the operation monitoring includes data accuracy monitoring, equipment status monitoring, and fault early warning; Step S3: Acquire real-time feedback information of the embedded collector during operation, and transmit it to the embedded collector monitoring platform, and perform fault diagnosis and operation management according to the real-time feedback information.

[0006] Furthermore, the process of constructing the multimodal data collection system includes: Determine the type of multimodal data. Multimodal data includes image data, audio data and sensor data, which are marked as D1, D2 and D3 respectively. Set corresponding acquisition modules for different types of multimodal data. Set an independent cache area for each acquisition module. The cache area number corresponds to the acquisition module and is recorded as i, i = 1, 2, 3, ..., n, where n is a natural number greater than 0. The cache area is used to temporarily store the collected data. At the same time, a data integration area is set to integrate the data in each cache area according to the timestamp.

[0007] Furthermore, the process of constructing the monitoring network and binding the data communication between the monitoring network and the embedded collector includes: Obtain information related to the embedded collector, including the collector number, model, and historical operation data. The collector number is j, j = 1, 2, 3, ..., m, where m is a natural number greater than 0, and is used as an encapsulation index to encapsulate and generate j data packets to be accessed; Establish a temporary access space, input the verification key and j data packets to be accessed, the data packets to be accessed are associated with the access IP address, and a list of legal access IPs is preset in the temporary access space. Determine whether the data packets to be accessed are legal data packets based on the access IP address and the list of legal access IPs; if not, do nothing; if so, generate a verification database request, obtain the verification key through the verification database request, and obtain the monitoring network configuration file through the verification key; According to the monitoring network configuration file and the mapping association of the data packet to be accessed, a collector sub-network is constructed and the corresponding sub-network number K is assigned, where K =<i, j> , the monitoring control center is set up to aggregate several collector sub-networks to build a monitoring network; Set the communication protocol. The communication protocol has a corresponding communication state, which includes a stable protocol state, an interrupted protocol state, and an interference protocol state. In the stable protocol state, the data communication between the embedded collector monitoring platform and the monitoring network is bound by the communication protocol. In the interrupted protocol state and the interference protocol state, an abnormal log is generated and transmitted to the administrator, who will restore the communication and troubleshoot the interference source.

[0008] Furthermore, the data accuracy monitoring process includes: The embedded collector monitoring platform generates a monitoring instruction, which includes several sub-monitoring instructions, denoted as M[K], where K is the subnetwork number of several collector subnetworks, and transmits the monitoring instruction to the monitoring network through data communication; The sub-monitoring instructions are set with different monitoring types. Different monitoring types have corresponding monitoring numbers. The monitoring numbers have corresponding accuracy monitoring rules. The corresponding accuracy monitoring rules are executed according to the monitoring numbers corresponding to the monitoring types, and the accuracy of the collected multimodal data is verified. The verification content includes data integrity, data format correctness and data value rationality.

[0009] Furthermore, the process of equipment status monitoring includes: Get the hardware status information of the embedded collector. The hardware status information includes CPU usage, memory usage and battery power, which are marked as H1, H2 and H3 respectively. The CPU usage threshold, memory usage threshold and battery power threshold are preset, which are marked as T1, T2 and T3 respectively. When H1≥T1, the CPU status is marked as abnormal; when H2≥T2, the memory status is marked as abnormal; when H3≤T3, the battery status is marked as abnormal, otherwise, the CPU status, memory status and battery status are normal; A device status report is generated based on the marked abnormal status. The device status report records the normal status area and abnormal status area corresponding to the embedded collector. The normal status area records the hardware status information without abnormalities, and the abnormal status area records the hardware status information with abnormalities and their corresponding abnormality types.

[0010] Furthermore, the fault warning process includes: Get the real-time operating parameters of the embedded collector, denoted as R1, get several risk nodes in the fault risk model, and then generate a risk node set, denoted as R2, then R2 = {R21, R22, R23, …, R2K}, and the risk node has a corresponding parameter range in the fault risk model; Traverse the parameter ranges corresponding to several risk nodes R21, R22, R23, ..., R2K, and then obtain the difference between the real-time operating parameters and several parameter ranges, compare the difference with the preset risk threshold, and then execute different fault warnings. The fault warnings are associated with corresponding warning levels, including level one warning, level two warning, and level three warning.

[0011] Furthermore, the process of performing fault diagnosis according to the real-time feedback information includes: Obtain the real-time feedback information of the embedded collector and transmit it to the embedded collector monitoring platform. The real-time feedback information includes data collection abnormality information, equipment hardware failure information and communication failure information. Different real-time feedback information is set with corresponding fault judgment rules; If the data collection abnormality information, equipment hardware fault information, and communication fault information do not meet the fault judgment rules, the subsequent fault diagnosis operation will not be performed; If any of the data collection abnormality information, equipment hardware failure information, and communication failure information meets the fault judgment rules, the fault diagnosis program is started, and the fault type is found out by analyzing the relevant information. The embedded collector monitoring platform performs fault diagnosis based on the fault type. The fault diagnosis fault types include software fault types and hardware fault types. The corresponding fault repair measures are performed according to the fault type.

[0012] Furthermore, the operation management process includes: The operation management includes operation management in the event of a fault and operation management when the task is completed, which are respectively assigned different management tags, denoted as Mg1 and Mg2; When the management mark is Mg1, it corresponds to the operation management under the fault condition, obtains the embedded collector with the fault problem, and suspends the data transmission and restricts the device function of the embedded collector; When the management mark is Mg2, the operation management corresponding to the task is completed, the stop instruction of the embedded collector monitoring platform is obtained, the data collection function of all embedded collectors in the monitoring network is turned off, and data backup and device sleep operations are performed.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. By building a multimodal data collection system, different collection modules are used to achieve synchronous collection of various types of data, and the orderliness and synchronization of data are guaranteed through the cache area and data integration area, a monitoring network is built and a strict access verification mechanism is set up to avoid illegal collector access. At the same time, stable data communication between the monitoring platform and the collector is achieved, laying the foundation for subsequent monitoring and management.

[0014] 2. Carry out multi-faceted operation monitoring of the embedded collector. Data accuracy monitoring ensures the quality of collected data. Equipment status monitoring timely grasps the hardware operation status of the collector. Fault warning can detect potential fault risks in advance, effectively improving the reliability and stability of the collector operation.

[0015] 3. Perform accurate fault diagnosis based on real-time feedback information, take appropriate repair measures for different fault types, reasonably manage the operation of the collector in the event of a fault, shut down the collector in a standardized manner after the task is completed, and perform data backup and device hibernation, thereby improving the overall management efficiency and data security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, an embedded collector monitoring method based on multi-modal data synchronous acquisition includes the following steps: Step S1: construct a multimodal data collection system, obtain multimodal data and input it into the embedded collector monitoring platform, then build a monitoring network, and bind the data communication between the embedded collector monitoring platform and each embedded collector through the monitoring network; Step S2: Transmitting monitoring instructions to the embedded collector through data communication, and then performing operation monitoring on the embedded collector, wherein the operation monitoring includes data accuracy monitoring, equipment status monitoring, and fault early warning; Step S3: Acquire real-time feedback information of the embedded collector during operation, and transmit it to the embedded collector monitoring platform, and perform fault diagnosis and operation management according to the real-time feedback information.

[0018] It should be further explained that, in the specific implementation process, the process of building the multimodal data collection system includes: Taking the field of urban environmental monitoring as an example, in the smart city environmental monitoring system, multimodal data mainly includes image data (D1), audio data (D2) and sensor data (D3). Image data is used to capture the environmental conditions of urban streets, parks and other areas, such as whether there is garbage accumulation, the growth trend of green plants, etc.; audio data is used to monitor urban noise conditions, such as traffic noise, construction noise, etc.; sensor data covers the concentration data of pollutants such as PM2.5, PM10, sulfur dioxide, etc. collected by air quality sensors, as well as the ambient temperature and humidity data collected by temperature and humidity sensors.

[0019] For these different types of multimodal data, corresponding acquisition modules are equipped; The acquisition module includes an image acquisition module, an audio acquisition module and a sensor acquisition module; The corresponding relationship between different types of multimodal data and the acquisition modules they are equipped with is as follows: Image acquisition module—image data (D1); Audio acquisition module—audio data (D2); Sensor acquisition module—sensor data (D3); Image acquisition module: Use high-definition cameras with high resolution and wide dynamic range to collect image data to ensure that the details of the urban environment can be clearly presented; for example, such high-definition cameras are installed at key locations in the city's main streets and parks to obtain high-quality image information; The audio acquisition module uses a highly sensitive directional microphone to collect audio data. It can accurately capture sounds from a specific direction and reduce environmental noise interference. When monitoring traffic noise, the microphone can be pointed toward the road to more accurately collect the noise generated by driving vehicles. The sensor acquisition module is connected to a variety of environmental sensors, such as electrochemical sensors for detecting the concentration of harmful gases and capacitive sensors for measuring temperature and humidity. These sensors are reasonably distributed at various monitoring points in the city to comprehensively acquire sensor data as real-time monitored environmental parameters.

[0020] In order to ensure the orderly storage and processing of multimodal data, an independent cache area is set for each acquisition module. The number of the cache area is set to correspond to the acquisition module. Each cache area has high-speed reading and writing capabilities and can quickly temporarily store the collected multimodal data. Taking the cache area of ​​image data as an example, it uses high-speed flash memory technology to store a large amount of high-definition image data in a short time to avoid data loss. At the same time, a data integration area is set to integrate the data of each cache area according to the timestamp. When the collector starts working, each acquisition module synchronously collects data and stores the data in the corresponding cache area; The data integration area will periodically read multimodal data from the cache area (for example, every 10 seconds), arrange and combine them according to the timestamp order, and form a complete multimodal data record. In this way, the consistency of different types of data in the time dimension can be guaranteed, providing a reliable basis for subsequent analysis and processing.

[0021] It should be further explained that, in a specific implementation process, the process of constructing the monitoring network and binding the data communication between the monitoring network and the embedded collector includes: In the smart city environmental monitoring system, it is necessary to manage and monitor the embedded collectors distributed throughout the city, so it is crucial to build a monitoring network. First, obtain the relevant information of each embedded collector, including the collector number, model and historical operation data. The collector number is denoted as j, where j = 1, 2, 3, ..., m, m is a natural number greater than 0. The number serves as the unique identifier of the embedded collector, which facilitates accurate positioning and management in the system. For example, 100 embedded collectors are deployed in different areas of the city, and they each have a unique number from 001 to 100.

[0022] Using the collector number as the encapsulation index, j data packets to be accessed are generated. The data packets to be accessed record the basic information, configuration parameters and historical operation data of the embedded collector. A temporary access space is established, and the administrator enters the verification key and the data packet to be accessed. A list of legal access IPs is preset in the temporary access space. By comparing the access IP address of the data packet to be accessed with the legal access IP list, the system determines whether the data packet is legal. If the access IP address is not in the legal access IP list, the system will not take any action to prevent illegal collectors from accessing the network. If it is in the legal access IP list, a verification database request is generated to obtain the verification key, and then the monitoring network configuration file is obtained.

[0023] Mapping and associating the monitoring network configuration file with the data packets to be accessed is performed to form a collector subnetwork. For example, the data packet to be accessed with the collector number 005 is mapped and associated with the control partition numbered 3 after verification to form a collector subnetwork, and the collector subnetwork is assigned the number K=<3,005>. In the same way, all legal data packets to be accessed are mapped and associated with the corresponding control partitions to generate multiple collector subnetworks. A monitoring control center is set up, and all collector subnetworks are summarized by the monitoring control center to build a complete monitoring network, wherein the control partition number is denoted as i, i=1, 2, 3, ..., n, where n is a natural number greater than 0.

[0024] Set the communication protocol. The communication status of the communication protocol is divided into stable protocol status, interrupted protocol status and interference protocol status; In the stable protocol state, the monitoring network transmits multimodal data normally, and the data communication between the embedded collector monitoring platform and the monitoring network is stably bound; When the protocol interruption state or the protocol interference state occurs, the system will generate an exception log C and an exception log D, where the exception log C is used to record the frequency, duration, time of interruption and the source of interruption that caused the interruption of data communication between the embedded collector monitoring platform and the monitoring network, and the exception log D is used to record the frequency, location, intensity and duration of interference in the data communication between the embedded collector monitoring platform and the monitoring network; Exception logs C and D are transmitted to the administrator, who then recovers the communication and investigates the interference source. For example, if it is found that a newly built base station nearby causes the communication interference, the administrator can coordinate with relevant departments to adjust the base station parameters, eliminate the interference, restore normal communication, and rebind data communication.

[0025] It should be further explained that, in the specific implementation process, the data accuracy monitoring process includes: The embedded collector monitoring platform generates monitoring instructions, which contain several sub-monitoring instructions M[K], where K is the subnetwork number of the collector subnetwork. In the smart city environment monitoring system, the sub-monitoring instructions are set with different monitoring types. For example, for image data, there is an image clarity monitoring number, denoted as P11, and an image content integrity monitoring number, denoted as P12. For audio data, there is an audio signal strength monitoring number, denoted as P21, and an audio frequency range monitoring number, denoted as P22. For sensor data, there is a data value rationality monitoring number, denoted as P31, and a data continuity monitoring number, denoted as P32.

[0026] Each monitoring number corresponds to a specific accuracy monitoring rule. Taking the image clarity monitoring of image data (number P11) as an example, its monitoring rule is to evaluate the clarity by calculating the gradient information of the image. When the average gradient value of the image is lower than the preset threshold, it is determined that the image clarity is insufficient, which may affect the judgment of the environmental conditions. For audio signal strength monitoring of audio data (No. P21), if the audio signal strength is lower than the normal range, it indicates that there may be a microphone failure or sound transmission obstruction. The numerical rationality monitoring of sensor data (No. P31) will set a reasonable numerical range based on the sensor type and monitoring environment. For example, in a normal urban environment, the concentration range of PM2.5 is usually within a certain range. If the collected PM2.5 data exceeds this range, the system will conduct further verification and judgment.

[0027] The monitoring platform transmits the monitoring instructions to the monitoring network through data communication. Each collector sub-network executes the corresponding rules according to the monitoring number of the sub-monitoring instruction to verify the accuracy of the collected multimodal data. The verification content covers aspects such as data integrity, data format correctness, and data value rationality.

[0028] When verifying data integrity, check whether there are missing parts in the image data, whether there are interruptions in the audio data, and whether the sensor data is complete without omissions. For the data format, ensure that the image data is in standard JPEG or PNG format, the audio data is in common WAV or MP3 format, and the sensor data meets the specific communication protocol format requirements.

[0029] It should be further explained that, in the specific implementation process, the process of equipment status monitoring includes: The hardware status information of the embedded collector includes CPU usage, which is recorded as H1, memory usage, which is recorded as H2, and battery power, which is recorded as H3. The CPU usage threshold, memory usage threshold, and battery power threshold are preset and recorded as T1, T2, and T3 respectively. Set the corresponding threshold values ​​according to the hardware configuration and running tasks of the embedded collector. For example, for an embedded collector with a lower configuration, set the CPU usage threshold to 70% and the memory usage threshold to 80%. For a battery-powered collector, when the battery power is less than 20%, it is considered to be low on power.

[0030] When H1≥T1, the CPU status is marked as abnormal. The possible reason is that the task run by the embedded collector is too complex, resulting in excessive CPU load. At this time, the system can record the time when the abnormality occurred and the specific value of the CPU usage; When H2 ≥ T2, the memory status is marked as abnormal, which may be caused by program memory leaks or excessive data caches that are not cleaned up in time; When H3≤T3, the battery status is marked as abnormal, reminding the administrator to replace or recharge the battery in time; Otherwise, the CPU status, memory status, and battery status are each normal; Generate a device status report based on the marked abnormal status. The device status report records the normal status area and abnormal status area of ​​the embedded collector. The normal status area records the hardware status information without abnormalities, such as normal CPU usage, normal memory usage, and relevant data when the battery power is sufficient; The abnormal status area records in detail the abnormal hardware status information and its corresponding abnormal type, such as "CPU usage is too high (85%), time: October 10, 2024 10:00", "Memory usage is abnormal (85%), time: October 10, 2024 10:15", etc., which makes it convenient for administrators to timely understand the operating status of the embedded collector and perform targeted maintenance and management.

[0031] It should be further explained that, in the specific implementation process, the fault warning process includes: In order to discover possible faults of embedded collectors in advance, ensure the stable operation of the smart city environment monitoring system, and establish a fault warning mechanism, first, a fault risk model is constructed. The fault risk model contains several risk nodes, and several risk nodes are recorded as R2 = {R21, R22, R23, ..., R2K}; The risk node is determined based on the historical fault data, hardware characteristics, and operating environment of the embedded collector. For example, according to the historical fault data, when the CPU temperature exceeds 80°C for 10 consecutive minutes, the probability of the embedded collector failing is high. Therefore, "CPU temperature 80°C for 10 minutes" is set as a risk node R21; when the memory read and write error rate exceeds 1% within 1 hour, it is also likely to cause a fault, so it is set as a risk node R22; Each risk node has a corresponding parameter range in the fault risk model. Taking risk node R21 as an example, its parameter range is that the CPU temperature reaches 80°C and lasts for 10 minutes. The real-time operating parameter R1 of the embedded collector is obtained, and the CPU temperature and memory read and write error rate are collected in real time through sensors or system monitoring programs.

[0032] Traverse each risk node in the risk node set R2 to obtain the difference between the real-time operating parameters and the risk node parameter range. Take the risk node R21 as an example, calculate the difference between the current CPU temperature and 80°C, and the difference between the current temperature being continuously higher than 80°C and 10 minutes. Combine these factors to get the difference degree, which is recorded as R`. Compare the difference degree R` with the preset risk threshold and perform fault warnings of different levels. Fault warnings are divided into level 1 warning, level 2 warning and level 3 warning; When the difference degree R` approaches but does not reach the risk node parameter range, a third-level warning is issued; for example, when the CPU temperature reaches 75°C and lasts for 8 minutes, a third-level warning is issued to remind the administrator that the collector has potential risks and that the CPU heat dissipation situation needs to be paid attention to; When the difference degree R further approaches the risk node parameter range, a secondary warning is issued; If the CPU temperature reaches 78°C and lasts for 9 minutes, a Level 2 warning is issued and the administrator can take temporary measures, such as suspending some non-critical tasks to reduce the CPU load; When the difference degree R` reaches or exceeds the risk node parameter range, a first-level warning is issued, indicating that the collector is about to fail or has failed. At this time, the system automatically takes emergency measures, such as saving important data, stopping some collection tasks, etc., and notifies the administrator for emergency processing.

[0033] It should be further explained that, in a specific implementation process, the process of performing fault diagnosis according to the real-time feedback information includes: During operation, the embedded collector transmits real-time feedback information to the monitoring platform. In the smart city environment monitoring system, the real-time feedback information includes data collection abnormality information, equipment hardware failure information, and communication failure information. In order to accurately determine whether there is a fault, corresponding fault judgment rules are set for this information; For data collection abnormality information, if the image data collected for five consecutive times appears blurred, or the audio data has continuous noise or signal loss, or the sensor data exceeds the reasonable fluctuation range and repeats multiple times, it is considered as data collection abnormality; for device hardware failure information, when the CPU has frequent erroneous instructions, the memory cannot read and write data normally, or the battery is overheated or overcooled, it is judged as a device hardware failure; communication failure information is judged based on the monitoring platform not receiving the corresponding multimodal data from the embedded collector within a certain period of time (such as 5 minutes), or a large number of error check codes appear during data transmission.

[0034] If the data collection abnormal information, equipment hardware failure information, and communication failure information do not meet the fault judgment rules, it means that the collector is running normally and no subsequent fault diagnosis operation is performed; if any of the information meets the fault judgment rules, the fault diagnosis program is started; The fault diagnosis program analyzes relevant information to identify the fault type. The fault types mainly include software fault type and hardware fault type. For software fault type, it may be caused by loopholes in the acquisition program corresponding to the embedded collector, or errors in the data processing algorithm. For example, if it is found that the color of the collected image data is deviated, after analysis, it is found that there is an error in the color correction part of the image data processing algorithm. This is a software fault. For this kind of software fault, it can be repaired by remotely updating the program, sending the correct program version to the collector, and replacing the problematic program module.

[0035] If it is determined to be a hardware failure type, such as sensor damage, CPU overheating and burning, the monitoring platform will take different measures according to the severity of the failure. For some minor hardware failures, such as the decline in the accuracy of individual sensors, the fault location and type will be marked to remind the administrator to replace it at the appropriate time. For serious hardware failures, such as CPU damage, the system will stop the relevant tasks of the embedded collector to prevent further damage, and notify the administrator to repair or replace it as soon as possible.

[0036] It should be further explained that, in the specific implementation process, the operation management process includes: Operation management covers two situations: operation management in the event of a fault and operation management when the task is completed. Different management tags are assigned to the operation management in the two situations. The management tag of the operation management in the event of a fault is marked as Mg1, and the management tag of the operation management when the task is completed is marked as Mg2. When the management mark is Mg1, that is, a fault occurs, the embedded collector with the fault problem is obtained. In the smart city environmental monitoring system, if an embedded collector has an abnormal data collection failure, the system first suspends the data transmission of the embedded collector to avoid further propagation and use of erroneous data; then, the device functions of the embedded collector are restricted, such as shutting down some non-critical collection tasks and retaining only the collection function of critical data to reduce the equipment load and prevent further deterioration of the fault. At the same time, the system records detailed information about the fault, including the fault type, occurrence time, collector number, etc., to facilitate subsequent fault analysis and maintenance.

[0037] When the management mark is Mg2, that is, the task is completed, in the smart city environmental monitoring system, when the environmental monitoring task of a certain period of time (such as one day) is completed, the monitoring platform issues a stop command. After the embedded collector receives the stop command, it turns off the data collection function. Then, the embedded collector backs up the collected data. It can be backed up through internal storage devices or uploaded to cloud servers to ensure data security and traceability. After completing the data backup, the embedded collector enters sleep mode to reduce power consumption and extend the service life of the equipment. Before the next task starts, the monitoring platform will send a wake-up command, the embedded collector restarts, and resumes normal operation.

[0038] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An embedded collector monitoring method based on synchronous multimodal data acquisition, characterized in that: The following steps are involved: Step S1: construct a multimodal data collection system, obtain multimodal data and input it into the embedded collector monitoring platform, then build a monitoring network, and bind the data communication between the embedded collector monitoring platform and each embedded collector through the monitoring network; Step S2: Transmitting monitoring instructions to the embedded collector through data communication, and then performing operation monitoring on the embedded collector, wherein the operation monitoring includes data accuracy monitoring, equipment status monitoring, and fault early warning; Step S3: obtaining real-time feedback information of the embedded collector during operation, and transmitting it to the embedded collector monitoring platform, and performing fault diagnosis and operation management according to the real-time feedback information; The process of building a multimodal data collection system includes: Determine the type of multimodal data. Multimodal data includes image data, audio data and sensor data, which are marked as D1, D2 and D3 respectively. Set corresponding acquisition modules for different types of multimodal data. Set an independent cache area for each acquisition module. The cache area number corresponds to the acquisition module and is recorded as i, i = 1, 2, 3, ..., n, where n is a natural number greater than 0. The cache area is used to temporarily store the collected data. At the same time, a data integration area is set to integrate the data in each cache area according to the timestamp.

2. The embedded collector monitoring method based on multi-modal data synchronous acquisition according to claim 1 is characterized in that: The process of building the monitoring network and binding the data communication between the monitoring network and the embedded collector includes: Obtain information related to the embedded collector, including the collector number, model, and historical operation data. The collector number is j, j = 1, 2, 3, ..., m, where m is a natural number greater than 0, and is used as an encapsulation index to encapsulate and generate j data packets to be accessed; Establish a temporary access space, input the verification key and j data packets to be accessed, the data packets to be accessed are associated with the access IP address, and a list of legal access IPs is preset in the temporary access space. Determine whether the data packets to be accessed are legal data packets based on the access IP address and the list of legal access IPs; if not, do nothing; if so, generate a verification database request, obtain the verification key through the verification database request, and obtain the monitoring network configuration file through the verification key; According to the monitoring network configuration file and the mapping association of the data packet to be accessed, a collector sub-network is constructed and the corresponding sub-network number K is assigned, where K =<i, j> , the monitoring control center is set up to aggregate several collector sub-networks to build a monitoring network; Set the communication protocol. The communication protocol has a corresponding communication state, which includes a stable protocol state, an interrupted protocol state, and an interference protocol state. In the stable protocol state, the data communication between the embedded collector monitoring platform and the monitoring network is bound by the communication protocol. In the interrupted protocol state and the interference protocol state, an abnormal log is generated and transmitted to the administrator, who will restore the communication and troubleshoot the interference source.

3. The embedded collector monitoring method based on multi-modal data synchronous acquisition according to claim 2 is characterized in that: The data accuracy monitoring process includes: The embedded collector monitoring platform generates a monitoring instruction, which includes several sub-monitoring instructions, denoted as M[K], where K is the subnetwork number of several collector subnetworks, and transmits the monitoring instruction to the monitoring network through data communication; The sub-monitoring instructions are set with different monitoring types. Different monitoring types have corresponding monitoring numbers. The monitoring numbers have corresponding accuracy monitoring rules. The corresponding accuracy monitoring rules are executed according to the monitoring numbers corresponding to the monitoring types, and the accuracy of the collected multimodal data is verified. The verification content includes data integrity, data format correctness and data value rationality.

4. The embedded collector monitoring method based on multi-modal data synchronous acquisition according to claim 3 is characterized in that: The process of equipment status monitoring includes: Get the hardware status information of the embedded collector. The hardware status information includes CPU usage, memory usage and battery power, which are marked as H1, H2 and H3 respectively. The CPU usage threshold, memory usage threshold and battery power threshold are preset, which are marked as T1, T2 and T3 respectively. When H1≥T1, the CPU status is marked as abnormal; when H2≥T2, the memory status is marked as abnormal; when H3≤T3, the battery status is marked as abnormal, otherwise, the CPU status, memory status and battery status are normal; A device status report is generated based on the marked abnormal status. The device status report records the normal status area and abnormal status area corresponding to the embedded collector. The normal status area records the hardware status information without abnormalities, and the abnormal status area records the hardware status information with abnormalities and their corresponding abnormality types.

5. The embedded collector monitoring method based on multi-modal data synchronous acquisition according to claim 4 is characterized in that: The fault warning process includes: Get the real-time operating parameters of the embedded collector, denoted as R1, get several risk nodes in the fault risk model, and then generate a risk node set, denoted as R2, then R2 = {R21, R22, R23, …, R2K}, and the risk node has a corresponding parameter range in the fault risk model; Traverse the parameter ranges corresponding to several risk nodes R21, R22, R23, ..., R2K, and then obtain the difference between the real-time operating parameters and several parameter ranges, compare the difference with the preset risk threshold, and then execute different fault warnings. The fault warnings are associated with corresponding warning levels, including level one warning, level two warning, and level three warning.

6. The embedded collector monitoring method based on multi-modal data synchronous acquisition according to claim 5 is characterized in that: The process of performing fault diagnosis according to the real-time feedback information includes: Obtain the real-time feedback information of the embedded collector and transmit it to the embedded collector monitoring platform. The real-time feedback information includes data collection abnormality information, equipment hardware failure information and communication failure information. Different real-time feedback information is set with corresponding fault judgment rules; If the data collection abnormality information, equipment hardware fault information, and communication fault information do not meet the fault judgment rules, the subsequent fault diagnosis operation will not be performed; If any of the data collection abnormality information, equipment hardware failure information, and communication failure information meets the fault judgment rules, the fault diagnosis program is started, and the fault type is found out by analyzing the relevant information. The embedded collector monitoring platform performs fault diagnosis based on the fault type. The fault diagnosis fault types include software fault types and hardware fault types. The corresponding fault repair measures are performed according to the fault type.

7. The embedded collector monitoring method based on multi-modal data synchronous acquisition according to claim 6 is characterized in that: The operation management process includes: The operation management includes operation management in the event of a fault and operation management when the task is completed, which are respectively assigned different management tags, denoted as Mg1 and Mg2; When the management mark is Mg1, it corresponds to the operation management under the fault condition, obtains the embedded collector with the fault problem, and suspends the data transmission and restricts the device function of the embedded collector; When the management mark is Mg2, the operation management corresponding to the task is completed, the stop instruction of the embedded collector monitoring platform is obtained, the data collection function of all embedded collectors in the monitoring network is turned off, and data backup and device sleep operations are performed.

Images