Task chain-oriented adaptive monitoring method and system

Abstract

The invention relates to a task chain-oriented adaptive monitoring method and system. According to some embodiments, an electronic device is provided, comprising a processor configured to execute a computer program to: split a task chain within a task chain file to obtain a plurality of ordered tasks; reasoning to obtain model attribute information of at least one model required for executing the current task at least based on the text information of the current task; generating at least one model required for executing the current task based on the model attribute information of the at least one model required for executing the current task obtained by reasoning; deploying the at least one generated model; and running the at least one deployed model to obtain a reasoning result of the current task.

Description

Technical Field

[0001] This disclosure generally relates to intelligent monitoring, and more specifically, to adaptive monitoring methods and systems oriented towards task chains. Background Technology

[0002] With the rapid development of image perception and network transmission technologies, especially the booming development of artificial intelligence, intelligent monitoring systems have gradually become widespread and integrated into people's daily lives as a routine technology. However, in some extreme environments, such as the polar regions, deserts, and space covered by scientific expeditions, as well as rescue and exploration scenarios, current intelligent monitoring systems face many challenges in deployment and operation.

[0003] Specifically, on the one hand, existing intelligent monitoring systems for complex tasks are heavily reliant on a robust network environment. This is because they require frequent remote monitoring, image transmission, and over-the-air (OTA) model updates, which significantly hinders their widespread application and effective deployment in extreme environments. On the other hand, in offline environments, intelligent monitoring systems often only perform simple tasks. This is because extreme environments place extremely high demands on the energy consumption, size, and stability of equipment, severely limiting the scope of use of intelligent monitoring devices in offline environments and making it difficult to complete complex sequential tasks. Furthermore, intelligent monitoring systems supporting complex offline tasks typically require substantial storage space, inevitably leading to a significant increase in equipment costs. Complex monitoring tasks often require multiple models to work in stages, and in offline environments, all of these models must be stored on the device. Even with compression of the weighted components, they still consume a large amount of storage space, and compression also implies a loss of accuracy. Moreover, when the environment changes, existing offline intelligent devices often cannot adapt to the new environment, leading to malfunctions. This forces users to purchase new products when entering new environments, further increasing capital investment and the difficulty of equipment maintenance. Summary of the Invention

[0004] This section provides a brief overview of the present disclosure to offer a basic understanding of some aspects thereof. However, it should be understood that this overview is not an exhaustive summary of the present disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is merely to present certain concepts of the disclosure in a simplified form as a prelude to the more detailed description that follows.

[0005] According to one aspect of this disclosure, an electronic device is provided, including a processor. The processor is configured to execute a computer program to: disassemble a task chain within a task chain file to obtain multiple ordered tasks; infer model attribute information of at least one model required to execute the current task, based at least on textual information of the current task; generate the at least one model required to execute the current task based on the inferred model attribute information of the at least one model required to execute the current task; deploy the generated at least one model; and run the deployed at least one model to obtain the inference result of the current task.

[0006] In some embodiments, the model attribute information of the at least one model required to perform the current task includes at least a threshold set for the current task and a model index and model deployment location for each of the at least one model.

[0007] In some embodiments, the model attribute information of the at least one model required to perform the current task may also include the number of models and the model deployment order for each of the at least one model.

[0008] In some embodiments, the processor is further configured to execute a computer program to: obtain a model weight heatmap and model structure information for each of the at least one model based on the model attribute information; and generate each of the at least one model based on the obtained model weight heatmap and model structure information for each of the at least one model.

[0009] In some embodiments, the processor is further configured to execute a computer program for each of the at least one models: after each generation of such a model, incrementing a count value associated with such a model by one; determining whether the count value has reached the number of models of such a model; if the count value has reached the number of models of such a model, stopping the generation of such a model; otherwise, continuing to generate such a model.

[0010] In some embodiments, the processor is further configured to execute a computer program to deploy the at least one model according to the model deployment order and the model deployment location of each model.

[0011] In some embodiments, the model deployment location includes one of a sensor and a processor.

[0012] In some embodiments, the processor is further configured to execute a computer program to: obtain at least one heatmap by running the deployed at least one model; and obtain inference results for the current task based at least on the at least one heatmap.

[0013] In some embodiments, the processor is further configured to execute a computer program to: infer model attribute information required to perform the current task based on textual information of the current task and at least one heatmap obtained for the previous task.

[0014] In some embodiments, the processor is further configured to execute a computer program to: compare the inference result of the current task with a threshold set for the current task; in response to the inference result of the current task satisfying the threshold set for the current task, continue processing the next task in the task chain and clear the model executing the current task; and in response to the inference result of the current task not satisfying the threshold set for the current task, not continue processing the next task in the task chain.

[0015] In some embodiments, the processor is further configured to execute a computer program to: in response to the inference result of the current task not meeting the threshold set for the current task, repeatedly run the deployed at least one model within a preset time period until the inference result of the current task meets the threshold set for the current task; and if the inference result of the current task does not meet the threshold set for the current task throughout the preset time period, stop running the deployed at least one model and enter a sleep state.

[0016] In some embodiments, the processor is further configured to execute a computer program to: receive a user selection of an execution mode before unpacking the task chains within the task chain file; download the task chain file remotely in response to the user selecting a low-power online mode; and embed the task chain file into the processor in response to the user selecting an offline mode.

[0017] In some embodiments, the text information of the current task includes at least one of the following: task role, task scenario, and task objective.

[0018] In some embodiments, the task objective includes multiple sub-task objectives and the dependencies between the multiple sub-task objectives.

[0019] According to another aspect of this disclosure, an adaptive monitoring method for task chains is provided, comprising: disassembling task chains within a task chain file to obtain multiple ordered tasks; inferring model attribute information of at least one model required to execute the current task based at least on the text information of the current task; generating the at least one model required to execute the current task based on the inferred model attribute information of the at least one model required to execute the current task; deploying the generated at least one model; and running the deployed at least one model to obtain the inference result of the current task.

[0020] In some embodiments, the model attribute information of the at least one model required to perform the current task includes at least a threshold set for the current task and a model index and model deployment location for each of the at least one model.

[0021] In some embodiments, the model attribute information of the at least one model required to perform the current task may also include the number of models and the model deployment order for each of the at least one model.

[0022] In some embodiments, the operation of generating at least one model required to perform the current task based on model attribute information of at least one model derived from inference further includes: obtaining a model weight heatmap and model structure information of each of the at least one model based on the model attribute information; and generating each of the at least one models based on the obtained model weight heatmap and model structure information of each of the at least one model.

[0023] In some embodiments, the operation of generating at least one model required to perform the current task based on model attribute information of at least one model required to perform the current task derived from reasoning further includes, for each of the at least one models: after each generation of such a model, incrementing a count value associated with such a model by one; determining whether the count value has reached the number of models of such a model; if the count value has reached the number of models of such a model, stopping the generation of such a model; otherwise, continuing to generate such a model.

[0024] In some embodiments, the method further includes: deploying the at least one model according to the model deployment order and the model deployment location of each model.

[0025] In some embodiments, the model deployment location includes one of a sensor and a processor.

[0026] In some embodiments, the method further includes: obtaining at least one heatmap by running the deployed at least one model; and obtaining inference results for the current task based at least on the at least one heatmap.

[0027] In some embodiments, the operation of inferring model attribute information of at least one model required to perform the current task based at least on the text information of the current task further includes: inferring model attribute information required to perform the current task based on the text information of the current task and at least one heatmap obtained for the previous task of the current task.

[0028] In some embodiments, the method further includes: comparing the inference result of the current task with a threshold set for the current task; in response to the inference result of the current task satisfying the threshold set for the current task, continuing to process the next task in the task chain and clearing the model executing the current task; and in response to the inference result of the current task not satisfying the threshold set for the current task, not continuing to process the next task in the task chain.

[0029] In some embodiments, the method further includes: in response to the inference result of the current task not meeting the threshold set for the current task, repeatedly running the deployed at least one model within a preset time period until the inference result of the current task meets the threshold set for the current task; and if the inference result of the current task does not meet the threshold set for the current task within the preset time period, stopping the running of the deployed at least one model and entering a dormant state.

[0030] In some embodiments, the method further includes: receiving a user's selection of an execution mode before disassembling the task chain within the task chain file; downloading the task chain file remotely in response to the user selecting a low-power online mode; and embedding the task chain file into the processor in response to the user selecting an offline mode.

[0031] In some embodiments, the text information of the current task includes at least one of the following: task role, task scenario, and task objective.

[0032] In some embodiments, the task objective includes multiple sub-task objectives and the dependencies between the multiple sub-task objectives.

[0033] According to another aspect of this disclosure, a non-transient computer-readable storage medium is provided, on which a computer program is stored, which, when executed by one or more processors of a system, performs the method described above.

[0034] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by one or more processors, causes the method described above to be performed. Attached Figure Description

[0035] This disclosure can be better understood by referring to the detailed description given below in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar elements. All the drawings, together with the following detailed description, are incorporated in and form a part of this specification, and are used to further illustrate embodiments of this disclosure and explain the principles and advantages of this disclosure. Wherein:

[0036] Figure 1An exemplary structural diagram of a task-chain-based adaptive monitoring system according to some embodiments of the present disclosure is shown.

[0037] Figure 2 An exemplary flowchart of a task chain-based adaptive monitoring method according to some embodiments of the present disclosure is shown.

[0038] Figure 3 An exemplary flowchart of a task chain-based adaptive monitoring method according to an embodiment of the present disclosure is shown.

[0039] Figure 4 A flowchart illustrating an example method for generating a model according to an embodiment of this disclosure is shown.

[0040] Figure 5 A flowchart illustrating an example method performed according to an embodiment of the present disclosure when multiple models of a certain type need to be generated for a given model.

[0041] Figure 6 An exemplary flowchart of a task chain-based adaptive monitoring method according to an embodiment of the present disclosure is shown.

[0042] Figure 7 A block diagram illustrating an exemplary hardware configuration of a computer system capable of implementing embodiments of the present invention is shown.

[0043] The features and aspects of this disclosure will become clear from the following detailed description taken in conjunction with the accompanying drawings. Detailed Implementation

[0044] Various exemplary embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. For clarity and brevity, not all implementations of the embodiments are described in this specification. However, it should be noted that many implementation-specific settings can be made when implementing embodiments of this disclosure to meet specific needs and achieve the developer's specific goals. Furthermore, it should be understood that while development work may be complex and time-consuming, such development is merely a routine task for those skilled in the art who benefit from this disclosure.

[0045] Furthermore, it should be noted that, in order to avoid obscuring this disclosure with unnecessary details, only processing steps and / or device structures closely related to the technical solutions of this disclosure are shown in the accompanying drawings. The following description of exemplary embodiments is merely illustrative and is not intended to limit this disclosure or its application in any way.

[0046] In recent years, remote monitoring systems have been widely used in many fields such as home monitoring, production monitoring, security monitoring, robotics, and scientific research and exploration. However, under extreme and non-ideal conditions, when meeting certain highly complex requirements, limitations such as poor or no network transmission signals, storage capacity, power consumption, and device size still exist. Therefore, this paper proposes a task-chain-based adaptive monitoring system. This system, even under low or no transmission conditions, can ensure that highly complex tasks can autonomously adjust to the environment and achieve functionality on low-power, small-size devices.

[0047] Figure 1 An exemplary structural diagram of a task-chain-based adaptive monitoring system 100 according to some embodiments of the present disclosure is shown.

[0048] like Figure 1 As shown, system 100 may include sensor 101 and processor 102.

[0049] Those skilled in the art will understand that, according to embodiments of this disclosure, sensor 101 can be any suitable sensor, including but not limited to visual sensors for, for example, object recognition, environmental perception, target tracking, etc.; infrared sensors for, for example, detecting the heat distribution of an object for obstacle avoidance, temperature perception, etc.; lidar for, for example, accurately measuring distance, building environmental maps for precise positioning and navigation, etc.; ultrasonic sensors for, for example, detecting distance for, for example, short-range obstacle avoidance; force sensors for sensing forces and torques acting on the robot (e.g., force feedback when grasping an object, etc.); microphones for sound recognition and environmental sound detection; magnetic sensors for detecting magnetic fields, assisting navigation, etc.; and pressure sensors for detecting pressure conditions on contact surfaces.

[0050] In some embodiments, sensor 101 may be a smart image sensor. Smart image sensors generally capture clear, detailed images, providing rich detail information. They can quickly acquire and process image information, enabling real-time monitoring and response. They can effectively reduce noise interference in images and improve image quality. Smart image sensors possess certain image processing and analysis functions, such as autofocus and target recognition. Smart image sensors are generally relatively small in size, facilitating integration into various devices. They have low power consumption, extending device battery life. They can easily connect and interact with other digital systems and adapt to various complex environments and application scenarios.

[0051] The processor 102 can be a general-purpose processor or a dedicated processor. The processor 102 can be connected to various sensors to receive and process various sensing signals.

[0052] In some embodiments, the processor 102 may be an intelligent image signal processor. Intelligent image signal processors generally possess powerful image processing capabilities, enabling them to perform various complex algorithmic processing on input image data, such as noise reduction, sharpening, and color correction, thereby significantly improving image quality. Secondly, intelligent image signal processors are highly intelligent, capable of automatically analyzing and understanding image content, and realizing intelligent functions such as target detection, recognition, and tracking. Furthermore, intelligent image signal processors have excellent real-time processing performance, capable of rapidly processing large amounts of image data, ensuring that real-time image display and processing are without delay. Intelligent image signal processors can also adapt to different types of image sensors, exhibiting strong compatibility, and can be flexibly configured and programmed according to different application requirements. In addition, intelligent image signal processors typically have low power consumption, which is crucial for mobile devices or applications with high power consumption requirements. Intelligent image signal processors also support multiple data interfaces and transmission protocols, facilitating efficient data interaction with other systems or devices.

[0053] Figure 2 An exemplary flowchart of a task chain-based adaptive monitoring method 200 according to some embodiments of the present disclosure is shown.

[0054] For ease of illustrative purposes, the description will use an intelligent image sensor 201 and an intelligent image signal processor 202 as examples. However, those skilled in the art will understand that the methods and systems according to embodiments of this disclosure can be applied to any suitable sensor and processor.

[0055] like Figure 2 As shown, processor 202 may include task chain decomposer 203, multimodal discrimination model 204, model generator and deployer 205, inference runtime 206 and postprocessor 207, wherein model generator and deployer 205 may include model weight generator 2051, model structure exporter 2052 and model assembly and deployment interface 2053.

[0056] like Figure 2 As shown, method 200 can begin with operation ①, in which the user can select a low-power online mode or an offline mode.

[0057] Based on the user's choice of operating mode, operation ② is executed. In this operation, if the user selects the low-power online mode, the task chain file can be downloaded to the processor 202 via OTA; if the user selects the offline mode, the task chain file can be pre-frozen to the processor 202.

[0058] The task chain file can include basic information such as one or more of the following: task role, task scenario, and task objective. For example, based on a pre-defined list of scenarios, the basic information can be broken down into detailed information. This detailed information may include, for example, the task role, task scenario, task objective, and the dependencies between task objectives. A task objective can include multiple sub-task objectives and the dependencies between them.

[0059] Taking the adaptive monitoring method based on task chains according to some embodiments of this disclosure applied to a rescue robot as an example, the basic information of the task chain may include performing a rescue and entering a standby state after the task is completed. The detailed information of the task chain may include: determining the current scene name; determining whether there are signs of life; determining the number of living beings; determining the status of signs of life; determining whether to take rescue action; determining whether communication is possible, and if communication is possible, communicating with the control console according to the scene and storing local information simultaneously; if not, only storing local information; and entering a standby state after the task is completed.

[0060] like Figure 2 As shown, the processing of the task chain begins in operation ③, in which the task chain decomposer 203 decomposes the task chain file to be processed.

[0061] The task chain file containing a series of ordered tasks is broken down layer by layer.

[0062] Taking the rescue robot example above as an example, the task chain is broken down as follows: Determine the name of the current scene; → Determine if there are signs of life (subtask 1: Determine if there is a target life form (human, animal, etc.), → Subtask 2: Determine if there is a life form) → Determine whether to take rescue action (subtask 1: Determine if rescue is possible, → Subtask 2: How to rescue) → Determine if communication is possible → Whether the task has been completed (if completed, enter standby mode).

[0063] After completing the layer-by-layer decomposition of the task chain, the task chain decomposer 203, starting from the lowest-level task it depends on, delivers the text information of the task to be processed to the multimodal discrimination model 204. In operation ④, the multimodal discrimination model 204, based on the received text information of the current task, infers the model attribute information of at least one model required to execute the current task. The task processed by the multimodal discrimination model 204 can be a subtask with the smallest granularity.

[0064] The multimodal discrimination model 204 can infer the model attribute information of the model required to perform the current task based on the text information of the current task and the first heat map from the intelligent image sensor 201 and the second heat map from the intelligent image signal processor 202.

[0065] The first and second heatmaps are based on the results of the execution of the previous task of the current task. When the current task is the initial task in a task chain, the first and second heatmaps can be empty or contain predefined fixed values.

[0066] A heatmap is a visualization method that uses color coding to represent data, showing the magnitude or intensity of values ​​in different regions of an image. Different heatmaps have different meanings. For example, in a heatmap used for human keypoint detection, the value of each pixel represents the probability or confidence that the point is a keypoint. Darker areas (usually warm colors like red) indicate a higher probability that the point is a keypoint, while lighter areas (usually cool colors like blue) indicate a lower probability. This method visually presents the possible locations of various keypoints in an image and their relative probabilities, thus aiding in the analysis and understanding of human posture and movement. It provides a visual probability distribution representation for determining the specific location of key human body parts. For example, in a heatmap used for edge detection, areas with higher edge intensity will be displayed as brighter colors. Similarly, in a heatmap used to detect moving regions in an image, areas with higher motion intensity will be displayed as brighter colors, while stationary areas will be displayed as darker colors.

[0067] The models required to perform the current task can include multiple models. The multimodal discrimination model 204 can infer model attribute information for each model, including the model index (indicating which type of model it is), the number of models (indicating how many of this type of model to generate), the model deployment location (where this type of model should be deployed, for example, on the intelligent image sensor 201 or the intelligent image signal processor 202), the model deployment order (assuming that performing the current task requires a series of multiple models, where this type of model is deployed in this series of multiple models), and a threshold (used to determine whether the result obtained by running the model for the current task indicates success).

[0068] If only one model is required to perform the current task, the number of models and the model deployment order can be omitted, or the values ​​for the number of models and the model deployment order can be set to 0 or null.

[0069] Thresholds can be used to determine the performance of a model used to perform the current task.

[0070] In some embodiments, a threshold can be set only for the current task. That is, the threshold can be set by comprehensively considering the results of running all models required to perform the current task. In this case, it is not necessary to infer a corresponding threshold for each model required to perform the current task.

[0071] In other embodiments, a threshold may be set for each model required to perform the current task. In this case, the threshold set for the current task may be a vector containing the thresholds set separately for each model.

[0072] It is understandable that the multimodal discrimination model 204 can be a pre-trained artificial intelligence model. Its training dataset includes task text information, a first heatmap, and a second heatmap as input, and includes corresponding model attribute information as output.

[0073] As shown in the figure, the model attribute information obtained by the multimodal discrimination model 204 is provided to the model generator and the deployer 205. Specifically, the model index information is provided to the model weight generator 2051 and the model structure deriver 2052.

[0074] like Figure 2 As shown in operation ⑧, for each model, if the number of generated models i is less than the number of models, an enable signal is provided to the model weight generator 2051 and the model structure deriver 2052.

[0075] Model weight generator 2051 generates a corresponding model weight heatmap based on the model index, as shown in operation ⑤. The model weight heatmap can indicate, for example, the importance distribution (showing the relative importance of each parameter (weight) in the model; darker areas (usually representing larger values) may represent the parts that have a greater impact on the model's decision or output) and feature correlation (revealing the degree of correlation between input features and model output, helping to understand which features are more critical to the model).

[0076] The Model Structure Exporter 2052 exports model structure information based on the model index, as shown in operation ⑥. Model structure information can include, for example, indicating layer types and order (clearly specifying which types of layers the model contains, such as convolutional layers, fully connected layers, etc., and their order, which determines the data processing flow in the model), the number of parameters, displaying the number of parameters for each layer and the entire model (related to the model's complexity and computational requirements), connection relationships (indicating the connection methods between layers, including how information is passed and computations are performed), and overall architectural characteristics (helping to understand the architectural design philosophy upon which the model is built, such as a convolutional neural network architecture for image recognition or a recurrent neural network architecture for natural language processing, etc.).

[0077] As a specific example, suppose the multimodal discrimination model 204 determines that the model for the first sub-task of the current rescue mission is a classification network. In this case, it recommends the classification network index (previously set application network index) that has the best matching accuracy and the highest efficiency, such as Mnasnet (index 0001). Then, the model weight generator 2051 generates the weight heatmap of Mnasnet based on the model index 0001, and the model structure deriver 2052 derives the network structure information of Mnasnet based on the index 0001, such as a Directed Acyclic Graph (DAG) structure.

[0078] As shown in operation ⑦, after the reasoning of the model weights and structural information of a model of a certain type is completed, the counter value i associated with that type of model is incremented by 1, which is used to determine at operation ⑧ whether the generation of the number of models associated with that type of model has been completed.

[0079] In some embodiments, the weight generation and structure derivation of multiple models can be performed sequentially in a certain order. In other embodiments, the weight generation and structure derivation of at least some of the multiple models can also be performed in parallel.

[0080] As shown in operation ⑨, the model weight heatmap generated by the model weight generator 2051 and the model structure information exported by the model structure exporter 2052 are provided to the model assembly and deployment interface 2053.

[0081] The Model Assembly and Deployment Interface 2053 assembles the weights and structure of each model to generate the model for each model, as shown in Operation 9.

[0082] The model assembly and deployment interface 2053, for each model, deploys each model to its corresponding deployment location according to the model deployment order associated with that model. In other words, it queues the various models performing the current task according to the order in which they are to be deployed, and then deploys them to their respective locations in that queue order. For example, suppose there are 10 models performing the current task, and 3 of them are models with index 0001, which can be deployed in the first, sixth, and seventh order, respectively. Then, models with index 0001 are deployed at the first, sixth, and seventh positions.

[0083] As shown in operation ⑩, for each model, it is necessary to determine whether the model is deployed at the smart image sensor 201 or the smart image signal processor 202. If it is determined that the model is deployed at the smart image sensor 201, the model is burned to the smart image sensor 201 for execution at the smart image sensor 201. If it is determined that the model is deployed at the smart image signal processor 202, the model is sent to the inference runtime 206 within the smart image sensor 201 for execution at the inference runtime 206.

[0084] The deployment location of a model can be determined by one or more factors. These factors include, for example, the computational complexity involved. For instance, tracking human motion or distinguishing between different people may involve complex calculations, thus the corresponding model is more suitable for deployment on a device with greater computing power and higher performance (e.g., the intelligent image signal processor 202). These factors also include the model's position in the overall inference process; for example, a model closer to the front end can be deployed at the intelligent image sensor 201, while a model closer to the back end can be deployed at the intelligent image signal processor 202. These factors also include the characteristics of the data involved in the task itself; for example, if a task only involves the analysis of audio data and not image data, then the corresponding model can be deployed at the intelligent image signal processor 202.

[0085] After all the models required to perform the current task are deployed, run these models to obtain the inference results for the current task.

[0086] Specifically, running one or more models deployed at the smart image sensor 201 will yield one or more first heatmaps. The first heatmaps may be based on the analysis of image signals captured by the smart image sensor 201.

[0087] Running one or more models deployed at the intelligent image signal processor 202 will yield one or more second heatmaps. The second heatmaps can be based on the analysis of any suitable type of data processed by the image signal processor 202. For example, the second heatmaps can be based on the analysis of image signals, audio signals, video signals, etc. In some embodiments, the second heatmaps can be based on stability analysis, etc.

[0088] In some embodiments, a corresponding heatmap will be obtained for each model.

[0089] The first and second heatmaps generated for the current task can be provided to the post-processor 207.

[0090] In some embodiments, the inference runtime 206 may also provide control information to the post-processor 207 to control the accurate operation of the post-processing module. The control information may indicate, for example, the task type (e.g., classification, detection (general detection, keypoint detection, etc.) or whether to receive or use results provided by a smart image sensor). Different control information may be provided for each task.

[0091] The post-processor 207 can obtain the final inference result of the current task based on the first heatmap, the second heatmap, and the control information, such as operation. As shown.

[0092] As in operation As shown, the final inference result of the current task obtained by the post-processor 207 is compared with the threshold of the current task.

[0093] If the final inference result of the current task meets the threshold, the next task can be processed. In this case, the model generator and deployer 205 can be reset, the cache cleared, and the counter reset to zero. The task chain decomposer 203 can then send the text information of the next task to the multimodal discrimination model 204, and repeat operation ③ to... When all tasks in the task chain have been successfully executed, operation ③ is complete. Then, the task chain processing ends.

[0094] In some embodiments, in the low-power online mode, after the current task is completed, the execution result of the current task can be sent to the remote control terminal, and the remote control terminal can be waited for an instruction to determine whether to continue processing the next task.

[0095] If the final inference result of the current task does not meet the threshold, the processing of the next task is not continued. In some embodiments, if the final inference result of the current task does not meet the threshold, the inference output of the intelligent image sensor 201 and the intelligent image signal processor 202 is enabled continuously for a preset time period. In other words, during this preset time period, the generation and deployment of the model are not repeated, but the already generated and deployed model is repeatedly run to obtain the inference output, waiting for the successful completion of the current task. For example, if the current task is to detect a target object, and the target object does not appear, the already generated and deployed model can be repeatedly run until the target object is detected. During this preset time period, other parts of the system can enter a low-power mode to reduce energy consumption. If the current task is not completed after the preset time period, the system can enter a sleep state. In some embodiments, the system can wake up from the sleep state at certain time intervals and run the already generated and deployed model. In some embodiments, if the current task still cannot be completed after multiple wake-ups, the wake-up time interval can be gradually extended.

[0096] In some embodiments, in low-power online mode, if the current task is not completed after a preset time period, the remote control terminal can be notified that the current task is not completed and wait for further instructions from the remote control terminal to decide how to proceed.

[0097] Those skilled in the art will understand that various rules can be designed according to the needs of the application scenario.

[0098] As mentioned above, different types of thresholds can be set for different tasks.

[0099] For example, when determining key points on the human body, 16 heatmaps representing 16 key points can be generated, and the coordinates of the maximum value point and the corresponding confidence score can be determined in each heatmap. The confidence score corresponding to each key point can be compared with a pre-set confidence threshold.

[0100] For example, when using Faster R-CNN to identify meaningful content, you can consider confidence thresholds (e.g., indicating the confidence of the prediction), overlap thresholds (e.g., indicating the degree of overlap between the detected target and the real target), or category-related thresholds (e.g., indicating the confidence of being identified as a certain category).

[0101] Those skilled in the art can design different thresholds based on the application scenario.

[0102] In this example embodiment, the intelligent image signal processor 202 itself has storage capabilities. However, those skilled in the art will understand that, if the processor 202 has a memory connected to it, the task chain file can be pre-stored in memory accessible to the processor 202.

[0103] Understandable. Figure 2 This is merely one specific example of a task-chain-based adaptive monitoring method / system according to embodiments of this disclosure. Those skilled in the art can omit, add, or modify some components or operations as needed. For example, in some embodiments, the post-processor 207 can be omitted for simple tasks. In some embodiments, the model required to execute the second task can be generated and deployed simultaneously with the processing of the first task, and the deployed model for executing the second task can be run directly after the first task is completed to achieve better real-time performance.

[0104] Figure 3 An exemplary flowchart of a task chain-based adaptive monitoring method 300 according to an embodiment of the present disclosure is shown.

[0105] like Figure 3As shown, method 300 may include operation 301, in which the task chain within the task chain file is disassembled to obtain multiple ordered tasks.

[0106] Method 300 may further include operation 303, at which, based at least on the textual information of the current task, model attribute information of at least one model required to perform the current task is inferred.

[0107] The model attribute information of the at least one model required to perform the current task may include at least a threshold set for the current task and a model index and model deployment location for each of the at least one model.

[0108] As described above Figure 2 The threshold set for the current task can be an overall value set for the current task, or it can be a vector that includes thresholds corresponding to each of the at least one model.

[0109] In cases where the at least one model involves multiple models, the model attribute information of the at least one model required to perform the current task may further include the number of models and the order of model deployment for each model.

[0110] Method 300 may further include operation 305, in which the at least one model required to perform the current task is generated based on model attribute information of the at least one model required to perform the current task, derived by reasoning.

[0111] Figure 4 A flowchart is shown for an example method 400 for generating a model according to an embodiment of the present disclosure.

[0112] like Figure 4 As shown, method 400 may include operation 401, in which, based on model attribute information of at least one model required to perform the current task, model weight heatmap and model structure information of each of the at least one model are obtained.

[0113] In some embodiments, a model weight heatmap corresponding to the model index is generated based on the model index included in the model attribute information, and the model structure information corresponding to the model index is exported.

[0114] Method 400 may further include operation 403, in which each model of the at least one model is generated based on the obtained model weight heatmap and model structure information of each model in the at least one model.

[0115] In some embodiments, a model weight heatmap corresponding to a model index and model structure information corresponding to the model index are assembled to generate a model corresponding to the model index.

[0116] Figure 5 A flowchart illustrating an example method 500 performed according to an embodiment of the present disclosure when multiple models of a certain type need to be generated for a given model.

[0117] like Figure 5 As shown, for a given model, after each generation of that model, the count associated with that model is incremented by one.

[0118] Method 500 may include operation 503, in which it is determined whether the count value has reached the number of models of that type.

[0119] If it is determined in operation 503 that the count value has reached the number of models of this type, then in method 500, operation 505 is performed, in which the generation of this type of model is stopped.

[0120] If it is determined in operation 503 that the count value has not reached the number of models of that type, then in method 500, operation 507 is performed, in which the model of that type is generated again.

[0121] Continue to refer to Figure 3 Method 300 may further include operation 307, at which the generated at least one model is deployed.

[0122] The deployment of at least one model can be based on the model deployment order and the model deployment location for each model. As described above, each model of a type is deployed at the model deployment location corresponding to that type of model, according to the model deployment order.

[0123] Method 300 may further include operation 309, whereby the deployed at least one model is run to obtain the inference results for the current task.

[0124] In some embodiments, at least one heatmap is obtained by running the deployed at least one model (e.g., as shown in the image). Figure 2 The at least one first heatmap from image sensor 201 and / or at least one second heatmap from image signal processor 202, as shown, and the inference results of the current task are obtained based on at least the at least one heatmap.

[0125] The task-chain-based adaptive monitoring method according to embodiments of this disclosure can complete complex monitoring tasks in the absence of a network or on a low-bandwidth network. By breaking down the task chain and automatically generating the required models sequentially for each task, it eliminates the need to pre-store all models required for the task chain. Furthermore, during model generation, it avoids generating all models needed for the entire task chain at once, and the models and storage space required for executing a task can be cleared after its completion. Therefore, it significantly reduces the storage space required for complex multi-model systems. Breaking down complex task chains and generating models for each task allows complex and variable systems to run on miniaturized, low-power devices. Moreover, breaking down the task chain and generating different applicable models for different tasks gives the method and system according to embodiments of this disclosure an adaptive and self-adjusting capability to cope with complex environments.

[0126] Figure 6 An exemplary flowchart of a task chain-based adaptive monitoring method 600 according to an embodiment of the present disclosure is shown.

[0127] like Figure 6 As shown, method 600 may include operation 601, in which the inference result of the current task is compared with a threshold set for the current task.

[0128] In response to the inference result of the current task satisfying the threshold set for the current task, method 600 proceeds to operation 603, in which the next task in the task chain is processed and the model executing the current task is cleared.

[0129] In response to the inference result of the current task not meeting the threshold set for the current task, method 600 proceeds to operation 605, in which the next task in the task chain is not processed.

[0130] Operation 605 may include operation 605-1, in which the deployed at least one model is repeatedly run within a preset time period until the inference result of the current task meets the threshold set for the current task.

[0131] Operation 605 may also include operation 605-2, in which if the inference result of the current task does not meet the threshold set for the current task within a preset time period, the at least one deployed model is stopped and enters a dormant state.

[0132] The functions of the various elements disclosed herein can be implemented using circuitry or processing circuitry including general-purpose processors, application-specific processors, integrated circuits, AISC (“Application-Specific Integrated Circuits”), conventional circuits, and / or combinations thereof, which are configured or programmed to perform the disclosed functions. A processor is considered a circuit or processing circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or component is hardware that performs or is programmed to perform the described functions. The hardware can be any hardware disclosed herein or otherwise known that is programmed or configured to perform the described functions. When the hardware is a processor (which is considered a circuit), the circuit, unit, or component is a combination of hardware and software used to configure the hardware and / or the processor.

[0133] For example, Figure 2 The task chain decomposer 203, multimodal discrimination model 204, model generator and deployer 205 (including model weight generator 2051, model structure deriver 2052 and model assembly and deployment interface 2053), inference runtime 206 and post-processor 207 shown can all be implemented by a general-purpose processor, a special-purpose processor, an integrated circuit, an AISC (“Application-Specific Integrated Circuit”), conventional circuits and / or combinations thereof.

[0134] It should be understood that the functions according to the embodiments of this disclosure can also be implemented by software and / or firmware. In the case of implementation by software and / or firmware, the signal from a storage medium or network to a computer with a dedicated hardware architecture, such as... Figure 7 The general-purpose computer 1100 shown is equipped with the programs that constitute the software, and when various programs are installed, the computer is able to perform various functions, etc. Figure 7 This is a block diagram illustrating an example structure of a computer with an optimized device that may be employed in an embodiment of the present disclosure.

[0135] In one example, the computer may correspond to the exemplary monitoring device described above according to embodiments of this disclosure. At least some modules included in the detection device according to embodiments of this disclosure can be utilized by… Figure 7 The central processing unit of the general-purpose computer 1100 shown executes associated program instructions to carry out this process.

[0136] exist Figure 7 In this system, the central processing unit (CPU) 1101 performs various processes based on the program stored in the read-only memory (ROM) 1102 or the program loaded into the random access memory (RAM) 1103 from the storage section 1108. The RAM 1103 also stores, as needed, the data required when the CPU 1101 performs various processes.

[0137] It is understood that in some embodiments, the memory may be integrated into the central processing unit.

[0138] CPU 1101, ROM 1102 and RAM 1103 are connected to each other via bus 1104. Input / output interface 1105 is also connected to bus 1104.

[0139] The following components are connected to the input / output interface 1105: input section 1106, including a keyboard, mouse, etc.; output section 1107, including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 1108, including a hard disk, etc.; and communication section 1109, including a network interface card, such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network, such as the Internet.

[0140] As needed, drive 1110 is also connected to input / output interface 1105. Removable media 1111, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 1110 as needed, so that computer programs read from them can be installed into storage section 1108 as needed.

[0141] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1111.

[0142] Those skilled in the art will understand that such storage media are not limited to Figure 7 The illustration shows a removable medium 1111 that stores a program and is distributed separately from the device to provide the program to the user. Examples of removable media 1111 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-discs (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 1102, a hard disk included in storage section 1108, etc., containing programs and distributed to the user along with the device containing them.

[0143] It should be noted that the methods and devices described herein can be implemented as software, firmware, hardware, or any combination thereof. Some components may be implemented, for example, as software running on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or application-specific integrated circuits (ASICs).

[0144] Furthermore, the methods and systems of the present invention can be implemented in various ways. For example, the methods and systems of the present invention can be implemented by software, hardware, firmware, or any combination thereof. The order of steps of the method described above is merely illustrative, and unless otherwise specifically stated, the steps of the method of the present invention are not limited to the order specifically described above. In addition, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers recording media storing programs for implementing the method according to the present invention. Such storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0145] Those skilled in the art will recognize that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial overlap in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations, and substitutions are equally possible. Therefore, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

[0146] [Exemplary Implementation of this Disclosure]

[0147] Based on the embodiments of this disclosure, various implementations of the concepts of this disclosure are conceivable, including but not limited to:

[0148] 1. An electronic device, comprising:

[0149] Processor, the processor being configured to execute computer programs to

[0150] Disassemble the task chain within the task chain file to obtain multiple ordered tasks;

[0151] Based at least on the textual information of the current task, infer the model attribute information of at least one model required to execute the current task;

[0152] Based on the model attribute information of the at least one model required to execute the current task derived from reasoning, the at least one model required to execute the current task is generated.

[0153] Deploy the generated at least one model; and

[0154] Run the deployed model to obtain the inference results for the current task.

[0155] 2. The electronic device as described in item 1, wherein the model attribute information of the at least one model required to perform the current task includes at least a threshold set for the current task and a model index and model deployment location for each of the at least one model.

[0156] 3. The electronic device as described in item 2, wherein the model attribute information of the at least one model required to perform the current task further includes the number of models and the model deployment order for each of the at least one model.

[0157] 4. The electronic device as described in item 2 or 3, wherein the processor is further configured to execute a computer program to:

[0158] Based on the model attribute information, obtain the model weight heatmap and model structure information for each of the at least one model; and

[0159] Based on the obtained model weight heatmap and model structure information of each of the at least one models, each model of that type of model in the at least one model is generated.

[0160] 5. The electronic device of claim 3, wherein the processor is further configured to execute a computer program to:

[0161] For each of the at least one model:

[0162] Each time a model of this type is generated, the count associated with that model is incremented by one;

[0163] Determine whether the count value reaches the number of models of this type;

[0164] If the count value is determined to have reached the number of models of that type, then stop generating that type of model;

[0165] Otherwise, continue generating this type of model.

[0166] 6. The electronic device of claim 3, wherein the processor is further configured to execute a computer program to:

[0167] Deploy at least one model according to the deployment order and deployment location of each model.

[0168] 7. The electronic device as described in item 2, wherein the model deployment location includes one of a sensor and a processor.

[0169] 8. The electronic device of claim 1, wherein the processor is further configured to execute a computer program to:

[0170] At least one heatmap is obtained by running the deployed at least one model; and

[0171] The reasoning results for the current task are obtained based on at least one of the heatmaps.

[0172] 9. The electronic device of claim 8, wherein the processor is further configured to execute a computer program to:

[0173] Based on the textual information of the current task and at least one heatmap obtained from the previous task, the model attribute information required to execute the current task is inferred.

[0174] 10. The electronic device of claim 2, wherein the processor is further configured to execute a computer program to:

[0175] Compare the inference result of the current task with the threshold set for the current task;

[0176] If the inference result of the current task meets the threshold set for the current task, continue processing the next task in the task chain and clear the model executing the current task; and

[0177] If the inference result of the current task does not meet the threshold set for the current task, the next task in the task chain will not be processed.

[0178] 11. The electronic device of claim 10, wherein the processor is further configured to execute a computer program to:

[0179] In response to the inference result of the current task not meeting the threshold set for the current task, the deployed at least one model is repeatedly run within a preset time period until the inference result of the current task meets the threshold set for the current task; and

[0180] If the inference result of the current task does not meet the threshold set for the current task within a preset time period, the deployment of at least one model will be stopped and the system will enter a dormant state.

[0181] 12. The electronic device of claim 1, wherein the processor is further configured to execute a computer program to:

[0182] Before disassembling the task chain within the task chain file, receive the user's selection of the execution mode;

[0183] In response to the user-selected execution mode as low-power online mode, the task chain file is downloaded remotely; and

[0184] In response to the user's selection of offline execution mode, the task chain file is persisted to the processor.

[0185] 13. The electronic device as described in item 1, wherein the text information of the current task includes at least one of task role, task scenario, and task objective.

[0186] 14. The electronic device of claim 13, wherein the task objective includes a plurality of sub-task objectives and dependencies between the plurality of sub-task objectives.

[0187] 15. An adaptive monitoring method for task chains, comprising:

[0188] Disassemble the task chain within the task chain file to obtain multiple ordered tasks;

[0189] Based at least on the textual information of the current task, infer the model attribute information of at least one model required to execute the current task;

[0190] Based on the model attribute information of the at least one model required to execute the current task derived from reasoning, the at least one model required to execute the current task is generated.

[0191] Deploy the generated at least one model; and

[0192] Run the deployed model to obtain the inference results for the current task.

[0193] 16. The task-chain-oriented adaptive monitoring method as described in item 15, wherein the model attribute information of the at least one model required to execute the current task includes at least a threshold set for the current task and a model index and model deployment location for each of the at least one model.

[0194] 17. The task-chain-oriented adaptive monitoring method as described in item 16, wherein the model attribute information of the at least one model required to execute the current task further includes the number of models and the model deployment order for each of the at least one models.

[0195] 18. The task-chain-oriented adaptive monitoring method as described in item 16 or 17, wherein the operation of generating the at least one model required for executing the current task based on model attribute information of the at least one model required for executing the current task derived from inference further includes:

[0196] Based on the model attribute information, obtain the model weight heatmap and model structure information for each of the at least one model; and

[0197] Based on the obtained model weight heatmap and model structure information of each of the at least one models, each model of that type of model in the at least one model is generated.

[0198] 19. The task-chain-oriented adaptive monitoring method as described in item 17, wherein the operation of generating the at least one model required to execute the current task based on model attribute information of the at least one model required to execute the current task, derived by inference, further includes, for each of the at least one model:

[0199] Each time a model of this type is generated, the count associated with that model is incremented by one;

[0200] Determine whether the count value reaches the number of models of this type;

[0201] If the count value is determined to have reached the number of models of that type, then stop generating that type of model;

[0202] Otherwise, continue generating this type of model.

[0203] 20. The adaptive monitoring method for task chains as described in item 17 further includes:

[0204] Deploy at least one model according to the deployment order and deployment location of each model.

[0205] 21. The task-chain-oriented adaptive monitoring method as described in item 16, wherein the model deployment location includes one of a sensor and a processor.

[0206] 22. The task-chain-oriented adaptive monitoring method as described in item 15 further includes:

[0207] At least one heatmap is obtained by running the deployed at least one model; and

[0208] The reasoning results for the current task are obtained based on at least one of the heatmaps.

[0209] 23. The task-chain-oriented adaptive monitoring method as described in item 22, wherein the operation of inferring model attribute information of at least one model required to execute the current task based at least on the text information of the current task further includes:

[0210] Based on the textual information of the current task and at least one heatmap obtained from the previous task, the model attribute information required to execute the current task is inferred.

[0211] 24. The adaptive monitoring method for task chains as described in item 16 further includes:

[0212] Compare the inference result of the current task with the threshold set for the current task;

[0213] If the inference result of the current task meets the threshold set for the current task, continue processing the next task in the task chain and clear the model executing the current task; and

[0214] If the inference result of the current task does not meet the threshold set for the current task, the next task in the task chain will not be processed.

[0215] 25. The task-chain-oriented adaptive monitoring method as described in item 24 further includes:

[0216] In response to the inference result of the current task not meeting the threshold set for the current task, the deployed at least one model is repeatedly run within a preset time period until the inference result of the current task meets the threshold set for the current task; and

[0217] If the inference result of the current task does not meet the threshold set for the current task within a preset time period, the deployment of at least one model will be stopped and the system will enter a dormant state.

[0218] 26. The adaptive monitoring method for task chains as described in item 15 further includes:

[0219] Before disassembling the task chain within the task chain file, receive the user's selection of the execution mode;

[0220] In response to the user-selected execution mode as low-power online mode, the task chain file is downloaded remotely; and

[0221] In response to the user's selection of offline execution mode, the task chain file is persisted to the processor.

[0222] 27. The adaptive monitoring method for task chains as described in item 15, wherein the text information of the current task includes at least one of the following: task role, task scenario, and task objective.

[0223] 28. The adaptive monitoring method for task chains as described in item 17, wherein the task objective includes multiple sub-task objectives and the dependencies between the multiple sub-task objectives.

[0224] 29. A non-transient computer-readable storage medium having a computer program stored thereon, the computer program performing the method as described in any one of items 15-29 when executed by one or more processors of a system.

[0225] 30. A computer program product comprising a computer program that, when executed by one or more processors, causes the method as described in any one of items 15-29 to be performed.

Claims

1. An electronic device, comprising: a processor configured to execute a computer program to: obtain a plurality of ordered tasks by disassembling a task chain within a task chain file; infer, based at least on text information of a current task, model attribute information of at least one model required to perform the current task; generate the at least one model required to perform the current task based on the inferred model attribute information of the at least one model required to perform the current task; deploy the generated at least one model; and run the deployed at least one model to obtain an inference result of the current task. The model attribute information of the at least one model required to perform the current task at least includes a threshold value set for the current task and a model index and a model deployment location for each of the at least one model.

2. The electronic device of claim 1, wherein, The model attribute information of the at least one model required to perform the current task further includes a model quantity and a model deployment order for each of the at least one model.

3. The electronic device of claim 2, wherein, The processor is further configured to execute the computer program to:

4. The electronic device of claim 2 or 3, wherein, obtain a model weight heat map and model structure information of each of the at least one model based on the model attribute information; and generate each model of the at least one model based on the obtained model weight heat map and model structure information of each of the at least one model. The processor is further configured to execute the computer program to:

5. The electronic device of claim 3, wherein, for each of the at least one model: after generating one model of the at least one model each time, increase a count value associated with the model of the at least one model by one; determine whether the count value reaches a model quantity of the model of the at least one model; if it is determined that the count value reaches the model quantity of the model of the at least one model, stop generating the model of the at least one model; and otherwise, continue to generate the model of the at least one model. The processor is further configured to execute the computer program to:

6. The electronic device of claim 3, wherein, deploy the at least one model according to a model deployment order of each model and a model deployment location of each model. The model deployment location includes one of a sensor and a processor.

7. The electronic device of claim 2, wherein, The processor is further configured to execute the computer program to:

8. The electronic device of claim 1, wherein, obtain at least one heat map by running the deployed at least one model; and obtain an inference result of the current task based at least on the at least one heat map. The processor is further configured to execute the computer program to:

9. The electronic device of claim 8, wherein, infer model attribute information required to perform the current task based on text information of the current task and at least one heat map obtained for a previous task of the current task. The processor is further configured to execute the computer program to:

10. The electronic device of claim 2, wherein, compare the inference result of the current task with a threshold value set for the current task; in response to the inference result of the current task satisfying the threshold value set for the current task, continue to process a next task in the task chain and clear the model performing the current task; and in response to the inference result of the current task not satisfying the threshold value set for the current task, not continue to process the next task in the task chain. ​

Images