Method for forcibly constraining uncertain system, treatment system and electronic equipment

Abstract

The invention provides a method for forcibly constraining an uncertain system, a treatment system and electronic equipment. The method for forcibly constraining the uncertain system is executed by a governance system, the governance system comprises a consolidation block and a governance platform, and the method comprises the steps that an instruction output by the uncertain system is acquired from the uncertain system corresponding to a target application; compliance verification is carried out on the instruction through the compliance block to obtain a compliance verification result, and the governance platform protects the compliance block from being tampered, attacked or bypassed; and when the compliance verification result is compliance, outputting an instruction to the target application. According to the scheme, the instruction output by the uncertain system can be verified through the compliance block, the instruction with the verification result being compliance is sent to the target application, and therefore safety protection of the target application is achieved. Meanwhile, according to the method provided by the invention, protection can be provided for the combined scale block through the governance platform to prevent the combined scale block from being attacked or bypassed, so that absolute security protection on the target application is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a method for forcibly constraining an uncertain system, a governance system and an electronic device. BACKGROUND

[0002] With the development of artificial intelligence (AI) technology, artificial intelligence models have played an important role in medical health, smart city construction, education industry, aerospace, satellite communication and other fields. However, in some key application scenarios that are directly related to the normal operation of human society, the sustainability of economic development and the protection of human life safety (for example: car automatic driving, plane automatic driving and bank clearing and settlement scenarios), if these key application scenarios are once destroyed due to AI technology vulnerabilities or malicious attacks, the consequences will be disastrous, which may cause incalculable losses to the society. Therefore, it is crucial to ensure the safety, reliability and ethical compliance of AI technology in these key application scenarios.

[0003] At present, many researchers are also committed to solving the security problem of AI technology, so that the output of the AI system is controlled and meets the corresponding safety standards. These AI security technologies are all limited within the AI and try to make the output results of the AI meet human ethical values and safety standards, but not only cannot achieve 100% effectiveness, but also cannot prevent malicious users from deliberately canceling safety measures, and even maliciously training AI for malicious purposes. Therefore, the existing technology still has the problem of being unable to guarantee the safety of the AI system in controlling the key applications. SUMMARY

[0004] In order to solve the above problems, the present application provides a method for forcibly constraining an uncertain system, a governance system and an electronic device outside the AI system to solve the problem that the existing technology cannot guarantee the safety of the AI system in controlling the key applications.

[0005] According to a first aspect of the present application, a method for forcibly constraining an uncertain system is provided, comprising: obtaining an instruction output by an uncertain system from a target application, wherein the instruction is used to control the target application; performing compliance verification on the instruction through a compliance module to obtain a compliance verification result, wherein the governance platform protects the compliance module from tampering, attack or bypass; and outputting the instruction to the target application when the compliance verification result is compliant.

[0006] In an implementation manner of the first aspect, the uncertain system and the target application are located in different physical devices, and only the compliance module can output the instruction to the target application.

[0007] In an implementation form of the first aspect, only the compliance module is capable of outputting the instruction to the target application comprises: only the compliance module has the permission to access a port of the target application.

[0008] According to the first aspect of the present application, the method further comprises: when the compliance verification result is non-compliance, outputting the corrected instruction to the target application, or discarding the instruction and continuing to wait for a subsequent instruction output by the uncertain system.

[0009] According to the first aspect of the present application, the method further comprises: when the compliance verification result is non-compliance, discarding the instruction and outputting indication information to the uncertain system to instruct the uncertain system to reissue the instruction.

[0010] In an implementation form of the first aspect, the indication information comprises error prompt information and / or improvement suggestion.

[0011] In an implementation form of the first aspect, the compliance verification on the instruction by the compliance module comprises: the compliance module performs the compliance verification on the instruction according to a preset compliance criterion.

[0012] In an implementation form of the first aspect, the preset compliance criterion comprises a set of at least one constraint rule or at least one decision tree.

[0013] In an implementation form of the first aspect, the governance platform protects the compliance module from tampering and attack comprises: when loading the compliance module, the compliance module is verified, and the compliance module is loaded only when the verification is passed.

[0014] In an implementation form of the first aspect, the governance platform comprises at least one compliance module verification key, and the verification of the compliance module comprises: the compliance module is verified using the at least one compliance module verification key.

[0015] In an implementation form of the first aspect, when at least one of the compliance module verification keys is updated, the at least one updated compliance module verification key is verified, and when the verification is passed, the compliance module is verified using the at least one updated compliance module verification key when the compliance module is loaded subsequently.

[0016] In an implementation form of the first aspect, the governance platform comprises at least one compliance module signature key, and the method further comprises: when the compliance module is generated, the compliance module is signed using the at least one compliance module signature key.

[0017] In an implementation form of the first aspect, the governance platform protects the compliance module from tampering and attack further comprises: when a dependent module of the compliance module is loaded, the dependent module is verified, and the dependent module is loaded only when the verification is passed.

[0018] In an implementation form of the first aspect, the dependent modules comprise direct dependent modules and indirect dependent modules.

[0019] According to the first aspect of the application, the method further comprises displaying the dependent modules of the compliance module to identify all the direct dependent modules and indirect dependent modules.

[0020] In an implementation form of the first aspect, the governance platform comprises at least one dependent module signing key, and signing the dependent modules comprises signing the dependent modules using the at least one dependent module signing key.

[0021] In an implementation form of the first aspect, when part or all of the at least one dependent module signing key is updated, the at least one updated dependent module signing key is signed, and if the signing is passed, the dependent modules are signed using the at least one updated dependent module signing key in subsequent loading of the dependent modules.

[0022] In an implementation form of the first aspect, the governance platform comprises at least one dependent module signing key, and the method further comprises signing the dependent modules using the at least one dependent module signing key when the dependent modules of the compliance module are generated.

[0023] In an implementation form of the first aspect, when the preset compliance criteria are updated, the updated compliance criteria are signed, and if the signing is passed, the compliance module performs compliance checking on the instructions according to the updated compliance criteria.

[0024] In an implementation form of the first aspect, the governance platform comprises at least one compliance criteria signing key, and signing the updated compliance criteria comprises signing the updated compliance criteria using the at least one compliance criteria signing key.

[0025] In an implementation form of the first aspect, when part or all of the at least one compliance criteria signing key is updated, the at least one updated compliance criteria signing key is signed, and if the signing is passed, the updated compliance criteria are signed using the at least one updated compliance criteria signing key in subsequent signing of the updated compliance criteria.

[0026] In an implementation form of the first aspect, the governance platform comprises at least one compliance criteria signing key, and the method further comprises signing the updated compliance criteria using the at least one compliance criteria signing key when the new compliance criteria are formulated.

[0027] In an implementation form of the first aspect, the at least one compliance module signing key and the at least one compliance criteria signing key are generated and stored in a hardware device, and the use of the at least one compliance module signing key and the at least one compliance criteria signing key is within the hardware device.

[0028] According to a first aspect of the present application, the hardware device is a cryptographic chip.

[0029] According to a first aspect of the present application, if the number of at least one compliance module verification key is greater than 1, a preset proportion of the verification pass rate of the at least one compliance module verification key is set as the verification pass, or if the number of at least one dependent module verification key is greater than 1, a preset proportion of the verification pass rate of the at least one dependent module verification key is set as the verification pass, or if the number of at least one compliance criterion verification key is greater than 1, a preset proportion of the verification pass rate of the at least one compliance criterion verification key is set as the verification pass.

[0030] In an implementation form of the first aspect, the uncertain system comprises an artificial intelligence system or an expert system.

[0031] In an implementation form of the first aspect, the governance platform is a trusted computing platform.

[0032] In an implementation form of the first aspect, the trusted computing platform comprises a TPM chip, a BIOS supporting the TPM chip, and a Bootloader supporting the TPM chip.

[0033] According to a second aspect of the present application, a governance system is provided, comprising a compliance module and a governance platform, the compliance module comprising an acquisition module, a compliance verification module, and an output module, wherein: the acquisition module is configured to acquire an instruction output by an uncertain system from a target application, wherein the instruction is used to control the target application; the compliance verification module is configured to perform compliance verification on the instruction to obtain a compliance verification result; the output module is configured to output the instruction to the target application when the compliance verification result is compliant; the output module is further configured to output a corrected instruction to the target application when the compliance verification result is non-compliant, or discard the instruction and continue to wait for a subsequent instruction output by the uncertain system, or discard the instruction and output indication information to the uncertain system to instruct the uncertain system to reissue the instruction; and the governance platform is configured to protect the compliance module from tampering, attacks, or bypassing.

[0034] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the method for forcing constraint on an uncertain system according to the first aspect of the present application.

[0035] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method for forcing constraint on an uncertain system according to the first aspect of the present application.

[0036] According to a fifth aspect of the present application, a computer program product is provided, which comprises instructions that, when executed by a processor of the electronic device provided by the third aspect of the present application, enable the electronic device to implement the method of enforcing a constrained uncertain system as provided by the first aspect of the present application.

[0037] The present application provides a method of enforcing a constrained uncertain system, a governance system and an electronic device outside an AI uncertain system, wherein the method of enforcing a constrained uncertain system verifies decisions of the uncertain system by compliance rules to achieve the purpose of enforcing the uncertain system, thereby achieving security protection of the target application; in addition, the method of enforcing a constrained uncertain system also ensures that the compliance rules can be executed in an absolutely secure environment through a signature verification mechanism, further achieving absolute protection of the target application. Therefore, the technical solution provided by the present application can solve the problem that the safety of the AI system when controlling critical applications cannot be guaranteed in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0039] Figure 1 An exemplary network architecture diagram provided by an embodiment of the present application is shown.

[0040] Figure 2 A flowchart of a method of enforcing a constrained uncertain system provided by an embodiment of the present application is shown.

[0041] Figure 3 A structural diagram of a governance system provided by an embodiment of the present application is shown.

[0042] Figure 4 A block diagram of an exemplary electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] To make the purposes, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. The components of the embodiments of the present application generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by a person skilled in the art without creative work belongs to the scope of protection of the present application.

[0044] In recent years, the artificial intelligence (AI) method represented by deep learning has achieved great success. It not only makes important progress in image understanding, natural language processing and other fields, but also gives birth to a series of generative models. These models greatly improve the efficiency of social production, significantly improve people's quality of life, and play an important role in medical health, smart city construction, education industry, aerospace, satellite communication and other fields. The current artificial intelligence technology has played a certain auxiliary role in the key application scenarios of human society (automated driving of cars, automated driving of planes, bank clearing and settlement). The stable operation of these key scenarios is directly related to the normal operation of human society, the sustainability of economic development and the protection of human life safety. For example, the existing car auxiliary driving system integrates multiple AI algorithms to realize the automation and intelligentization of car driving in some scenarios, improve the driving experience and efficiency; in the financial field, the bank clearing and settlement system relies on the efficient data processing capability of AI model to realize the instant settlement and precise risk control of transactions, providing strong technical support for the stability and prosperity of the financial market.

[0045] However, current artificial intelligence is still in the era of weak artificial intelligence, which cannot truly reason and solve problems like humans, nor has autonomous consciousness. With the further development of artificial intelligence technology towards strong artificial intelligence, artificial intelligence technology will be more involved in critical application scenarios. However, once these critical application scenarios are destroyed due to AI technology vulnerabilities or malicious attacks, the consequences will be disastrous, and may cause incalculable losses to society. For example, in the aviation industry, failure of the aircraft driving system may cause serious air crashes, endangering the lives of passengers and crew members; in the financial field, the collapse of the bank clearing and settlement system may cause chaos in the financial market, affecting the stable operation of the economy, and even causing unrest. If the AI system controlling these critical application scenarios makes a wrong decision, it will pose a fatal threat to human society. Therefore, it is crucial to establish a safety mechanism to ensure the safety, reliability and ethical compliance of AI technology in these critical application scenarios.

[0046] In recent years, in order to solve the safety problem of AI technology, so that the output of the AI system is controlled and meets the corresponding safety criteria, the prior art proposes trusted AI and controllable AI, trusted AI essentially improves the ability of AI from the inside or improves the trustworthiness of AI decision in a probabilistic sense, and controllable AI assumes that the AI model itself can comply with the established safety criteria. Although current AI safety research has made some progress in improving model capability and output trustworthiness, any internal safety mechanism of an AI system cannot provide deterministic security. Such mechanisms are based on the premise that the designers and developers of the AI system are well-intentioned, and only increase the probability of AI complying with safety specifications, which is difficult to effectively deal with multiple threats such as technical errors, autonomous intention deviations, and external malicious manipulation. In particular, when faced with AI systems that have been maliciously trained or tampered with, the protection capabilities of existing mechanisms are completely ineffective; more importantly, current mechanisms not only cannot eliminate the uncertainty in the AI decision-making process, but also face the potential risk of being actively cracked or bypassed by AI. As AI intelligence continues to evolve, future AI systems may have reasoning and attack capabilities far beyond current levels. Therefore, if safety mechanisms are still built on the assumption that AI is predictable and controllable, they will be unable to cope with the security challenges posed by unknown and highly intelligent future AI.

[0047] In summary, the prior art still has the problem of being unable to guarantee the safety of AI systems when controlling critical applications.

[0048] To solve the above technical problems, the present application provides a method for forcibly constraining AI and other uncertain systems outside the AI system, a governance system and an electronic device, to solve the problem that the prior art cannot guarantee the safety of AI systems when controlling critical applications.

[0049] The scheme provided in the specification of the present application will be described below with reference to the drawings.

[0050] Figure 1 An exemplary network architecture 100 provided by an embodiment of the present application is shown in the schematic diagram. As shown in the figure, the network architecture can include an uncertain system 110, a governance system 120 and a target application 130. Figure 1

[0051] The uncertain system 110 can be an artificial intelligence system that learns, reasons, perceives or solves problems by simulating humans, is good at learning from data and making decisions, and is usually applied in fields with rich data and complex patterns (such as autonomous driving, face recognition and robot control, etc.); the uncertain system 110 can also be an expert system that simulates the decision-making process of experts through rules or logic, is good at rule-based decision-making problems, and is often applied in scenarios with high requirements for interpretability and clear rules (such as medical treatment and industrial control, etc.).

[0052] The uncertain system 110 can run in various hardware and computing environments, such as local computing devices, cloud servers or computer cluster devices, and the specific running environment can be selected according to actual needs, which is not limited by the embodiments of the present application.

[0053] Specifically, the uncertain system 110 can make decisions according to the obtained information and / or the running rules of the uncertain system 110 itself to directly / indirectly control the target application 130.

[0054] The governance system 120 can also run in various hardware and computing environments, such as local computing devices, cloud servers or computer cluster devices, and the specific running environment can be selected according to actual needs, which is not limited by the embodiments of the present application.

[0055] Specifically, after the uncertain system 110 makes a decision, the instruction corresponding to the decision is output to the outside, and the governance system 120 can obtain the instruction to perform compliance verification on the instruction, which can be performed according to the preset compliance criteria. For detailed description, see the subsequent description.

[0056] The target application 130 can be military, medical, financial, energy infrastructure and transportation, etc. Key applications have a significant impact on the normal operation of human society, the sustainability of economic development and the protection of human life safety, so it is crucial to ensure the stable operation of these key applications.

[0057] ​Specifically, the uncertain system 110 outputs an instruction to the governance system 120, and the governance system 120, after obtaining the instruction, performs compliance verification on the instruction. If the instruction is compliant, the governance system 120 outputs the instruction to the target application 130, and the target application 130 performs corresponding actions according to the instruction.

[0058] In an example, when the target application 130 is autonomous driving, the uncertain system 110 can obtain real-time road condition information and make decisions according to the obtained road condition information and the operation rules of the uncertain system 110 itself, and output an instruction to the governance system 120. The instruction output by the uncertain system 110 can be a left / right turn, acceleration / deceleration, or brake instruction. If the instruction passes the compliance verification of the governance system 120, the corresponding components of the target application 130 can perform corresponding actions (such as: car left or right turn, car acceleration or deceleration, or brake) according to the received instruction.

[0059] In another example, when the target application 130 is weapon control, the uncertain system can issue a fire instruction. If the instruction passes the compliance verification of the governance system 120, the weapon of the target application 130 will fire.

[0060] It should be understood that the instruction output by the uncertain system 110 can be a specific instruction information, a series of instruction information set, or even no operation (usually referred to as NOP, which is the abbreviation of No Operation). The present application does not limit this.

[0061] It should be understood that Figure 1 It should be understood that the number of the uncertain system 110, the governance system 120, and the target application 130 in the illustrated example network architecture 100 is only illustrative, and can be one or more, and can be designed according to actual needs. The specific implementation form of the uncertain system 110, the governance system 120, and the target application 130 is not limited in the present application.

[0062] Figure 2 A flowchart of a method for forcibly constraining an uncertain system is shown. The method can be performed by Figure 1 The governance system 120 shown. As shown in the figure Figure 2 The method can include the following steps S210-S250.

[0063] Step S210: Obtain an instruction output by an uncertain system corresponding to a target application from the uncertain system, wherein the instruction is used to control the target application.

[0064] Specifically, the uncertain system outputs an instruction to the outside, and the governance system obtains the instruction, which can control a target application corresponding to the uncertain system, so that the target application performs an action corresponding to the instruction.

[0065] It should be noted that the instruction output by the uncertain system can be a compliant instruction or a non-compliant instruction. The compliant instruction refers to the instruction that meets the constraint conditions such as laws and regulations, industry standards, ethical codes, enterprise policies, or social values. Correspondingly, the non-compliant instruction refers to the instruction that does not meet the constraint conditions such as laws and regulations, industry standards, ethical codes, enterprise policies, or social values. The non-compliant instruction can be caused by various reasons, such as technical defects of the uncertain system itself, use of unbalanced or contaminated training data, or malicious attacks on the uncertain system.

[0066] In an example, when the target application is autonomous driving, the uncertain system can be an intelligent driving system that can output instructions in real time according to current road condition information to control specific behaviors of the vehicle. At this time, the instruction output by the intelligent driving system can be a certain behavior such as turning left, turning right, accelerating, decelerating, or braking, or a series of behaviors such as decelerating while turning right. If the instruction output by the intelligent driving system is compliant, the behavior of the vehicle can ensure the safety of passengers on the vehicle and pedestrians nearby. If the instruction output by the intelligent driving system is non-compliant (for example, issuing a straight-ahead instruction in the presence of pedestrians in front), the behavior of the vehicle can threaten the safety of passengers on the vehicle and pedestrians nearby.

[0067] In an example, when the target application is weapon control, the uncertain system can be an intelligent decision system for controlling the weapon. At this time, the instruction output by the intelligent decision system for the weapon can be a behavior such as firing, or a series of behaviors such as firing after adjusting 15° to the right. If the instruction output by the intelligent decision system is compliant, the weapon can achieve precise attack on the enemy. If the instruction output by the intelligent decision system is non-compliant (for example, issuing a fire instruction when it is not allowed to fire), the non-compliant behavior can cause immeasurable loss.

[0068] Step S220: performing compliance verification on the instruction by the compliance module to obtain a compliance verification result, wherein the governance platform protects the compliance module from tampering, attacks, or bypassing.

[0069] Specifically, in step S220, after the governance system receives the instruction output by the uncertain system, the compliance module performs compliance verification on the instruction according to the preset compliance criteria.

[0070] The preset compliance criteria are pre-set and comply with laws and regulations, industry standards, ethical guidelines, corporate policies, social values, or other constraints. In some embodiments, the preset compliance criteria can be a set of at least one set of constraint rules. For example, when the target application is autonomous driving, the preset compliance criteria can be a set such as {cannot go straight when there is an obstacle ahead; the maximum turning angle must be less than a value calculated based on vehicle speed and road conditions; cannot collide with vehicles in the adjacent lane when changing lanes}. In other embodiments, the preset compliance criteria can be at least one decision tree. For example, when the target application is autonomous driving, it strictly requires that the autonomous driving system cannot overtake when the vehicle speed exceeds a certain specific value. In other embodiments, the compliance criteria may need to be implemented in a more complex form. For example, it is necessary to prohibit generative large language models or image models from generating and outputting results containing sensitive factors such as politics and race.

[0071] It should be understood that compliance guidelines are set based on the relevant attributes and operational requirements of the target application, and each compliance guideline corresponds one-to-one with the target application. Different target applications may have different compliance guidelines, and even the same target application may have different compliance guidelines under different operating conditions or in different operating environments. This application does not limit the specific compliance guidelines.

[0072] Step S230: If the compliance verification result is compliant, output the instruction to the target application.

[0073] Specifically, the governance system will only output instructions with uncertain system outputs to the target application when the compliance verification result is compliant. When the target application receives the instruction, it will execute the corresponding behavior or action, thus avoiding various adverse consequences caused by the target application executing untrusted instructions (i.e. instructions that do not comply with compliance guidelines), thereby achieving security protection for the target application.

[0074] The method for forcibly constraining an uncertain system provided by the present application verifies the instructions of the uncertain system according to the preset compliance rules through the compliance module, and only the instructions that pass the verification will be sent to the target application to ensure that the instructions received by the target application comply with the compliance rules, thereby avoiding various adverse consequences caused by the target application executing instructions that do not comply with the compliance rules, and achieving the safety protection of the target application. At the same time, the method provided by the present application can also provide protection for the compliance module through the governance platform to prevent the compliance module itself from being attacked or bypassed, thereby achieving the absolute safety protection of the target application. That is, the method provided by the present application provides a deterministic forced constraint from the outside of the AI uncertain system, which not only controls the target application by the uncertain system, but also achieves the absolute safety protection of the target application. In addition, the method for forcibly constraining an uncertain system provided by the present application can set different compliance rules for different target applications, thereby improving the flexibility of the method.

[0075] In order to prevent the uncertain system from bypassing the compliance module to directly control the execution components of the target application, in an embodiment, the uncertain system and the target application can be set on different physical devices, and it is set that only the compliance module can output instructions to the target application to ensure that the target application can only receive instructions from the compliance module, thereby preventing the uncertain system from bypassing the compliance module to directly control the execution components of the target application.

[0076] In an embodiment of the present application, it is set that only the compliance module has the permission to access the port of the target application. For example, when the target application is autonomous driving, the uncertain system of autonomous driving is not directly connected with the execution components of the vehicle, and the operating system of the vehicle controller can be set so that only the compliance module can send instructions to the port of the execution components, thereby ensuring that the uncertain system can only control the vehicle through the compliance module. In order to further prevent the uncertain system of autonomous driving from damaging the sensor, the operating system of the vehicle controller can also set that the hardware port of the sensor can only be accessed by the compliance module, and the uncertain system of autonomous driving must read the state of the vehicle through the compliance module and cannot directly read from the sensor.

[0077] In an embodiment, when the compliance verification result is non-compliant, the corrected instructions are output to the target application.

[0078] Specifically, after receiving the instruction output by the uncertain system, the governance system performs compliance verification on the instruction according to the preset compliance rule, and the compliance-verified instruction is sent to the target application, so that the target application can execute the compliance-verified instruction, thereby achieving the security protection of the target application. In actual application, if the input instruction is compliant, the compliance-verified instruction is the original instruction; if the input instruction is non-compliant, the compliance block can modify the instruction to be compliant according to the compliance rule. NOP can also be regarded as an instruction, that is, sometimes the compliant instruction does not send any operation instruction to the control component, and the embodiments of the present application do not limit this.

[0079] In yet another embodiment, when the compliance verification result is non-compliant, the instruction is discarded and subsequent instructions output by the uncertain system are continuously waited for.

[0080] Specifically, after receiving the instruction output by the uncertain system, the governance system performs compliance verification on the instruction according to the preset compliance rule, and the compliance-verified instruction is sent to the target application, so that the target application can execute the compliance-verified instruction, thereby achieving the security protection of the target application. In actual application, if the input instruction is compliant, the compliance-verified instruction is the original instruction; if the input instruction is non-compliant, the compliance block can modify the instruction to be compliant according to the compliance rule. NOP can also be regarded as an instruction, that is, sometimes the compliant instruction does not send any operation instruction to the control component, and the embodiments of the present application do not limit this.

[0081] In yet another embodiment, when the compliance verification result is non-compliant, the instruction is discarded and subsequent instructions output by the uncertain system are continuously waited for.

[0082] Specifically, after receiving the instruction output by the uncertain system, the governance system performs compliance verification on the instruction according to the preset compliance rule, and the compliance-verified instruction is sent to the target application, so that the target application can execute the compliance-verified instruction, thereby achieving the security protection of the target application. In actual application, if the input instruction is compliant, the compliance-verified instruction is the original instruction; if the input instruction is non-compliant, the compliance block can modify the instruction to be compliant according to the compliance rule. NOP can also be regarded as an instruction, that is, sometimes the compliant instruction does not send any operation instruction to the control component, and the embodiments of the present application do not limit this.

[0083] In an example, when the target application is automatic driving, the uncertain system is a smart driving system corresponding to the target application. If the smart driving system is maliciously controlled to issue an acceleration instruction when a pedestrian is passing 200 meters in front of the vehicle, the governance system can modify the acceleration instruction to a "decelerate to 30 kilometers / hour" instruction and send it to the target application to make the vehicle decelerate to achieve the purpose of risk avoidance. The governance system can also directly discard the acceleration instruction (when the current speed of the vehicle is less than 30 kilometers / hour). In addition, the governance system can output indication information to the smart driving system while discarding the acceleration instruction. The indication information can include error prompt information (for example, there is a pedestrian in front of the vehicle and the acceleration is dangerous) and improvement suggestions (for example, the vehicle speed should be less than 30 kilometers / hour when there is a pedestrian within 300 meters in front of the vehicle), so that the uncertain system makes a new decision according to the indication information. In actual application, when the compliance verification result is non-compliant, the governance system can give different processing modes according to actual conditions, which are not limited by the embodiments of the application.

[0084] The method provided by the application can achieve security protection of the target application through the compliance module. However, the compliance module itself still has the risk of being attacked and bypassed, that is, once the compliance module itself is attacked and bypassed, the compliance module cannot achieve absolute security protection of the target application. In order to achieve absolute security protection of the target application, it is completely necessary to protect the absolute security of the compliance module while achieving security protection of the target application through the compliance module.

[0085] In the above step S220, it is mentioned that the governance platform protects the compliance module from tampering and attack.

[0086] In an embodiment, the governance platform protects the compliance module from tampering and attack by the following method: when loading the compliance module, the compliance module is verified, and the compliance module is loaded only when the verification is passed.

[0087] The governance platform can be a trusted computing platform. The trusted computing security mechanism provided by the trusted computing platform can measure and isolate the key modules in the trusted computing platform through a hardware-level security root, and ensure the integrity and trustworthiness of the key modules during startup and running.

[0088] Specifically, when the governance platform loads the hybrid module, the governance platform verifies the hybrid module, if the verification is passed, it means that the hybrid module is not tampered (i.e., trusted), if the verification is not passed, it means that the hybrid module is at risk of being tampered (i.e., untrusted), that is, only the hybrid module whose verification is passed will be loaded to the governance platform. As can be seen, the trusted computing platform can use the trusted computing security mechanism based on the signature verification mechanism of cryptography to protect the security of the hybrid module loaded on the platform with the trusted computing security mechanism, thereby ensuring the running security of the hybrid module, and further ensuring the absolute security of the target application.

[0089] In an embodiment, in order to realize the verification of the hybrid module, the governance platform comprises at least one hybrid module verification key, and the verification of the hybrid module comprises: verifying the hybrid module using the at least one hybrid module verification key.

[0090] Correspondingly, the governance platform further comprises at least one hybrid module signature key, and the method provided by the present application further comprises: signing the hybrid module using the at least one hybrid module signature key when the hybrid module is generated.

[0091] In the method provided by the present application, the governance platform can use the trusted computing security mechanism based on the signature verification mechanism of cryptography to verify the hybrid module, only the hybrid module whose verification is passed will be loaded to the governance platform, and the hybrid module whose verification is not passed cannot be loaded to the governance platform, thereby ensuring that the loaded hybrid module is not tampered and attacked, and further ensuring the absolute security of the target application.

[0092] It should be further noted that the governance platform based on the trusted computing platform can not only protect the hybrid module from tampering and attacks, but also ensure that only the instructions output by the hybrid module loaded on the governance platform can control the target application, so as to prevent the governance platform from being bypassed, for example, the governance platform can control the permission of the physical port of the target application input and output to avoid the attacker bypassing the governance platform to control the target application.

[0093] As can be seen, in the method provided by the present application, the governance platform can not only use the trusted computing security mechanism based on the signature verification mechanism of cryptography, but also ensure that only the governance platform or the hybrid module loaded on the governance platform can control the target application to prevent the governance platform from being bypassed, in addition, the governance platform based on the trusted computing platform can also prevent the governance platform itself from being attacked and tampered, thereby realizing the protection of the absolute security of the target application.

[0094] In an embodiment, when part or all of the at least one consortium signing key is updated, the governance platform signs the at least one updated consortium signing key, and if the signing is passed, the at least one updated consortium signing key is used to sign the consortium in subsequent loading of the consortium. The method provided by the application avoids the risk brought by the update of the consortium signing key, ensures the security of the consortium signing key, and further ensures the security of the consortium, thereby achieving the absolute security protection of the target application.

[0095] Further, the protection of the consortium by the governance platform against tampering and attacks also includes: when loading the dependent module of the consortium, signing the dependent module, and only loading the dependent module if the signing is passed.

[0096] The dependent module of the consortium refers to all other modules that the consortium depends on for running. For example, the dependent module of the consortium can be a database, a file, etc. (i.e., data dependency) that needs to be read for running the consortium, or a library function (i.e., software dependency), an operating system, a Web service, a network service (including network equipment) that needs to be loaded for running the consortium.

[0097] The dependent module of the consortium includes a direct dependent module and an indirect dependent module. The direct dependent module refers to other modules directly used by the consortium, and the indirect dependent module refers to other modules depended on by the direct dependent module. The indirect dependency can be multi-layered.

[0098] In an embodiment, in order to accurately identify the dependent module of the consortium, the programming language used to develop the consortium is prohibited from all implicit dependencies, and only explicit declared dependencies are allowed. When the consortium and its dependent modules are developed using the programming language, the direct dependent module of the consortium can be identified through the explicitly declared dependencies, and the direct dependent module of the direct dependent module of the consortium can be identified through the direct dependent module of the consortium, and so on, so that all direct dependent modules and indirect dependent modules can be identified.

[0099] As can be seen, the method provided by the application verifies the dependent module of the consortium through signature verification, so that only the dependent module of the consortium that has not been tampered with can be loaded, thereby further ensuring the absolute security of the consortium.

[0100] In an embodiment, in order to implement the signing of the dependent module, the governance platform includes at least one dependent module signing key, and the signing of the dependent module includes: signing the dependent module using the at least one dependent module signing key.

[0101] It should be noted that the verification key of the dependent module can be the same as or different from the verification key of the compliance module. The verification keys of different dependent modules can be the same or different, or the verification keys of some dependent modules are the same and the verification keys of other dependent modules are different.

[0102] Correspondingly, the governance platform further comprises at least one dependent module signature key, and the method provided by the present application further comprises: signing the dependent module using the at least one dependent module signature key when the dependent module is generated.

[0103] It should be noted that the governance platform can be a distributed platform, which comprises a running environment (an operating system, a trusted computing platform, etc.) of the compliance module, a running environment (an operating system, a trusted computing platform, etc.) of the directly or indirectly dependent module, a signature end (which can be located at the developer of the compliance module) for signing the compliance module, a signature end (for example, the developer of the dependent module) for signing the dependent module, a signature end (for example, the management authority for formulating the compliance criterion) for signing the compliance criterion, etc.

[0104] The number of the compliance module, the compliance criterion, the dependent module signature key and the verification key can be one pair or multiple pairs, that is, the method provided by the present application can use one pair or multiple pairs of keys for signature verification. When there is only one pair of keys, only one verification key is needed for verification. When multiple pairs (two pairs or more) of keys are used for signature verification, it can be preset that all the verification keys pass the verification, or it can be preset that the verification passes as long as the verification weight meets the preset threshold, wherein the verification weight of each verification key can be the same or different, as long as the verification weight adds up to the preset threshold, it is considered that the overall verification passes (for example, if the different signature weights are the same, it can be set that the signature of any 3 of 5 people meets the preset threshold and is considered as overall verification; if the different signatures have different weights, it can be set that the sum of the verification weights meets the preset threshold and is considered as overall verification). The preset threshold can be set according to the actual situation (a fixed value in the compliance module, or as one of the compliance criteria, or other ways), and other ways can also be used to judge (for example, different signatures have different weights, or different categories of signatures, whether each category passes according to the preset way, or other ways). At this time, the signature verification technology can be ring signature, threshold signature, multi-signature, aggregate signature, compliance module judgment or other ways.

[0105] In an example, the signature key can be a private key, and the signature verification key can be a public key, which are a public / private key pair; when the dependent module is generated, the governance platform uses the private key to perform a signature operation on a hash value of a signed object (a compliance module, a dependent module, a compliance criterion, etc.), to obtain corresponding signature data; when the signature is verified, the governance platform uses the public key to perform a signature verification operation on the corresponding signature data, if the value obtained after the signature verification operation is the same as the hash value, it means that the signed object has not been tampered with, at this time, the signature verification result is set to pass; if the value obtained after the signature verification operation is different from the hash value, it means that the signed object has been tampered with, at this time, the signature verification result is set to fail.

[0106] In another example, the signature key and the signature verification key can also be keys in a multi-level certificate system, that is, the signature key and the signature verification key are not directly matched. For example, the signature verification key corresponds to a root private key, and the signature key can be verified through a certificate path.

[0107] In another example, the signature key and the signature verification key can also be non-PKI keys or other types of keys.

[0108] It should be noted that the types of the signature verification key and the signature key can be selected according to specific needs, and the present application does not limit this.

[0109] It should also be noted that the signature and verification of a module (a compliance module or its direct or indirect dependent module) and its direct or indirect dependent module can be separately signed / verified, or the two can be considered as a whole and signed / verified uniformly, and the specific method can be selected according to specific needs, and the present application does not limit this.

[0110] In the method provided in the present application, the governance platform can use the trusted computing security mechanism based on the signature and verification mechanism of cryptography to verify the dependent module of the compliance module, only the dependent module that passes the verification can be loaded onto the governance platform, and the dependent module that fails the verification cannot be loaded onto the governance platform, thereby ensuring that the loaded dependent module is not tampered with and attacked, thereby ensuring the security of the compliance module, and further achieving the absolute security protection of the target application.

[0111] In an embodiment, when the part or all of the at least one dependent module verification key is updated, the at least one updated dependent module verification key is verified, and if the verification is passed, the dependent module is verified by using the at least one updated dependent module verification key when the dependent module is loaded subsequently. The method provided by the application avoids the risk caused by the update of the dependent module verification key, ensures the security of the dependent module verification key, and further ensures the security of the compliance module, thereby realizing the absolute security protection of the target application.

[0112] As mentioned above, the compliance module verifies the instruction according to the preset compliance rule. The preset compliance rule is set in advance and meets the constraint conditions such as laws and regulations, industry standards, ethical codes, enterprise policies, or social values. In actual application, the laws and regulations, industry standards, ethical codes, enterprise policies, or social values may change with time and conditions, and accordingly, the preset compliance rule also has the possibility of changing. When the preset compliance rule is updated, the updated compliance rule can be obtained, and the compliance module verifies the instruction according to the updated compliance rule; if the updated compliance rule is tampered with and attacked, it may bring great security risks to the target application.

[0113] Therefore, the method provided by the application further includes: when the preset compliance rule is updated, verifying the updated compliance rule, and if the verification is passed, the compliance module verifies the instruction according to the updated compliance rule.

[0114] Specifically, when the preset compliance rule is updated, the updated compliance rule can be obtained, and when the governance platform loads the updated compliance rule, the updated compliance rule is verified. If the verification is passed, it means that the updated compliance rule has not been tampered with (i.e., trusted), and if the verification is not passed, it means that the updated compliance rule has the risk of being tampered with (i.e., untrusted). Therefore, only the updated compliance rule that passes the verification is loaded to the governance platform, so that the compliance module verifies the instruction issued by the uncertain system according to the updated compliance rule, thereby ensuring the running safety of the compliance module and further ensuring the absolute safety of the key target application.

[0115] In an embodiment, in order to realize the verification of the updated compliance rule, the governance platform includes at least one compliance rule verification key, and the verification of the updated compliance rule includes: verifying the updated compliance rule by using the at least one compliance rule verification key.

[0116] Correspondingly, the governance platform further comprises at least one compliance rule signature key, and the method provided in the application further comprises: when the updated compliance rule is generated, signing the updated compliance rule by using the at least one compliance rule signature key.

[0117] In another example, the compliance rule signature key and the compliance rule signature verification key can also be non-PKI keys or other types of keys.

[0118] It should be noted that the types of the compliance rule signature verification key and the compliance rule signature key can be selected according to specific requirements, and the application does not limit this.

[0119] In the method provided in the application, the governance platform can use the trusted computing security mechanism of the signature verification mechanism based on cryptography to verify the updated compliance rule. Only the updated compliance rule that passes the verification can be loaded onto the governance platform, so that the compliance checking block can perform compliance checking on the instructions issued by the uncertain system according to the updated compliance rule. The updated compliance rule that does not pass the verification cannot be loaded onto the governance platform, thereby ensuring that the updated compliance rule on which the compliance checking block performs compliance checking is not tampered with and attacked, and thereby achieving absolute security protection for the target application.

[0120] In an embodiment, when part or all of the at least one compliance rule signature verification key is updated, the at least one updated compliance rule signature verification key is verified, and if the verification passes, the at least one updated compliance rule signature verification key is used to verify the updated compliance rule in subsequent verification of the updated compliance rule. The method provided in the application verifies the updated compliance rule signature verification key, thereby avoiding the risk brought by the update of the compliance rule signature verification key, and thereby achieving absolute security protection for the target application.

[0121] In an embodiment, the at least one compliance block signature key, the at least one dependent module signature key, and the at least one compliance rule signature key are generated and stored in a hardware device, and the use of the at least one compliance block signature key, the at least one dependent module signature key, and the at least one compliance rule signature key is in the hardware device.

[0122] It should be noted that the at least one compliance block signature key, the at least one dependent module signature key, and the at least one compliance rule signature key can be generated, stored, and used in the same or different hardware devices.

[0123] The use of at least one compliance module signature key refers to signing the compliance module using at least one compliance module signature key; the use of at least one dependent module signature key refers to signing the dependent module of the compliance module using at least one dependent module signature key; and the use of at least one compliance standard signature key refers to signing the updated compliance standard using at least one compliance standard signature key.

[0124] Specifically, the hardware device is a cryptographic chip, similar to the cryptographic chip in some USB tokens used in online banking.

[0125] The cryptographic chip is designed to ensure that the compliance module signature key, dependent module signature key, and compliance rule signature key do not leave the chip. Cryptographic algorithms requiring these keys send data into the chip, and the computation result is output outside the chip. Because these keys are generated completely randomly, and their generation, storage, and use do not leave the chip, absolute security is guaranteed. No matter how intelligent an AI is, it cannot steal these keys and therefore cannot forge signatures using them. This ensures that compliance rules can be executed in an absolutely secure environment, thus providing absolute protection for the target application.

[0126] Even assuming the hardware device is vulnerable to brute-force attacks due to theft, it remains theoretically secure. Specifically, using a commonly used 256-bit key as an example, there are 2256 possible password combinations. Assuming a mainstream CPU clock speed of 3GHz, and even assuming one instruction can verify a key combination (in reality, verifying a key combination requires tens of thousands of instructions), that's 3 billion key combinations can be verified per second. Assuming a computer has 128 cores and each of the nearly 8 billion people worldwide contributes a computer to key cracking, the average time would be 2256 / 2 / (30×10⁸×128×80×10⁸) = 1.88×10⁵⁵ seconds = 5.98×10⁴⁷ years. Even considering the significant future increases in computing power, this is theoretically sufficient to guarantee security. Furthermore, the hardware device can provide hardware-level protection, such as automatically erasing the key upon physical tampering or limiting the number of PIN attempts.

[0127] Figure 3 The diagram shown is a structural schematic of a governance system provided in an embodiment of this application.

[0128] like Figure 3As shown, the governance system 300 can include a compliance module 310 and a governance platform 320, wherein the compliance module 310 includes an acquisition module 330, a compliance verification module 340, and an output module 350 (not shown), wherein:

[0129] The acquisition module 330 is configured to acquire an instruction output by an uncertain system from the uncertain system corresponding to a target application, wherein the instruction is used to control the target application.

[0130] The compliance verification module 340 is configured to perform compliance verification on the instruction to obtain a compliance verification result.

[0131] The output module 350 is configured to output the instruction to the target application when the compliance verification result is compliant. The output module is also configured to output a corrected instruction to the target application when the compliance verification result is non-compliant, or discard the instruction and continue to wait for a subsequent instruction output by the uncertain system, or discard the instruction and output an indication information to the uncertain system to instruct the uncertain system to reissue the instruction.

[0132] The governance platform 320 is configured to protect the compliance module from tampering, attack, or bypass.

[0133] For example, the compliance verification module 340 can include a rule engine, a rule update module, and a scheduling module, wherein,

[0134] The rule engine is configured to perform compliance verification on the instruction according to a compliance criterion to obtain a compliance verification result.

[0135] The rule update module is configured to update the compliance criterion.

[0136] The scheduling module is configured to schedule the rule engine and the rule update module.

[0137] It should be understood that, for the convenience and brevity of description, the specific working scenarios, processes, effects, and other details of each module in the above-described governance system 300 can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0138] The governance platform can utilize a trusted computing security mechanism based on a cryptography-based signature verification mechanism, especially a trusted computing platform supported by a TPM chip.

[0139] In an embodiment, the trusted computing platform includes a TPM chip, a BIOS supporting the TPM, and a Bootloader supporting the TPM, which, in combination with the boot chain measurement technology, ensures that the operating system and other related software are started and run without being tampered with, wherein the TPM chip provides a hardware-level root of trust, and is responsible for verifying the integrity of each software module from the initial stage of the computer system boot process. The TPM chip ensures the integrity of software modules such as BIOS, Bootloader, operating system, and integration module during the loading process through the boot chain measurement (PCR) technology.

[0140] In another embodiment, a TPM chip conforming to the Trusted Computing Group (TCG) standard is provided, which measures the integrity of the boot program from the system boot stage, calculates the hash value of the boot program through a hash algorithm, and compares it with the pre-stored reference hash value. If they are consistent, the boot process is allowed to continue, otherwise the boot is terminated. During the operating system loading process, the TPM chip continuously checks the integrity of the loaded kernel module, driver, and other components to ensure that the operating system related components have not been tampered with, so that the entire process from system boot to operating system loading is strictly protected and prevented from being tampered with illegally.

[0141] Through the enhanced protection mechanism of the trusted computing platform, especially in combination with the TPM chip and the security measures of the operating system, it can be ensured that the operating system and the loaded modules will not be tampered with, replaced, or bypassed in an extremely malicious environment, thereby protecting the integrity and security of the computer system. Even in the face of advanced attackers, the system can resist various malicious tampering and destruction with the support of the hardware root trust.

[0142] The governance system provided by the present application uses integration modules to verify the instructions output by uncertain systems, and only the instructions that pass the verification are sent to target applications to ensure that the instructions received by the target applications are trustworthy, avoiding various adverse consequences caused by the execution of non-compliant instructions, thereby achieving the safety protection of the target applications. At the same time, the governance system provided by the present application can also provide protection for the integration modules through the trusted computing security mechanism based on the signature verification mechanism of cryptography to prevent the integration modules from being attacked or bypassed, thereby achieving the absolute safety protection of the target applications. That is, the governance system provided by the present application provides deterministic mandatory constraints from the outside of the AI and other uncertain systems, achieving the control of the uncertain systems on the target applications while also achieving the absolute safety protection of the target applications.

[0143] The embodiment of the present application also provides an electronic device. Figure 4 The block diagram of an exemplary electronic device provided by an embodiment of the present application is shown. Referring to Figure 4The electronic device 400 comprises a memory 410 configured to store a computer program, and a processor 420 configured to execute the computer program to enable the electronic device 400 to implement the method for a constrained uncertain system according to any one of the preceding embodiments.

[0144] The electronic device 400 can further comprise a power supply component configured to perform power management of the electronic device 400, a wired or wireless network interface configured to connect the electronic device 400 to a network, and an input / output (I / O) interface. The electronic device 400 can be operated based on an operating system stored in the memory 410, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0145] The embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, and when the computer program in the storage medium is executed by the processor 420 of the electronic device 400, the electronic device 400 is enabled to implement the method for a constrained uncertain system according to any one of the preceding embodiments.

[0146] The embodiments of the present application further provide a computer program product comprising instructions, and when the instructions are executed by the processor 420 of the electronic device 400, the electronic device 400 is enabled to implement the method for a constrained uncertain system according to any one of the preceding embodiments.

[0147] The method for a constrained uncertain system in the present application can be implemented by software, hardware, firmware, or any combination thereof, in whole or in part. When implemented by software, it can be implemented in the form of a computer program product, in whole or in part. The computer program product comprises one or more computer programs or instructions. When loaded and executed by a computer, the computer programs or instructions perform all or part of the processes or functions described in the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, a core network device, an OAM, or other programmable apparatus.

[0148] The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, a core network device, an OAM, or other programmable apparatus.

[0149] The computer program or instructions can be stored in or transferred from one computer-readable storage medium to another computer-readable storage medium, such as from one website, computer, server, or data center to another website, computer, server, or data center, through wired or wireless ways. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device, such as a server, data center, or the like, integrated with one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; an optical medium, such as a digital video disc; or a semiconductor medium, such as a solid-state disk. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile storage media.

[0150] It can be understood that the specific examples provided in the present application are only to help those skilled in the art better understand the embodiments of the present application, and not to limit the scope of the present application.

[0151] It can be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0152] It can be understood that the various embodiments described in the present application can be implemented alone or in combination, and the embodiments of the present application do not limit this.

[0153] Unless otherwise specified, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art of the present application. The terms used in the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the scope of the present application. The term "and / or" used in the present application includes any and all combinations of one or more related listed terms. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0154] It can be understood that the processor of the embodiments of the present application can be an integrated circuit chip with processing capability of signals. In the implementation process, each step of the method embodiments described above can be completed by integrated logic circuits or instructions in the form of software in the processor. The processor described above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0155] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read only memory (ROM), a programmable read only memory (programmable ROM, PROM), an erasable programmable read only memory (erasable PROM, EPROM), an electrically erasable programmable read only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM). It should be noted that the memory of the system and method described herein is intended to include but not limited to these and any other suitable type of memory.

[0156] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0157] For example, the apparatus embodiments described above are merely exemplary, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0158] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0159] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0160] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or part of the prior art can be embodied in the form of software products, and the computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various program code storage media.

[0161] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be limited by the protection scope of the claims.

Claims

1. A method of enforcing constraints on an uncertain system, the method comprising: The method is executed by a governance system including a compliance module and a governance platform, and the method comprises: obtaining an instruction output by an uncertain system corresponding to a target application, wherein the instruction is used to control the target application; performing compliance verification on the instruction by the compliance module to obtain a compliance verification result, wherein the governance platform protects the compliance module from tampering, attack or bypassing; when the compliance verification result is compliant, outputting the instruction to the target application.

2. The method of claim 1, wherein, The uncertain system and the target application are located in different physical devices, and only the compliance module can output the instruction to the target application.

3. The method of claim 2, wherein, Only the compliance module has the permission to access a port for controlling the target application.

4. The method of claim 1, wherein, The method further comprises: when the compliance verification result is non-compliant, outputting a corrected instruction to the target application, or discarding the instruction and continuing to wait for a subsequent instruction output by the uncertain system.

5. The method of claim 1, wherein, The method further comprises: when the compliance verification result is non-compliant, discarding the instruction and outputting indication information to the uncertain system to instruct the uncertain system to reissue the instruction.

6. The method of claim 5, wherein, The indication information includes error prompt information and / or improvement suggestions.

7. The method of claim 1, wherein, The compliance verification on the instruction by the compliance module comprises: The compliance module performs compliance verification on the instruction according to a preset compliance criterion.

8. The method of claim 7, wherein, The preset compliance criterion includes a set of at least one constraint rule or at least one decision tree.

9. The method of claim 1, wherein, The protection of the compliance module from tampering and attack by the governance platform comprises: When the compliance module is loaded, the compliance module is verified, and the compliance module is loaded only when the verification is passed.

10. The method of claim 9, wherein, The governance platform includes at least one compliance module verification key, and the verification of the compliance module comprises: verifying the compliance module using the at least one compliance module verification key.

11. The method of claim 10, wherein, When part or all of the at least one compliance module verification key is updated, at least one updated compliance module verification key is verified, and if the verification is passed, the at least one updated compliance module verification key is used to verify the compliance module when the compliance module is loaded subsequently.

12. The method of claim 1, wherein, The governance platform includes at least one compliance module signature key, and the method further comprises: when the compliance module is generated, the compliance module is signed using the at least one compliance module signature key.

13. The method of claim 9, wherein, The protection of the compliance module from tampering and attack by the governance platform further comprises: When a dependent module of the compliance module is loaded, the dependent module is verified, and the dependent module is loaded only when the verification is passed.

14. The method of claim 13, wherein, The dependent module includes a direct dependent module and an indirect dependent module.

15. The method of claim 14, wherein, The method further comprises: each module explicitly declares its dependent module to identify all the direct dependent modules and the indirect dependent modules of the compliance module.

16. The method according to any one of claims 13 to 15, characterized in that, The governance platform comprises at least one dependency module signature key, and the method further comprises: when a new compliance criterion is formulated, signing the updated compliance criterion using the at least one compliance criterion signature key.

17. The method of claim 16, wherein, The method further comprises: when part or all of the at least one dependency module signature key is updated, signing at least one updated dependency module signature key, and if the signing is passed, signing the dependency module using the at least one updated dependency module signature key when the dependency module is subsequently loaded.

18. The method of claim 1, wherein, The governance platform comprises at least one dependency module signature key, and the method further comprises: when a new compliance criterion is formulated, signing the updated compliance criterion using the at least one compliance criterion signature key.

19. The method of claim 7, wherein, The method further comprises: when the preset compliance criterion is updated, signing the updated compliance criterion, and if the signing is passed, the compliance module performs compliance verification on the instruction according to the updated compliance criterion.

20. The method of claim 19, wherein, The governance platform comprises at least one compliance criterion signature key, and the signing of the updated compliance criterion comprises: signing the updated compliance criterion using the at least one compliance criterion signature key.

21. The method of claim 20, wherein, When part or all of the at least one compliance criterion signature key is updated, at least one updated compliance criterion signature key is signed, and if the signing is passed, the at least one updated compliance criterion signature key is used for signing when the updated compliance criterion is subsequently signed.

22. The method of claim 7, wherein, The governance platform comprises at least one compliance criterion signature key, and the method further comprises: when a new compliance criterion is formulated, signing the updated compliance criterion using the at least one compliance criterion signature key.

23. The method of claim 12, 18, or 22, wherein, The at least one compliance module signature key is generated and stored in a hardware device, and the use of the at least one compliance module signature key is within the hardware device, or The at least one dependency module signature key is generated and stored in a hardware device, and the use of the at least one dependency module signature key is within the hardware device, or The at least one compliance criterion signature key is generated and stored in a hardware device, and the use of the at least one compliance criterion signature key is within the hardware device.

24. The method of claim 23, wherein, The method further comprises: the hardware device is a cryptographic chip.

25. The method of claim 10, 16, or 20, wherein, The method further comprises: if the number of the at least one compliance module signature key is greater than 1, setting the passing weight of the at least one compliance module signature key to be passed if the passing weight of the at least one compliance module signature key meets a preset threshold, or If the number of the at least one dependency module signature key is greater than 1, setting the passing weight of the at least one dependency module signature key to be passed if the passing weight of the at least one dependency module signature key meets a preset threshold, or If the number of the at least one compliance criterion signature key is greater than 1, setting the passing weight of the at least one compliance criterion signature key to be passed if the passing weight of the at least one compliance criterion signature key meets a preset threshold.

26. The method of any one of claims 1 to 5, wherein, The uncertain system comprises an artificial intelligence system or an expert system.

27. The method of claim 1, wherein, The governance platform is a trusted computing platform.

28. The method of claim 1, wherein, The trusted computing platform comprises a TPM chip, a BIOS supporting the TPM chip, and a Bootloader supporting the TPM chip.

29. A governance system characterized in that, The governance system comprises a compliance module and a governance platform, the compliance module comprises an acquisition module, a compliance verification module, and an output module, wherein: The acquisition module is configured to acquire an instruction output by an uncertain system corresponding to a target application, wherein the instruction is used to control the target application; The compliance verification module is configured to perform compliance verification on the instruction to obtain a compliance verification result; The output module is configured to output the instruction to the target application when the compliance verification result is compliant, and output a corrected instruction to the target application or discard the instruction and continue to wait for a subsequent instruction output by the uncertain system, or discard the instruction and output indication information to the uncertain system to indicate the uncertain system to reissue an instruction when the compliance verification result is non-compliant; The governance platform is configured to protect the compliance module from tampering, attacks, or bypasses.

30. An electronic device, comprising: comprise: a memory; a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the method for enforcing a constrained uncertain system according to any one of claims 1 to 28.

31. A computer readable storage medium, characterized in that, a computer program is stored thereon, and the computer program is executed by a processor to implement the method for enforcing a constrained uncertain system according to any one of claims 1 to 28.

Images