A blockchain-based complex fusion network training system and method

Abstract

The application relates to a kind of complex fusion network training system and method based on blockchain, including target visual computing network and target feature chain;Target visual computing network is based on public data set and adaptive loss function, and pre-training is carried out for different task types to the target frame in the target monitoring video collected, and target feature information related to the current specific task is calculated;Target feature chain is used to store the target feature information after encryption.The application can process the identification of multiple tasks of pedestrians and vehicles in parallel, while introducing blockchain technology to effectively guarantee the security and non-tamperability of data, greatly improving the usability and practicality of the system, achieving the effect of enhancing system robustness, promoting multi-source data security fusion and improving complex scene perception ability.

Description

Technical Field

[0001] This invention relates to a complex fusion network training system and method based on blockchain, belonging to the field of computer vision technology. Background Technology

[0002] The massive amounts of video data generated by cameras have brought immeasurable value to the development of modern smart cities, and various visual perception tasks have always been an important research direction in this field. As the development of modern smart cities increasingly relies on video data, how to extract homogeneous semantics from video data and utilize complex fusion network training techniques to achieve parallel processing of multiple tasks has become a hot topic in the field of smart cities. Although different visual perception tasks require attention to different scene semantic information, these semantics fundamentally depend on specific structural and particulate semantics. In view of this, researchers have proposed complex fusion network training techniques, aiming to achieve efficient knowledge sharing and reuse by constructing a shared neural network capable of simultaneously processing multiple visual perception tasks. A significant feature of this type of technology is its powerful multi-task processing capability, which can integrate multiple visual perception tasks into a single model. However, existing complex fusion network training techniques focus on improving overall performance in centralized scenarios, often neglecting issues such as data privacy and security, cross-domain differences and synchronization, and external attack and defense of distributed systems in real-world applications. Limited by usability and practicality, these techniques are difficult to apply in real-world environments. In modern smart city scenarios involving cross-camera or multi-source data fusion, ensuring data privacy and security, addressing cross-domain differences and synchronization issues, and defending against external attacks on distributed systems have become pressing technical challenges. Summary of the Invention

[0003] This invention addresses the following technical problems: overcoming the challenges of data privacy and security, cross-domain differences and synchronization, and external attack and defense issues in distributed systems under existing complex fusion network training technologies in scenarios involving cross-camera or multi-source data fusion. It provides a blockchain-based complex fusion network training system and method capable of parallel processing of multiple tasks related to pedestrian and vehicle recognition. Furthermore, the introduction of blockchain technology effectively ensures data security and immutability, significantly improving system availability and practicality. This results in enhanced system robustness, promoted secure fusion of multi-source data, and improved perception capabilities in complex scenarios.

[0004] Technical solution of the present invention: Firstly, this invention provides a blockchain-based complex fusion network training system, comprising a target visual computing network and a target feature chain. The target visual computing network performs parallel pre-training on target frames from acquired target surveillance videos, based on a public dataset and an adaptive loss function, for different task types across multiple tasks, calculating target feature information related to the current specific task type. This target feature information is then stored in the target feature chain via smart contracts and chaincode. The target feature chain stores encrypted target feature information, ensuring its security and privacy. The target feature chain decrypts the encrypted historical target feature information via smart contracts and chaincode and sends it to the target visual computing network. The multiple tasks are determined by the user based on their needs. Simultaneously, it supports the independent execution of these multiple tasks for both pedestrian and vehicle target types.

[0005] Furthermore, the different task types among the various tasks are target pose estimation, target semantic segmentation, target detection, target re-identification, and target attribute recognition.

[0006] Furthermore, the target visual computing network includes: a surveillance video acquisition module, a feature encoding module, a task control module, and a feature decoding module; the target feature chain includes a blockchain trusted storage module and a blockchain interaction module. The surveillance video acquisition module acquires target surveillance video and extracts target frames from the target surveillance video, which are then input into the feature encoding module. This module consists of multiple cameras that do not overlap, with each camera existing as a separate node. The feature encoding module includes multiple Transformer encoders to preprocess the extracted target frames. It also supports synchronous preprocessing of target frames of different sizes. When these target frames are preprocessed, they are downsampled according to their size to obtain a set of scaled target frames of fixed size. Based on the scaled target frames, a dimension-consistent position embedding is learned. The position embedding is interpolated and input together with the scaled target frame group into multiple Transformer encoders for layer-by-layer optimization to obtain encoded global feature information. The encoded global feature information is then input into the feature decoding module. The task control module is responsible for the quantization processing of tasks, obtaining the corresponding embeddings and their position embeddings. Users input one or more tasks through this module. An adaptive target quantization method is used to adaptively quantize each task based on its specific task type, target type, and query, quantizing each task into its corresponding embedding and its position embedding, which are then input into the feature decoding module. The multiple tasks share the feature encoding and feature decoding modules, with each task's quantization process being independent and non-interfering with each other. The feature decoding module can simultaneously determine the feature information related to the current task from the global feature information, enabling parallel quantization processing of these multiple tasks and synchronous input to the feature decoding module. The feature decoding module includes multiple Transformer decoders. Based on the specific task quantized in the task control module, it decodes the global feature information generated by the feature encoding module into target feature information related to the current specific task. It further assigns output units to the obtained target feature information to obtain output information, which includes feature vectors, a global probability map, a local probability map, and bounding boxes. The output information is stored in the blockchain trusted storage module via a smart contract and chaincode, through the blockchain interaction module. Simultaneously, the blockchain interaction module determines whether the blockchain trusted storage module contains the same target historical information, which also includes feature vectors, a global probability map, a local probability map, and bounding boxes. If it does, the current output information and the target historical information are returned to the user; otherwise, only the current output information is returned to the user.

[0007] The blockchain trusted storage module contains multiple nodes that form a blockchain. All nodes have a consistent blockchain ledger locally, which is used to store the encrypted information output by the blockchain interaction module. This encrypted information has the decentralized, trusted, and traceable structural characteristics of the blockchain, ensuring the immutability of the encrypted information. The blockchain interaction module initiates blockchain transactions based on smart contracts and chaincode, enabling interoperability between the target visual computing network and the blockchain trusted storage module; it encrypts the output information stored in the blockchain by the feature decoding module using a chain-driven encryption algorithm, packages it into a blockchain transaction, and sends it to the blockchain trusted storage module; it is also responsible for decrypting the encrypted information already stored in the blockchain in the blockchain trusted storage module using a chain-driven encryption algorithm, packaging it into a blockchain transaction, and returning it to the feature decoding module.

[0008] Furthermore, the adaptive loss function is obtained by combining several sub-loss functions with a set of control gates; the several sub-loss functions include decoding loss function, segmentation loss function, regression loss function, detection loss function, and classification loss function; one or a combination of the above loss functions is obtained by controlling a set of control gates according to the task type; The decoding loss function is used to supervise the output information in the feature target visual computing network and correct the difference between the input and decoding output of global feature information according to the task type and target type. The segmentation loss function is used to measure the degree of overlap between the predicted segmentation region and the real segmentation in two tasks: target semantic segmentation and target detection. The regression loss function is used to calculate the error between the predicted value and the true value in two tasks: target pose estimation and target detection. The detection loss function is used to address the class imbalance problem in object detection tasks, emphasizing training for detected hard-to-classify targets; The classification loss function is used to correct the error between the predicted bounding box and the real bounding box (the predicted bounding box is the bounding box output by the feature decoding module) corresponding to the global feature information in the three tasks of target re-identification, target attribute identification and target semantic segmentation. The loss function is used to correct the distance between the detected samples and the positive and negative samples in the feature space during the target re-identification task; positive samples are close to each other, and negative samples are far from each other; the samples are target pedestrians or target vehicles, positive samples are the same target, and negative samples are different targets. The control gate configures one or a combination of corresponding loss functions for various tasks in the target visual computing network according to the task type and target type, thereby realizing an adaptive loss function.

[0009] Furthermore, in the task control module, the query method is as follows: the target pose estimation task is a pedestrian target-specific task; the target semantic segmentation task is responsible for segmenting the human body structure for pedestrian targets and for separating each vehicle instance from the background for vehicle targets; the target attribute recognition task is responsible for recognizing the characteristic attributes of pedestrians for pedestrian targets and for recognizing the specific attributes of vehicles for vehicle targets. The query is determined by both the task type and the target type. For pedestrian targets, the query for target pose estimation is the pose keypoint, and the number of queries is the number of pose keypoints; the query for target semantic segmentation is the pedestrian semantic segmentation category, and the number of queries is the number of semantic segmentation categories; the query for target detection is the size of the detected target at different anchor points, and the number of queries is the number of anchor points; the query for target re-identification is the dimension of the re-identification feature; the query for target attribute identification is the pedestrian attribute type, and the number of queries is the number of pedestrian attributes; for vehicle targets, the quantitative definition of target detection and target re-identification tasks is the same as that for pedestrian targets; there is no target pose estimation task for vehicle targets; the query for target semantic segmentation is the vehicle semantic segmentation category, and the number of queries is the number of semantic segmentation categories; the query for target attribute identification is the vehicle attribute type, and the number of queries is the number of vehicle attributes.

[0010] Furthermore, in the task control module, the adaptive quantization method is implemented as follows: (1) First, initialize all tasks input by the user as a query set related to the task. ,in The number of queries related to the corresponding task. Represents the content of each specific query; (2) Then, the query set corresponding to all tasks input by the user is obtained. Randomly initialized as a learnable embedding Optimize layer by layer in all Transformer encoders in the feature decoding module; (3) Finally, for each query vector set Bind a learnable position embedding Among them, for the four tasks of target pose estimation, target semantic segmentation, target re-identification, and target attribute recognition, the corresponding Perform random initialization; for the object detection task, its corresponding... Initialized as a mapping of anchor points that follow a uniform distribution in the target frame; learnable position embeddings. Shared across all Transformer decoders in the feature decoding module.

[0011] Furthermore, in the feature decoding module, the output information includes feature vectors, global probability maps, local probability maps, and bounding boxes. Corresponding to the output information, the feature decoding module output unit includes four types: a feature vector output unit, a global probability map output unit, a local probability map output unit, and a bounding box output unit. The feature vector output unit is activated for all tasks. In addition, the target semantic segmentation task and the target pose estimation task also require activation of the local probability map output unit and the global probability map output unit; the target detection task also requires activation of the global probability map output unit and the bounding box output unit; and the target attribute recognition task also requires activation of the global probability map output unit. The feature vectors, global probability maps, local probability maps, and bounding box information output by the above output units are all stored in the blockchain interaction module via smart contracts and chaincode.

[0012] Furthermore, in the blockchain trusted storage module, all nodes are trusted nodes by default, that is, nodes participating in blockchain interaction. If a node repeatedly inserts fabricated information, the node is updated to an untrusted node according to a reputation-based blockchain node election mechanism. The reputation-based blockchain node election mechanism periodically selects the node with the highest reputation as the leader node and prevents nodes with low reputation below a set threshold from participating in blockchain interaction, thereby preventing untrusted nodes from storing fabricated information. The implementation method of the reputation-based blockchain node election mechanism is as follows: (1) Create a pair of public and private keys for each node in the blockchain interaction module using the national cryptographic SM2 elliptic curve algorithm, and generate a digital signature based on the public key and broadcast it to other nodes in the blockchain; (2) When deploying the blockchain in the blockchain interaction model, store the public key, digital signature, and initial reputation value of all nodes in the genesis block. When a new node joins the blockchain, its public key and digital signature need to be packaged into the transaction and stored in the blockchain through smart contract and chain code. (3) Execute the leader node election algorithm. The node that wants to be elected as the leader node calculates a random number and a reputation proof through the VRF verifiable random function. If the reputation value of the node is non-negative, the node is considered to be qualified to run for leader node and is added to the leader node candidate list. Finally, the node's digital signature, public key, VRF-generated random number and reputation proof are broadcast to other nodes. The VRF is a special algorithm that can identify low-reputation nodes and verifiable random functions. (4) When a node receives digital signatures, public keys, random numbers generated by VRF and reputation proofs from other nodes, it verifies the credibility of the received random numbers and reputation proofs according to the verifiable random function of VRF. If they are credible, the node is approved to join the candidate list of leader nodes. If they are not credible, the node's reputation value is reduced. (5) Finally, select the node with the highest random value from the list of candidate leaders as the leader node.

[0013] The above process ensures that low-reputation nodes cannot become leader nodes, and the election process for leader nodes is unpredictable and sufficiently fair.

[0014] Furthermore, the blockchain trusted storage module also employs a hash graph-based blockchain consensus algorithm to ensure the data consistency of the blockchain ledgers located locally on each node; The process of the hash graph-based blockchain consensus algorithm is as follows: (1) Create an empty hash map copy on all nodes. The root node of the vertical axis of the hash map represents each node in the blockchain, and the other nodes on the vertical axis represent the blocks generated on their corresponding root nodes. The block data structure contains two hash pointers pointing to the parent block. (2) Each node maintains a copy of the hash map of each blockchain node through the Gossip protocol; each node attempts to randomly find other nodes and synchronize its local hash map copy through the Gossip protocol. (3) The node that receives the Gossip request creates a new block in its local hash map copy and points the two parent hash pointers in the block to its own last block and the last block of the Gossip source node; (4) Assign a creation round to all blocks in the hash graph replica and determine whether a block is a witness block in that round; (5) Conduct a virtual vote on all known witness blocks to elect a well-known witness block; (6) Determine the consensus order of all blocks through the well-known witness block to complete the consensus process; The witness block refers to the first block created on each node in round R of the hash graph replica; The well-known witness block refers to a block in the hash graph replica that is strongly visible to more than two-thirds of the witness blocks in round R+1. The term "strong visibility of a block" refers to the fact that the set of all paths from block A to block B in the hash graph replica includes more than two-thirds of the number of nodes. The aforementioned round allocation creation refers to the round being divided into a creation round and an acceptance round. The creation round is the round in which a block is created, and it generally belongs to the same round as its parent block. The acceptance round refers to the round in which a block is promoted to the next round when more than two-thirds of the witness blocks in the current round are strongly visible. The virtual voting refers to the election process of well-known witnesses. The witness blocks of round r+1 vote on the witness blocks of round r. As can be seen from the definition of round, the witness blocks of round r+2 are strongly visible to the witness blocks of round r+1. Therefore, the witness blocks of round r+2 count the votes on the witness blocks of round r+1 to determine the final voting result. The blocks mentioned above all refer to blocks in a blockchain.

[0015] Furthermore, in the blockchain trusted storage module, the blockchain transaction types are the following four: The first type is information storage transaction, which is responsible for encrypting the global feature information output by the feature encoding module and the specific information generated by each output unit in the feature decoding module by executing the chain-driven encryption algorithm, and storing the encrypted information in the blockchain trusted storage module to ensure the privacy of the information output by the target visual computing network. The second type is transaction query, which is responsible for initiating queries to other nodes in the blockchain to verify whether the target corresponding to the currently stored encrypted information has been stored in the blockchain, so as to achieve maximum effective knowledge sharing. The third type is to confirm the transaction, which is responsible for responding to whether the target corresponding to the encrypted information of the query initiating node of the transaction has been previously stored in the blockchain. If it exists, it responds to the query initiating node of the transaction; if it does not exist, it does not respond. The fourth type is information transmission transactions, which are responsible for extracting the encrypted information that the target previously stored in the blockchain, decrypting it using the chain-driven encryption algorithm, and then sending it to the node that initiated the query transaction.

[0016] Furthermore, in the blockchain interaction module, the encryption algorithm adopted is a chain-driven encryption algorithm; The chain-driven encryption algorithm includes a block encryption mechanism, a block decryption mechanism, a trusted transmission mechanism, and a distributed defense mechanism. The block encryption and decryption mechanisms work together to encrypt and decrypt information. The trusted transmission mechanism packages the information into blockchain transactions for transmission between nodes and the blockchain interaction module. The distributed defense mechanism overcomes the replay attack problem inherent in blockchain technology. The blockchain stores encrypted information; when storing information, it uses the block encryption mechanism, and when retrieving it, it uses the block decryption mechanism to restore the ciphertext. The trusted transmission mechanism ensures that the process of transmitting information to the blockchain is not attacked when a node initiates a storage or retrieval command. The distributed defense mechanism solves the replay attack problem. The trusted transmission mechanism, the distributed defense mechanism, and the encryption / decryption mechanism together ensure the overall security of the blockchain contained in the target feature chain. In the block cipher encryption mechanism, the node sends the information to be encrypted to the blockchain interaction module through smart contracts and chaincode. After receiving this information, the smart contract and chaincode obtain the key according to the node's identity, execute the SM4 block symmetric encryption algorithm on the information to be encrypted, and store the encrypted information in the blockchain trusted storage module. In the aforementioned block decryption mechanism, the node sends a request transaction to the blockchain transaction module through a smart contract and chaincode to obtain encrypted information. After receiving this encrypted information, the smart contract and chaincode obtain the key according to the node's identity and decrypt the encrypted information based on the SM4 block symmetric encryption algorithm, and return the decrypted information to the node. In the trusted transmission mechanism, the TLS protocol is deployed for all nodes, and the corresponding TLS encryption certificate is generated for all nodes using the national cryptographic SM2 elliptic curve algorithm to defend against active attack behavior during the transmission process. In the distributed defense mechanism, a random number (Nonce) is deployed in the blockchain transaction payload of each blockchain transaction. The blockchain processes transactions according to the ascending order of the Nonce values. When a transaction with an incorrect Nonce value order is found, the transaction is placed in the unconfirmed transaction memory pool to wait for the arrival of the correct transaction, thereby overcoming replay attacks in the distributed system.

[0017] Secondly, the present invention also provides a method for training complex fusion networks based on blockchain, which is implemented as follows: (1) Extract the target frame from the target surveillance video; (2) Preprocess the target frame and learn a position embedding with consistent dimensions. Optimize it layer by layer in the Transformer encoder to obtain the encoded global feature information and input it into the decoder; (3) The user inputs the task type and selects the target type. The embedding layer quantizes the task into the corresponding embedding and its position embedding according to the adaptive quantization method and inputs it into the decoder. (4) Based on the quantized task, the global feature information of the target is decoded into target feature information related to the specific task using the Transformer decoder; (5) Based on the quantified task, specify the output unit for the obtained target feature information to obtain task-related feature information, send it to the target feature chain for storage, and return it to the user synchronously; (6) The target feature chain is based on the feature information output by the feature decoding module of the smart contract and chain code processing. First, it encrypts these feature information according to the block encryption mechanism of the chain-driven encryption method. Then, the encrypted information is packaged into information storage transactions for transmission according to the trusted transmission mechanism of the chain-driven encryption algorithm. It also relies on the distributed defense mechanism of the chain-driven encryption algorithm to overcome the replay attack problem in blockchain technology. (7) After receiving the information storage transaction, the transaction is sorted according to a hash graph-based blockchain consensus algorithm, and a leader node is selected based on a reputation-based blockchain node election mechanism. The transaction is packaged into a block and stored in the blockchain to complete the information storage transaction process. (8) If the user specifies that the target’s historical feature information previously stored in the blockchain needs to be obtained, a Boolean flag is sent to the decoder. When the Boolean flag is true, the decoder will send the output feature vector to the target feature chain separately. (9) The target feature chain packages the feature vector output by the feature decoding module based on the smart contract and chain code processing into a query transaction according to the trusted transmission mechanism of the chain-driven encryption algorithm, and completes the transaction based on a blockchain consensus algorithm based on hash graph and a blockchain node election mechanism based on reputation. (10) A node in the blockchain that wants to respond to a query transaction obtains the corresponding feature vector from the block corresponding to the query transaction, calculates the cosine similarity between the feature vector and its own previously stored feature vectors based on the smart contract and chaincode, and determines whether the target corresponding to the encrypted information deposited by the money has already been stored in the blockchain; if not, it does not respond to the query transaction initiating node; if so, it initiates a confirmation transaction to the query node and completes the transaction based on a hash graph-based blockchain consensus algorithm and a reputation-based blockchain node election mechanism. (11) After the transaction is confirmed, the node in the blockchain that needs to respond to the query transaction extracts the encrypted information that has been stored in the blockchain from the target feature chain through the blockchain interaction module, and decrypts the encrypted information according to the smart contract and the chain-driven encryption algorithm, packages it into an information transmission transaction, and completes the transaction based on a hash graph-based blockchain consensus algorithm and a reputation-based blockchain node election mechanism. The decrypted information is then returned to the decoder in the target visual computing network and finally returned to the user.

[0018] Thirdly, the present invention also provides an electronic device, including a processor and a memory, wherein; Memory, used to store computer programs; A processor is used to execute computer programs stored in memory, and in doing so, implements the aforementioned methods or systems.

[0019] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned method or the aforementioned system.

[0020] The advantages of this invention compared to the prior art are: (1) This invention can process multiple task types of pedestrians and vehicles in parallel, including target pose estimation, target semantic segmentation, target detection, target re-identification and target attribute identification. At the same time, the introduction of blockchain technology effectively ensures the security and immutability of data, greatly improves the availability and practicality of the system, and achieves the effects of enhancing system robustness, promoting the secure fusion of multi-source data and improving the perception capability of complex scenes.

[0021] (2) This embodiment of the invention achieves efficient and secure processing and storage of target features in surveillance videos by combining a target visual computing network and a target feature chain. First, in the target visual computing network, parallel training is performed for multiple task types to improve training efficiency. Second, the target feature chain serves as a secure storage medium, encrypting and protecting the security and privacy of target feature information. Furthermore, the system supports multiple task types and two target types, which can be determined according to user needs, improving training flexibility. Finally, through the application of smart contracts and chaincode, the storage and interaction of target feature information are realized, and the decentralized structure of the blockchain ensures the immutability and trustworthiness of the information. Compared with existing technologies, this invention can efficiently perform multi-task training, protect the security of target feature information, and provide flexibility and trustworthiness, improving usability while ensuring training effectiveness.

[0022] (3) The embodiments of the present invention are based on blockchain technology to ensure the decentralized interaction and immutability of all feature data.

[0023] (4) The present invention designs a reputation-based blockchain node election mechanism to prevent untrusted nodes from uploading forged data, while ensuring that trusted nodes are elected as leader nodes with equal probability, thus ensuring fairness.

[0024] (5) The present invention provides a chain-driven encryption algorithm that can prevent data leakage and provide effective protection for the data transmission process and the security of the distributed system.

[0025] (6) The present invention designs a blockchain consensus algorithm based on hash graph, which greatly reduces the consumption of network resources by the blockchain consensus algorithm and improves the blockchain throughput. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A block diagram of a complex fusion network training system based on blockchain provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the adaptive loss function control gate in an embodiment of the present invention; Figure 3 This is a schematic diagram of the task input of the task control module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the input of the feature decoding module in an embodiment of the present invention; Figure 5 This is a schematic diagram of the chain-driven encryption algorithm in an embodiment of the present invention; Figure 6 This describes the implementation process of the reputation-based blockchain node election mechanism in this embodiment of the invention. Figure 7 This is a schematic diagram of a blockchain consensus algorithm based on a hash graph in an embodiment of the present invention; Figure 8 This is the hash graph structure in the embodiment of the present invention; Figure 9 This is the feature encoder structure in an embodiment of the present invention; Figure 10 This is the feature decoder structure in an embodiment of the present invention; Figure 11 This is a flowchart illustrating the implementation of the method in an embodiment of the present invention; Figure 12 This is a schematic diagram illustrating the feature interpretation of the feature decoding module in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0029] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] Existing complex fusion network training techniques focus on improving overall performance in centralized scenarios, often neglecting issues such as data privacy and security, cross-domain differences and synchronization, and external attack and defense of distributed systems in practical applications involving cross-camera or multi-source data fusion. This invention proposes a blockchain-based complex fusion network training system and method. Through adaptive quantization and the Transformer attention structure, it extracts cross-task homogeneous features from video data, enabling parallel processing of multi-dimensional visual tasks involving pedestrians and vehicles. By introducing blockchain technology, each camera is deployed as a blockchain node. The decentralized and traceable characteristics of blockchain ensure the security and immutability of cross-camera data transmission, solving the data security and synchronization problems of existing technologies. Furthermore, a reputation-based blockchain node election method is proposed, innovatively combining the national cryptographic SM2 elliptic curve algorithm and the VRF verifiable random function to assign dynamically assessable reputation values ​​to nodes in the blockchain network, effectively curbing the insertion of untrusted nodes. The system employs several techniques to address system instability and fairness. Forged information and a random number structure are introduced to ensure that trusted nodes are elected as leader nodes on an equal footing. A chain-driven encryption algorithm is proposed, using the SM4 block symmetric encryption algorithm to achieve full lifecycle encryption management of on-chain feature data. The SM2 elliptic curve algorithm generates private keys, public keys, digital signatures, and TLS encryption certificates for all nodes to defend against active attacks during data transmission and to establish data ownership. A block header transaction nonce is deployed in the blockchain transaction payload to defend against replay attacks on the distributed system. A hash graph-based blockchain consensus algorithm is also proposed, using the Gossip protocol for inter-node communication, significantly reducing network bandwidth consumption and improving system performance and consensus stability. These technologies enable parallel processing of five tasks: target pose estimation for pedestrians and vehicles, target semantic segmentation, target detection, target re-identification, and target attribute recognition. Blockchain technology is introduced to manage and protect the generated feature data throughout its lifecycle. While ensuring the overall effectiveness and efficiency of the system, the system's usability and practicality are significantly improved, achieving enhanced system robustness, promoting secure fusion of multi-source data, and improving the ability to perceive complex scenarios.

[0031] like Figure 1 As shown in the figure, an embodiment of the present invention provides a complex fusion network training system based on blockchain, including a target visual computing network and a target feature chain. The target visual computing network pre-trains on target frames in the collected target surveillance video based on a public dataset and an adaptive loss function for different task types, and calculates target feature information related to the current specific task type. The target feature information is then encrypted and stored in the target feature chain through smart contracts or chaincode. The target feature chain is used to store the encrypted target feature information, ensuring the security and privacy of the target feature information. Specifically, this invention provides five task types: target pose estimation, target semantic segmentation, target detection, target re-identification, and target attribute recognition. It also supports the independent execution of the above five tasks for both pedestrian and vehicle target types. Users can then expand the task scope according to their needs.

[0032] like Figure 1 As shown, specifically, the target visual computing network in this embodiment of the invention includes: a surveillance video acquisition module, a feature encoding module, a task control module, and a feature decoding module; the target feature chain includes a blockchain trusted storage module and a blockchain interaction module; the surveillance video module extracts target frames and inputs them to the feature encoding module; the feature encoding module preprocesses the target frames and encodes global feature information, which is then input to the feature decoding module; the user inputs a task through the task control module; the task control module inputs the quantized task to the feature decoding module; the feature decoding module decodes the global feature information into target feature information based on the quantized task and assigns an output unit to obtain feature vectors, a global probability map, and local features. The probability graph and bounding box are returned to the user, and simultaneously sent to the blockchain interaction module in the target feature chain based on the smart contract and chaincode. The blockchain interaction module packages the above information into a blockchain transaction and sends it to the blockchain trusted storage module. If the user requests information that the target frame corresponds to and has already been stored in the blockchain (hereinafter referred to as historical information), the feature decoding module sends the feature vector to the blockchain interaction module in the target feature chain based on the smart contract and chaincode. After calculation, the blockchain interaction module packages the transaction and sends it to the blockchain trusted storage module. The blockchain trusted storage module returns the historical information to the blockchain interaction module, and further returns it to the feature decoding module. The feature decoding module then returns the historical information to the user.

[0033] The following section will explain the specific content of each module in conjunction with the diagram.

[0034] 1. Surveillance video acquisition module The system acquires target surveillance video and extracts target frames from the target surveillance video, which are then input into the feature encoding module. This module consists of multiple cameras that do not overlap, with each camera existing as a separate node.

[0035] 2. Feature Encoding Module The module includes multiple Transformer encoders to preprocess the extracted target frames, resulting in a set of scaled target frames of fixed size. This module also supports synchronous preprocessing of target frames of different sizes. During preprocessing, these target frames are downsampled according to their size to obtain a set of scaled target frames of fixed size. Based on the scaled target frames, a dimension-consistent position embedding is learned. Interpolation calculations are performed on the position embedding, and the result is input together with the scaled target frame set into the Transformer encoder for layer-by-layer optimization to obtain encoded global feature information. The encoded global feature information is then input into the feature decoding module.

[0036] like Figure 9 As shown, after downsampling the target frame by 16 times, a set of image patches is obtained. These patches are then linearly transformed by the embedding layer to obtain corresponding patch embeddings. A corresponding position embedding is learned and fed into the Transformer encoder. In the encoder, the embeddings are first normalized. The normalized data is then input into the multi-head attention layer, where linear transformation and attention weight calculation yield a weighted output. The output of the multi-head attention layer is normalized again and input into the multilayer perceptron. The multilayer perceptron performs nonlinear transformations through multiple fully connected layers, outputting the processed feature vector. If there are subsequent encoders, the output of the multilayer perceptron will be further passed to the next encoder. Finally, the encoded global feature information is obtained. In mathematics ,in Refers to the first The output of the encoder, Patch embedding, This refers to embedding at the corresponding location.

[0037] 3. Task Control Module This module is responsible for quantization processing of tasks, obtaining the corresponding embeddings and their position embeddings. Users input one of five tasks through this module. An adaptive target quantization method is used, performing adaptive quantization based on the specific task type, target type, and query, quantizing each task into its corresponding embedding and its position embedding, which are then input into the feature decoding module. The five tasks share the feature encoding and feature decoding modules in the target visual computing network. The quantization process of each task is independent and does not interfere with each other. Therefore, the feature decoding module can simultaneously determine the feature information related to the current task in the global feature information, achieving parallel quantization processing of these five tasks and synchronously inputting them into the feature decoding module.

[0038] like Figure 3As shown, in the task control module, the query method is as follows: the target pose estimation task is a pedestrian target-specific task; the target semantic segmentation task is responsible for segmenting the human body structure for pedestrian targets and for separating each vehicle instance from the background for vehicle targets; the target attribute recognition task is responsible for recognizing the characteristic attributes of pedestrians, such as "wearing a hat", for pedestrian targets and for recognizing the specific attributes of vehicles, such as "license plate number", etc.

[0039] The number of queries included in different task types is determined according to the specific task type. The number of queries for target pose estimation, target semantic segmentation, target re-identification, and target attribute identification are pedestrian target pose key points, target semantic segmentation categories, target re-identification feature dimensions, and target attributes, respectively. The number of queries for target detection is the size of the detected target that appears at different anchor point positions in the training target frame.

[0040] The query is determined by both the task type and the target type. For pedestrian targets, the query for target pose estimation is the pose key points, such as "left shoulder, left knee, torso, head," etc., and the number of queries is the number of pose key points. The query for target semantic segmentation is the pedestrian semantic segmentation category, such as "hair, shirt, pants," etc., and the number of queries is the number of semantic segmentation categories. The query for target detection is the size of the detected target at different anchor points, and the number of queries is the number of anchor points. The query for target re-identification corresponds to the dimension of the re-identification feature. The query for target attribute identification is the pedestrian attribute type, such as "male, elderly," etc., and the number of queries is the number of pedestrian attributes. For vehicle targets, the quantitative definition of target detection and target re-identification tasks is the same as that for pedestrian targets. There is no target pose estimation task for vehicle targets. The query for target semantic segmentation is the vehicle semantic segmentation category, such as "window, door," and the number of queries is the number of semantic segmentation categories. The query for target attribute identification is the vehicle attribute type, such as "license plate number, color," etc., and the number of queries is the number of vehicle attributes.

[0041] It should be noted that the query method in the task control module of this embodiment of the invention precisely specifies the query content and quantity according to the task type and target type, ensuring the targeting and accuracy of task execution. Compared with the prior art, this method improves the system's ability to handle complex tasks, enabling target pose estimation, target semantic segmentation, target detection, target re-identification, and target attribute recognition tasks to be processed more efficiently in parallel, further enhancing the system's flexibility and practicality.

[0042] like Figure 4 As shown, in the task control module, the adaptive quantization method is implemented as follows: (1) First, initialize all tasks input by the user as a query set related to the task. ,in The number of queries related to the corresponding task. Represents the content of each specific query. ,in Target re-identification Target attribute recognition Target attitude estimation Refers to target semantic segmentation, Target detection.

[0043] (2) Then, the query set corresponding to all tasks input by the user is obtained. Randomly initialized as a learnable embedding The optimization is performed layer by layer in all Transformer encoders in the feature decoding module.

[0044] (3) Finally, for each query vector set Bind a learnable position embedding Among them, for the four tasks of target pose estimation, target semantic segmentation, target re-identification, and target attribute recognition, the corresponding Perform random initialization; for the object detection task, its corresponding Initialize the mapping to anchor points that follow a uniform distribution in the target frame, i.e. , refer to Anchor points following a uniform distribution; learnable position embedding Shared across all Transformer decoders in the feature decoding module.

[0045] It should be noted that the adaptive quantization method in the task control module of this embodiment of the invention achieves accurate capture and dynamic adjustment of query content by initializing the task-related query set as a learnable embedding and optimizing it layer by layer in the Transformer encoder. Simultaneously, by binding a learnable positional embedding to each query vector set and performing differentiated initialization for different task types, the system's sensitivity to task characteristics and target location is enhanced. Compared with existing technologies, this adaptive quantization method improves the accuracy and flexibility of queries and promotes information integration and utilization during feature decoding.

[0046] 4. Feature Decoding Module It includes multiple Transformer decoders. Based on the specific task quantized in the task control module, it decodes the global feature information generated by the feature encoding module into target feature information related to the current specific task. According to the specific task quantized in the task control module, it assigns an output unit to the obtained target feature information to obtain output information, including feature vectors, global probability maps, local probability maps, and bounding boxes. The above output information is stored in the blockchain trusted storage module through the blockchain interaction module via smart contracts and chaincode. This module uses smart contracts and chaincode as a medium to determine whether the blockchain trusted storage module contains historical information of the same target. Historical information includes feature vectors, global probability maps, local probability maps, and bounding boxes. If it exists, it returns them to the user. If not, it only returns the current output information of the module to the user.

[0047] like Figure 10 As shown, in the decoder, based on global feature information Image patch embedding coordinates relative to the original image The mapping yields the corresponding position embedding. The position of the same task query set is embedded. The inputs are fed into a cross-attention layer, where they undergo linear transformation and attention weight calculation to obtain a weighted output. The output of the cross-attention layer is then residually concatenated with the input, and the result is normalized before being fed into a self-attention layer for attention calculation. The output of the self-attention layer is then residually concatenated with the input, and the result is normalized before being fed into a multilayer perceptron. The multilayer perceptron performs nonlinear transformations through multiple fully connected layers, outputting a processed feature vector. If there are subsequent decoders, the output of the multilayer perceptron is further passed to the next decoder. Finally, the decoded task-related target feature information is obtained. In mathematics ,in Refers to the first The output of each decoder.

[0048] like Figure 12As shown, specifically, the feature decoding module provided in this embodiment of the invention includes four output units: a feature vector output unit, a global probability map output unit, a local probability map output unit, and a bounding box output unit. The global probability map output unit, local probability map output unit, and bounding box output unit need to be output to a multilayer perceptron. After a series of linear transformations and nonlinear activation functions, the final output representation is obtained. All tasks activate the feature vector output unit. Furthermore, the target semantic segmentation task and the target pose estimation task also need to activate the local probability map output unit and the global probability map output unit; the target detection task also needs to activate the global probability map output unit and the bounding box output unit; and the target attribute recognition task also needs to activate the global probability map output unit. The feature vectors, global probability maps, local probability maps, and bounding box information output by the above output units are all stored in the blockchain interaction module via smart contracts and chaincode.

[0049] 5. Blockchain Trusted Storage Module It comprises multiple nodes that form a blockchain. All nodes maintain a consistent blockchain ledger locally, used to store encrypted information output by the blockchain interaction module. This information includes feature vectors, global probability graphs, local probability graphs, and bounding boxes. The decentralized, trustworthy, and traceable structural characteristics of the blockchain ensure the immutability of this encrypted information. All nodes are trusted by default, meaning they can participate in blockchain interactions. If a node repeatedly inserts fabricated information, it is updated to an untrusted node according to a reputation-based blockchain node election mechanism. This reputation-based blockchain node election mechanism periodically selects the node with the highest reputation as the leader node and prevents nodes with reputations below a set threshold from participating in blockchain interactions, thus avoiding untrusted nodes storing fabricated information. like Figure 6 As shown, in particular, the implementation method of a reputation-based blockchain node election mechanism provided by this embodiment of the invention is as follows: (1) Create a pair of public and private keys for each node in the blockchain interaction module using the national cryptographic SM2 elliptic curve algorithm, return the private key to the user for local storage, and generate a digital signature based on the public key and broadcast it to other nodes in the blockchain. All hash SHA256 algorithms are replaced by the national cryptographic SM3 hash algorithm. (2) When deploying the blockchain in the blockchain interaction model, store the public key, digital signature, and initial reputation value of all nodes in the genesis block. When a new node joins the blockchain, its public key and digital signature need to be packaged into the transaction and stored in the blockchain through smart contract and chain code. (3) During the proof phase, execute the leader node election algorithm and initialize the leader node set. A node that wants to be elected as the leader node can obtain a random number through a VRF verifiable random function. With a certificate of credibility If a node's reputation value is non-negative, then the node is considered eligible to run for leader and is added to the leader candidate list. Mathematically, this is represented as... ,in This serves as the identifier for the node; ultimately, it broadcasts the node's digital signature, public key, VRF-generated random number, and reputation proof to other nodes; VRF is a special algorithm that can identify low-reputation nodes and verify random functions. (4) During the verification phase, when a node receives digital signatures, public keys, VRF-generated random numbers, and reputation proofs broadcast from other nodes, it verifies the credibility of the received random numbers and reputation proofs using the VRF verifiable random function. If credible, the node is approved to join the leader node candidate list. Mathematically, this is represented as... If a node is not trustworthy, its reputation value will be lowered. (5) Finally, select the node with the highest random value from the list of candidate leaders as the leader node.

[0050] The above process ensures that low-reputation nodes cannot become leader nodes, and the election process for leader nodes is unpredictable and sufficiently fair.

[0051] It should be noted that the reputation-based blockchain node election mechanism proposed in this invention innovatively combines the national cryptographic SM2 elliptic curve algorithm with a verifiable random function (VRF) to assign dynamically assessable reputation values ​​to nodes in the blockchain network and introduces random numbers for leader node election. This mechanism not only responds quickly to malicious behavior and effectively curbs information fabrication, but also ensures that high-reputation nodes among trusted nodes do not repeatedly become leaders, guaranteeing the fairness of the election. Compared with existing technologies, the election mechanism of this invention is more fair and transparent, and can more accurately identify and exclude potential risk nodes, thereby maintaining the purity of the distributed network and providing a solid guarantee for the data security and stable operation of complex fusion network training systems.

[0052] like Figure 7 As shown, the blockchain trusted storage module also employs a hash graph-based blockchain consensus algorithm to ensure data consistency of the blockchain ledgers located locally on each node.

[0053] The process of the hash graph-based blockchain consensus algorithm is as follows: (1) Create an empty hash map copy locally on all nodes, such as Figure 8As shown, the root node on the vertical axis of the hash graph represents each node in the blockchain, and the other nodes on the vertical axis represent the blocks generated on their corresponding root nodes. The block data structure contains two hash pointers pointing to the parent block. The entire hash graph structure is built based on the two hash pointers of each block, and the blocks are divided according to their respective rounds R. (2) Each node maintains a copy of the hash map of each blockchain node through the Gossip protocol; each node attempts to initiate a Gossip request to randomly find other nodes and synchronize its local hash map copy through the Gossip protocol. (3) The node that receives the Gossip request creates a new block in its local hash map copy and points the two hash pointers in the block to its own last block and the last block of the Gossip source node; (4) Assign a creation round to all blocks in the hash graph replica and determine whether a block is a witness block in that round; (5) Conduct a virtual vote on all known witness blocks to elect a well-known witness block; (6) Determine the consensus order of all blocks through the well-known witness block to complete the consensus process; The witness block refers to the first block created on each node in round R of the hash graph replica; The well-known witness block refers to a block in the hash graph replica that is strongly visible to more than two-thirds of the witness blocks in round R+1. The term "strong visibility of a block" refers to the fact that the set of all paths from block A to block B in the hash graph replica includes more than two-thirds of the number of nodes. The aforementioned round allocation creation refers to the round being divided into a creation round and an acceptance round. The creation round is the round in which a block is created, and it generally belongs to the same round as its parent block. The acceptance round refers to the round in which a block is promoted to the next round when more than two-thirds of the witness blocks in the current round are strongly visible. The virtual voting mentioned refers to the election process of well-known witnesses. The witness blocks from round r+1 vote on the witness blocks from round r. As defined by the round number, the witness blocks from round r+2 have a strong visibility over the witness blocks from round r+1. Therefore, the witness blocks from round r+2 then count the votes against the witness blocks from round r+1, thus determining the final voting result. All the aforementioned blocks refer to blocks in the blockchain.

[0054] In the blockchain trusted storage module, the transaction types are as follows: The first type is information storage transaction, which is responsible for performing a chain-driven encryption method on the global feature information output by the feature encoding module and the specific information generated by each output unit in the feature decoding module, and storing the encrypted information in the blockchain trusted storage module, so as to ensure the privacy of the specific information output by the target visual computing network. The second type is transaction query, which is responsible for initiating queries to other nodes in the blockchain to verify whether the target corresponding to the currently stored encrypted information has been stored in the blockchain, so as to achieve maximum effective knowledge sharing. The third type is to confirm the transaction, which is responsible for responding to whether the target corresponding to the encrypted information of the query initiating node of the transaction has been previously stored in the blockchain. If it exists, it responds to the query initiating node of the transaction; otherwise, it does not respond. The fourth type is information transmission transactions, which are responsible for extracting the encrypted information that the target previously stored in the blockchain, decrypting it according to the chain-driven encryption algorithm, and then sending it to the node that initiated the query transaction.

[0055] It should be noted that the hash graph-based blockchain consensus algorithm of this invention significantly reduces network bandwidth consumption by constructing hash graph replicas and utilizing the Gossip protocol for inter-node synchronization, while also reducing the risk of consensus failure due to network latency or malicious behavior. Compared with existing technologies, the consensus algorithm of this invention provides a more efficient and stable consensus and transaction verification scheme for complex blockchain-based fusion network training systems.

[0056] 6. Blockchain Interaction Module Based on smart contracts and chaincode, blockchain transactions are initiated to achieve interoperability between the target visual computing network and the blockchain trusted storage module. This module is responsible for encrypting the feature vectors, global probability maps, local probability maps, and bounding boxes that the feature decoding module in the target visual computing network needs to store in the blockchain using a chain-driven encryption method, packaging them into blockchain transactions, and then sending them to the blockchain trusted storage module. This module is also responsible for packaging the encrypted information already stored in the blockchain in the blockchain trusted storage module into blockchain transactions and returning them to the feature decoding module. This information includes feature vectors, global probability maps, local probability maps, and bounding boxes.

[0057] like Figure 5 As shown, in the blockchain interaction module, a chain-driven encryption method is used to encrypt all information that needs to be written to the blockchain trusted storage module. This information includes the feature vector, global probability map, local probability map and calibration box information output by the feature decoding module. The described chain-driven encryption method includes a block encryption mechanism, a block decryption mechanism, a trusted transmission mechanism, and a distributed defense mechanism. The block encryption and decryption mechanisms work together to encrypt and decrypt information. The trusted transmission mechanism packages the information into blockchain transactions for transmission between nodes and the blockchain interaction module. The distributed defense mechanism overcomes the replay attack problem inherent in blockchain technology. The information stored in the blockchain is encrypted using the block encryption mechanism when it is stored, and the ciphertext is restored using the block decryption mechanism when it is retrieved. The trusted transmission mechanism ensures that the process of transmitting information to the blockchain is not attacked when a node initiates a storage or retrieval command. The distributed defense mechanism solves the replay attack problem. The trusted transmission mechanism, the distributed defense mechanism, and the encryption / decryption mechanism together ensure the overall security of the blockchain contained in the target feature chain. In the block cipher encryption mechanism, the node The corresponding camera sends video data to the target visual computing network. After calculation, the information that needs to be encrypted is sent to the blockchain interaction module through smart contracts and chaincode. After receiving this information, the smart contract and chaincode obtain the key of the corresponding SM4 block symmetric encryption algorithm according to the node identity, and execute the SM4 block symmetric encryption algorithm on the information that needs to be encrypted. The encrypted information is then stored in the blockchain trusted storage module through information storage transactions. In the aforementioned block decryption mechanism, the node Request to retrieve historical feature information of the target that has been stored in the blockchain, node Using smart contracts and chaincode as a medium, a query transaction is sent to the blockchain interaction module to request this information. If the node... In response to this transaction, a confirmation message is sent to the blockchain interaction module via a smart contract and chaincode to respond to the node. Historical data and features stored in the blockchain exist; after a transaction is confirmed, the node... Based on the node's identity, the corresponding SM4 block symmetric encryption key is obtained. The historical feature information previously stored in the blockchain is decrypted using the SM4 block symmetric encryption algorithm. Based on the smart contract and chaincode, the information is transmitted to the blockchain interaction module, and the decrypted information is returned to the node. .

[0058] In the trusted transmission mechanism, the TLS protocol is deployed for all nodes, and the corresponding TLS encryption certificate is generated for all nodes using the national cryptographic SM2 elliptic curve algorithm to defend against active attacks during transmission. Specifically, the transaction payload of each blockchain transaction initiated based on smart contracts and chaincode must carry the TLS encryption certificate corresponding to the node in order to be successfully initiated.

[0059] In the distributed defense mechanism, a random number (Nonce) is deployed in the blockchain transaction payload of each blockchain transaction. The blockchain processes transactions according to the ascending order of the Nonce values. When a transaction with an incorrect Nonce value order is found, the transaction is placed in the unconfirmed transaction memory pool to wait for the arrival of the correct transaction, thereby overcoming replay attacks in the distributed system.

[0060] It should be noted that the chain-driven encryption algorithm described in the above embodiments of the present invention integrates block encryption and decryption mechanisms, trusted transmission mechanisms, and distributed defense mechanisms. Compared with existing technologies, it provides a more comprehensive and efficient data protection solution. The block encryption and decryption mechanism ensures the high confidentiality of data during blockchain storage and transmission; the trusted transmission mechanism effectively resists man-in-the-middle attacks, guaranteeing the integrity and authenticity of data transmission; and the distributed defense mechanism provides a powerful solution to the replay attack problem unique to blockchain. These three mechanisms work together to not only improve the overall security of the blockchain network but also promote the reliable flow and utilization of data in complex fusion network training systems, thereby improving the system's availability and practicality.

[0061] like Figure 2 As shown, in particular, the adaptive loss function in this embodiment of the invention is obtained by combining several sub-loss functions with a set of control circuit gates; the several sub-loss functions include decoding loss function, segmentation loss function, regression loss function, detection loss function and classification loss function; the above-mentioned loss functions are obtained by controlling a set of control gates according to the task type; The decoding loss function is used to supervise the output information in the feature decoding module and correct the difference between the input and decoding output of global feature information according to the task type and target type. The segmentation loss function is used to measure the degree of overlap between the predicted segmentation region and the real segmentation in two tasks: target semantic segmentation and target detection. The regression loss function is used to calculate the error between the predicted value and the true value in two tasks: target pose estimation and target detection. The detection loss function is used to address class imbalance in object detection tasks, emphasizing training for detected, hard-to-classify targets. Specifically, a balance factor is set in the focus loss component of the detection loss function. This allows us to focus on training targets that are difficult to classify. The classification loss function is used to correct the error between the predicted bounding box and the real bounding box (the predicted bounding box is the bounding box output by the feature decoding module) corresponding to the global feature information in the three tasks of target re-identification, target attribute identification and target semantic segmentation. The loss function is used to correct the distance between the detected samples and the positive and negative samples in the feature space during the target re-identification task; positive samples are close to each other, and negative samples are far from each other; the samples are target pedestrians or target vehicles, positive samples are the same target, and negative samples are different targets. The control gates are used to configure one or a combination of the five corresponding loss functions for the target visual computing network according to the task type and target type, thereby realizing an adaptive loss function. Specifically, there are three levels of control gates. The first level of control gates includes pedestrian target gates and vehicle target gates to control the parameter settings for selecting the correct object. The second level of control gates includes five types: target re-identification gate, target attribute recognition gate, target semantic segmentation gate, target pose estimation gate, and target detection gate, to control the parameter design for selecting the correct task. The third level of control gates, based on the results of the first two levels of control gates, assigns an adaptive loss function to the system according to the target type and task type, thereby realizing an adaptive loss function.

[0062] It should be noted that the adaptive loss function design in this embodiment of the invention achieves adaptive adjustment for different task types by combining multiple sub-loss functions such as decoding, segmentation, regression, detection, and classification, and introducing a control circuit gate mechanism. This design not only improves the flexibility and accuracy of the loss function, but also effectively addresses challenges such as class imbalance and sample differences, thereby optimizing the training effect of the target visual computing network. Compared with existing technologies, the loss function of this invention can more accurately reflect task requirements, promote the performance improvement of the model in complex scenarios, and achieve better training and practical application results.

[0063] like Figure 11 As shown, in particular, this embodiment of the invention provides a blockchain-based method for training complex fusion networks, including a target visual computing network and a target feature chain, implemented as follows: (1) Extract the target frame from the target surveillance video; (2) Preprocess the target frame and learn a position embedding with consistent dimensions. Optimize it layer by layer in the Transformer encoder to obtain the encoded global feature information and input it into the decoder; (3) The user inputs the task type and selects the target type. The embedding layer quantizes the task into the corresponding embedding and its position embedding according to the adaptive quantization method and inputs it into the decoder. (4) Based on the quantized task, the global feature information of the target is decoded into target feature information related to the specific task using the Transformer decoder; (5) Based on the quantified task, specify the output unit for the obtained target feature information to obtain task-related feature information, send it to the target feature chain for storage, and return it to the user synchronously; (6) The target feature chain is based on the feature information output by the feature decoding module of the smart contract and chain code processing. First, it encrypts these feature information according to the block encryption mechanism of the chain-driven encryption method. Then, the encrypted information is packaged into information storage transactions for transmission according to the trusted transmission mechanism of the chain-driven encryption algorithm. It also relies on the distributed defense mechanism of the chain-driven encryption algorithm to overcome the replay attack problem in blockchain technology. (7) After receiving the information storage transaction, the transaction is sorted according to a hash graph-based blockchain consensus algorithm, and a leader node is selected based on a reputation-based blockchain node election mechanism. The transaction is packaged into a block and stored in the blockchain to complete the information storage transaction process. (9) If the user specifies that the target’s historical feature information previously stored in the blockchain needs to be obtained, a Boolean flag is sent to the decoder. When the Boolean flag is true, the decoder will send the output feature vector separately to the target feature chain. (10) The target feature chain packages the feature vector output by the feature decoding module based on the smart contract and chain code processing into a query transaction according to the trusted transmission mechanism of the chain-driven encryption algorithm, and completes the transaction based on a blockchain consensus algorithm based on hash graph and a blockchain node election mechanism based on reputation. (11) A node in the blockchain that wants to respond to a query transaction obtains the corresponding feature vector from the block corresponding to the query transaction, calculates the cosine similarity between the feature vector and its own previously stored feature vectors based on the smart contract and chaincode, and determines whether the target corresponding to the encrypted information deposited by the money has already been stored in the blockchain; if not, it does not respond to the query transaction initiating node; if so, it initiates a confirmation transaction to the query node and completes the transaction based on a hash graph-based blockchain consensus algorithm and a reputation-based blockchain node election mechanism. (12) After the transaction is confirmed, the node in the blockchain that needs to respond to the query transaction extracts the encrypted information that has been stored in the blockchain from the target feature chain through the blockchain interaction module, and decrypts the encrypted information according to the smart contract and chain-driven encryption algorithm, packages it into an information transmission transaction, and completes the transaction based on a hash graph-based blockchain consensus algorithm and a reputation-based blockchain node election mechanism. The decrypted information is then returned to the decoder in the target visual computing network and finally returned to the user.

[0064] The execution process described above is similar to that of the system provided in the embodiments of the present invention.

[0065] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (including a computer, server, smartphone, etc.) comprising a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes steps and systems for performing the methods and systems provided above in the embodiments of the present invention.

[0066] Based on the same inventive concept, another embodiment of the present invention also provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program. When the computer program is executed by a computer, it implements the various steps and systems of the methods provided in the above embodiments of the present invention.

[0067] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A complex fusion network training system based on blockchain, characterized in that: This includes target visual computing networks and target feature chains; The target visual computing network performs parallel pre-training on target frames in the acquired target surveillance video, based on a public dataset and an adaptive loss function, for different task types across multiple tasks. It calculates target feature information related to the current specific task type and stores this information in a target feature chain via smart contracts and chaincode. The target feature chain stores encrypted target feature information to ensure its security and privacy. The target feature chain then decrypts the encrypted historical target feature information via smart contracts and chaincode and sends it to the target visual computing network. The various tasks are determined by the user according to their needs; the system supports the independent execution of these tasks for both pedestrians and vehicles; the different task types among these tasks are target pose estimation, target semantic segmentation, target detection, target re-identification, and target attribute recognition. The target visual computing network includes a surveillance video acquisition module, a feature encoding module, a task control module, and a feature decoding module; the target feature chain includes a blockchain trusted storage module and a blockchain interaction module. The surveillance video acquisition module consists of multiple cameras that do not overlap, with each camera existing as a separate node. The system acquires target surveillance video and extracts target frames from the target surveillance video, which are then input into the feature encoding module. The feature encoding module preprocesses the target frame to obtain a set of scaled target frames of fixed size. Based on the scaled target frame, a dimension-consistent position embedding is learned. The position embedding is interpolated and input together with the scaled target frame group into multiple Transformer encoders for layer-by-layer optimization to obtain encoded global feature information. The encoded global feature information is then input into the feature decoding module. The task control module employs an adaptive target quantization method, which performs adaptive quantization based on the specific task type, target type, and query in various tasks, quantizing each task into a corresponding embedding and its position embedding input to the feature decoding module. The multiple tasks share a feature encoding module and a feature decoding module. The quantization process of each task is independent and does not interfere with each other. The feature decoding module can simultaneously use the feature decoding module to determine the feature information related to the current task in the global feature information, realize the parallel quantization processing of these multiple tasks, and synchronously input them to the feature decoding module. The feature decoding module includes multiple Transformer decoders. Based on the specific task quantized in the task control module, it decodes the global feature information generated by the feature encoding module into target feature information related to the current specific task. It further assigns output units to the obtained target feature information to obtain output information. The output information includes feature vectors, global probability maps, local probability maps, and bounding boxes. The blockchain trusted storage module contains multiple nodes that form a blockchain. All nodes have a consistent blockchain ledger locally, which is used to store the encrypted information output by the blockchain interaction module. This encrypted information has the decentralized, trusted, and traceable structural characteristics of the blockchain, ensuring the immutability of the encrypted information. The blockchain interaction module initiates blockchain transactions based on smart contracts and chaincode, enabling interoperability between the target visual computing network and the blockchain trusted storage module.

2. The blockchain-based complex fusion network training system according to claim 1, characterized in that: The adaptive loss function is obtained by combining several sub-loss functions with a set of control gates; the several sub-loss functions include decoding loss function, segmentation loss function, regression loss function, detection loss function, classification loss function, and metric loss function; one or a combination of the above loss functions is obtained by controlling a set of control gates according to the task type; The decoding loss function is used to supervise the output information in the feature target visual computing network and correct the difference between the input and decoding output of global feature information according to the task type and target type. The segmentation loss function is used to measure the degree of overlap between the predicted segmentation region and the real segmentation in two tasks: target semantic segmentation and target detection. The regression loss function is used to calculate the error between the predicted value and the true value in two tasks: target pose estimation and target detection. The detection loss function is used to address the class imbalance problem in object detection tasks, emphasizing training for detected hard-to-classify targets; The classification loss function is used to correct the error between the predicted bounding box and the real bounding box corresponding to the global feature information in the three tasks of target re-identification, target attribute identification, and target semantic segmentation. The metric loss function is used to correct the distance between the detected sample and the positive and negative samples in the feature space during the target re-identification task. The samples are target pedestrians or target vehicles; positive samples are the same target, and negative samples are different targets.

3. The blockchain-based complex fusion network training system according to claim 1, characterized in that, In the task control module, the query method is as follows: the target pose estimation task is a pedestrian target-specific task; the target semantic segmentation task is responsible for segmenting the human body structure for pedestrian targets and for separating each vehicle instance from the background for vehicle targets; the target attribute recognition task is responsible for recognizing the characteristic attributes of pedestrians for pedestrian targets and for recognizing the specific attributes of vehicles for vehicle targets. The query is determined by both the task type and the target type. For pedestrian targets, the query for target pose estimation is the pose key points, and the number of queries is the number of pose key points; the query for target semantic segmentation is the pedestrian semantic segmentation category, and the number of queries is the number of semantic segmentation categories; the query for target detection is the size of the detected target at different anchor point positions, and the number of queries is the number of anchor points; the query for target re-identification corresponds to the dimension of the re-identification feature. The query for target attribute recognition is the type of pedestrian attribute, and the number of queries is the number of pedestrian attributes; for vehicle targets, the quantitative definition of target detection and target re-identification tasks is the same as pedestrian targets. There is no target pose estimation task for vehicle targets; the query for target semantic segmentation is the vehicle semantic segmentation category, and the number of queries is the number of semantic segmentation categories; The query for target attribute identification is the type of vehicle attribute, and the number of queries is the number of vehicle attributes.

4. The blockchain-based complex fusion network training system according to claim 1, characterized in that, In the task control module, the adaptive quantization method is implemented as follows: (1) First, initialize all input tasks as task-related query sets. ,in The number of queries related to the corresponding task. Represents the content of each specific query; (2) Then, the query set corresponding to all input tasks. Randomly initialized as a learnable embedding Optimize layer by layer in all Transformer encoders in the feature decoding module; (3) Finally, for each query vector set Bind a learnable position embedding Among them, for the four tasks of target pose estimation, target semantic segmentation, target re-identification, and target attribute recognition, the corresponding Perform random initialization; for the object detection task, its corresponding... Initialized as a mapping of anchor points that follow a uniform distribution in the target frame; learnable position embeddings. Shared across all Transformer decoders in the feature decoding module.

5. The blockchain-based complex fusion network training system according to claim 1, characterized in that: In the feature decoding module, corresponding to the output information, the output unit of the feature decoding module includes four types: feature vector output unit, global probability map output unit, local probability map output unit, and bounding box output unit; wherein the feature vector output unit is activated for all tasks; in addition, the target semantic segmentation task and the target pose estimation task also need to activate the local probability map output unit and the global probability map output unit, the target detection task also needs to activate the global probability map output unit and the bounding box output unit, and the target attribute recognition also needs to activate the global probability map output unit.

6. The blockchain-based complex fusion network training system according to claim 1, characterized in that, In the blockchain trusted storage module, all nodes are trusted nodes by default, that is, nodes that participate in blockchain interaction. If a node repeatedly inserts fabricated information, the node will be updated to an untrusted node according to a reputation-based blockchain node election mechanism. The implementation method of the reputation-based blockchain node election mechanism is as follows: (1) Create a pair of public and private keys for each node in the blockchain interaction module using the national cryptographic SM2 elliptic curve algorithm, and generate a digital signature based on the public key and broadcast it to other nodes in the blockchain; (2) When deploying the blockchain in the blockchain interaction model, store the public key, digital signature, and initial reputation value of all nodes in the genesis block. When a new node joins the blockchain, its public key and digital signature need to be packaged into the transaction and stored in the blockchain through smart contract and chain code. (3) Execute the leader node election algorithm. The node that wants to be elected as the leader node calculates a random number and a reputation proof through the VRF verifiable random function. If the reputation value of the node is non-negative, the node is considered to be qualified to run for leader node and is added to the leader node candidate list. Finally, the node's digital signature, public key, VRF-generated random number and reputation proof information are broadcast to other nodes. The VRF is a special algorithm that can identify low-reputation nodes and verifiable random functions. (4) When a node receives digital signatures, public keys, random numbers generated by VRF and reputation proof information from other nodes, it verifies whether the received random numbers and reputation proof are trustworthy according to the VRF verifiable random function. If they are trustworthy, the node is approved to join the candidate list of leader nodes. If they are not trustworthy, the node's reputation value is reduced. (5) Finally, select the node with the highest random value from the list of candidate leaders as the leader node.

7. The blockchain-based complex fusion network training system according to claim 1, characterized in that, The blockchain trusted storage module also employs a hash graph-based blockchain consensus algorithm to ensure data consistency of the blockchain ledgers located locally on each node. The process of the hash graph-based blockchain consensus algorithm is as follows: (1) Create an empty hash map copy on all nodes. The root node of the vertical axis of the hash map represents each node in the blockchain, and the other nodes on the vertical axis represent the blocks generated on their corresponding root nodes. The block data structure contains two hash pointers pointing to the parent block. (2) Each node maintains a copy of the hash map of each blockchain node through the Gossip protocol; Each node randomly finds other nodes and synchronizes its local copy of the hash map using the Gossip protocol; (3) The node that receives the Gossip request creates a new block in its local hash map copy and points the two parent hash pointers in the block to its own last block and the last block of the Gossip source node; (4) Assign a creation round to all blocks in the hash graph replica and determine whether a block is a witness block in that round; (5) Conduct a virtual vote on all known witness blocks to elect a well-known witness block; (6) Determine the consensus order of all blocks through the well-known witness block to complete the consensus process; The witness block refers to the first block created on each node in round R of the hash graph replica; The well-known witness block refers to a block in the hash graph replica that is strongly visible to more than two-thirds of the witness blocks in round R+1. The term "strong visibility of a block" refers to the fact that the set of all paths from block A to block B in the hash graph replica includes more than two-thirds of the number of nodes. The aforementioned round allocation creation refers to the round being divided into a creation round and an acceptance round. The creation round is the round in which a block is created, and it generally belongs to the same round as its parent block. The acceptance round refers to the round in which a block is promoted to the next round when more than two-thirds of the witness blocks in the current round are strongly visible. The virtual voting refers to the election process of well-known witnesses. The witness blocks of round r+1 vote on the witness blocks of round r. As can be seen from the definition of round, the witness blocks of round r+2 are strongly visible to the majority of the witness blocks of round r+1. Therefore, the witness blocks of round r+2 count the votes on the witness blocks of round r+1 to determine the final voting result.

8. The blockchain-based complex fusion network training system according to claim 1, characterized in that: In the blockchain trusted storage module, the blockchain transaction types are as follows: The first type is information storage transaction, which is responsible for encrypting the specific information generated by each output unit in the feature decoding module and storing the encrypted information in the blockchain trusted storage module to ensure the privacy of the target visual computing network output information; The second type is transaction query, which is responsible for initiating queries to other nodes in the blockchain to verify whether the target corresponding to the currently stored encrypted information has been stored in the blockchain, so as to achieve maximum effective knowledge sharing. The third type is to confirm the transaction, which is responsible for responding to whether the target corresponding to the encrypted information of the query initiating node of the transaction has been previously stored in the blockchain. If it exists, it responds to the query initiating node of the transaction; if it does not exist, it does not respond. The fourth type is information transmission transactions, which are responsible for extracting encrypted information previously stored in the blockchain, decrypting it, and sending it to the node that initiated the query transaction.

9. A complex fusion network training system based on blockchain according to claim 1, characterized in that, The blockchain interaction module encrypts the output information of the feature decoding module using a chain-driven encryption algorithm, packages it into a blockchain transaction, and sends it to the blockchain trusted storage module. At the same time, it is also responsible for decrypting the encrypted information already stored in the blockchain in the blockchain trusted storage module using a chain-driven encryption algorithm, packaging it into a blockchain transaction, and returning it to the feature decoding module. The chain-driven encryption algorithm includes a block encryption mechanism, a block decryption mechanism, a trusted transmission mechanism, and a distributed defense mechanism. The block encryption and decryption mechanisms work together to encrypt and decrypt information. The trusted transmission mechanism packages the information into blockchain transactions for transmission between nodes and the blockchain interaction module. The distributed defense mechanism overcomes the replay attack problem inherent in blockchain technology. The information stored in the blockchain is encrypted using the block encryption mechanism when it is stored, and the block decryption mechanism is used to restore the ciphertext when it is retrieved. A trusted transmission mechanism is a mechanism that ensures the process of transmitting information to the blockchain is not attacked when a node initiates a access command to the blockchain; a distributed defense mechanism can solve the problem of replay attacks. The trusted transmission mechanism, distributed defense mechanism, and encryption / decryption mechanism work together to ensure the overall security of the blockchain contained in the target feature chain.

10. A method for training complex fusion networks based on blockchain, characterized in that, (1) Extract the target frame from the target surveillance video; (2) Preprocess the target frame and learn a position embedding with consistent dimensions. Optimize it layer by layer in the Transformer encoder to obtain the encoded global feature information and input it into the decoder; (3) The user inputs the task type and selects the target type. The embedding layer quantizes the task into the corresponding embedding and its position embedding according to the adaptive quantization method and inputs it into the decoder. (4) Based on the quantized task, the global feature information of the target is decoded into target feature information related to the specific task using the Transformer decoder; (5) Based on the quantified task, specify the output unit for the obtained target feature information to obtain task-related feature information, send it to the target feature chain for storage, and return it to the user synchronously; (6) The target feature chain is based on the feature information output by the feature decoding module of the smart contract and chain code processing. First, it encrypts these feature information according to the block encryption mechanism of the chain-driven encryption method. Then, the encrypted information is packaged into information storage transactions for transmission according to the trusted transmission mechanism of the chain-driven encryption algorithm. It also relies on the distributed defense mechanism of the chain-driven encryption algorithm to overcome the replay attack problem in blockchain technology. (7) After receiving the information storage transaction, the blockchain consensus algorithm based on the hash graph sorts the transactions according to consensus, and the blockchain node election mechanism based on reputation selects a leader node, packages the transaction into a block, stores it in the blockchain, and completes the information storage transaction process. (8) If the user specifies that the target’s historical feature information previously stored in the blockchain needs to be obtained, a Boolean flag is sent to the decoder. When the Boolean flag is true, the decoder will send the output feature vector to the target feature chain separately. (9) The target feature chain packages the feature vector output by the feature decoding module based on the smart contract and chain code processing into a query transaction according to the trusted transmission mechanism of the chain-driven encryption algorithm, and completes the transaction based on the blockchain consensus algorithm of hash graph and the blockchain node election mechanism based on reputation. (10) A node in the blockchain that wants to respond to a query transaction obtains the corresponding feature vector from the block corresponding to the query transaction, calculates the cosine similarity between the feature vector and its own previously stored feature vectors based on the smart contract and chaincode, and determines whether the target corresponding to the encrypted information deposited by the money has already been stored in the blockchain; if not, it does not respond to the query transaction initiating node; if so, it initiates a confirmation transaction to the query node and completes the transaction based on the blockchain consensus algorithm of hash graph and the blockchain node election mechanism based on reputation. (11) After the transaction is confirmed, the node in the blockchain that needs to respond to the query transaction extracts the encrypted information that has been stored in the blockchain from the target feature chain through the blockchain interaction module, and decrypts the encrypted information according to the smart contract and chain-driven encryption algorithm, packages it into an information transmission transaction, and completes the transaction based on the blockchain consensus algorithm based on hash graph and the blockchain node election mechanism based on reputation, and returns the decrypted information to the decoder in the target visual computing network, and finally returns it to the user.

11. An electronic device, characterized in that, Including processor and memory, of which; Memory, used to store computer programs; A processor for executing a computer program stored in memory to implement the system of any one of claims 1 to 9, or the method of claim 10.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the system according to any one of claims 1 to 9, or the method according to claim 10.

Images