Data collaborative processing system and method based on blockchain

Abstract

The present application relates to a data collaborative processing system and method based on blockchain. The method includes: extracting a food status image and a food status text description of the detected food from the first block and the second block of the blockchain network, and using image analysis technology based on computer vision and a text processing algorithm based on natural language processing to perform multi-scale state analysis of the food status image and semantic encoding of the food status text description, thereby automatically determining whether there is a bias between the food status text description and the food status image based on the comparison between the language deviation between the semantic description features obtained by decoding the food status and the semantic features of the food status text description and a preset threshold. In this way, through blockchain technology, it can be ensured that the food status data cannot be tampered with once it is on the chain, maintaining the integrity and originality of the data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically, to a blockchain-based data collaborative processing system and method. Background Art

[0002] Traditional food status verification methods mainly rely on manual inspection and paper records, which are not only inefficient but also easily affected by human factors, leading to misjudgments and omissions. Specifically, paper records are inconvenient to preserve and consult, and over time, they may also face problems such as damage, loss, or blurred handwriting, affecting the integrity and traceability of information. In addition, the manual inspection and recording process lacks an effective supervision and verification mechanism and is easily exploited by fraudulent activities, such as record tampering or information concealment, thereby increasing food safety risks and supply chain opacity. These problems not only harm consumer rights, but may also cause damage to corporate reputation and economic losses.

[0003] Therefore, an optimized data processing system is desired, which can utilize artificial intelligence technology to collaboratively analyze food state images and food state descriptions, so as to determine whether there is a bias between the two to perform food state detection. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present application provides a data collaborative processing system based on blockchain, the system comprising: a food state image text description extraction module, used to extract a food state image and a food state text description of the detected food from a first block and a second block of the blockchain network; a food state multi-scale feature extraction module, used to input the food state image into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map; a food state multi-scale salient perception module, used to input the food state shallow feature map and the food state semantic feature map into a salient joint perception module based on an attention mechanism to obtain a food state multi-scale joint perception salient feature map; A food status semantic description decoding module is used to decode the food status multi-scale joint perception salient feature map based on the food status semantic description to obtain a food status semantic description decoding feature vector; a food status text description semantic encoding module is used to semantically encode the food status text description to obtain a food status text description semantic encoding feature vector; a food status semantic offset calculation module is used to calculate the text semantic offset between the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector; a verification result generation module is used to generate a verification result based on the comparison between the text semantic offset and a preset threshold.

[0006] Optionally, the food state multi-scale feature extraction module is used to: input the food state image into a food state multi-scale image feature extractor based on a pyramid network to obtain the food state shallow feature map and the food state semantic feature map.

[0007] Optionally, the food state multi-scale salient perception module includes: a feature shape reshaping unit, used to perform feature shape reshaping on the food state shallow feature map and the food state semantic feature map to obtain a food state shallow feature shape reshaping matrix and a food state semantic feature shape reshaping matrix; a food state shallow-semantic channel-by-channel interactive perception unit, used to input the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a food state shallow-semantic dependency matrix and a food state semantic-shallow dependency matrix; a random dropout unit, used to input the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix into a random dropout module to obtain a pruned food state shallow-semantic dependency matrix and a pruned food state semantic-shallow dependency matrix; a semantic feature A feature matrix calculation unit is used to perform matrix multiplication on the pruned food state shallow-semantic dependency matrix and the food state semantic feature shape reshaping matrix to obtain a dependency-optimized food state semantic feature matrix; a shallow feature matrix calculation unit is used to perform matrix multiplication on the pruned food state semantic-shallow dependency matrix and the food state shallow feature shape reshaping matrix to obtain a dependency-optimized food state shallow feature matrix; a feature shape reshaping unit is used to perform feature shape reshaping on the dependency-optimized food state semantic feature matrix and the dependency-optimized food state shallow feature matrix to obtain an optimized food state shallow feature map and an optimized food state semantic feature map; a weighted sum calculation unit is used to calculate the weighted sum of the optimized food state shallow feature map and the optimized food state semantic feature map to obtain the food state multi-scale joint perception salient feature map.

[0008] Optionally, the food status shallow-semantic channel-by-channel interactive perception unit is used to: calculate the matrix product between the transposed matrix of the food status shallow feature shape reshaping matrix and the food status semantic feature shape reshaping matrix to obtain a first related feature matrix; calculate the positional division of the scales of the first related feature matrix and the first related feature matrix to obtain a first related feature modulation matrix; calculate the matrix product between the transposed matrix of the food status semantic feature shape reshaping matrix and the food status shallow feature shape reshaping matrix to obtain a second related feature matrix; calculate the positional division of the scales of the second related feature matrix and the second related feature matrix to obtain a second related feature modulation matrix; input the first related feature modulation matrix and the second related feature modulation matrix into the Softmax function respectively to obtain the food status shallow-semantic dependency matrix and the food status semantic-shallow dependency matrix.

[0009] Optionally, the food state semantic description decoding module is used to: input the food state multi-scale joint perceptual salient feature map into a decoder-based food state semantic descriptor to obtain the food state semantic description decoding feature vector.

[0010] Optionally, the food status text description semantic coding module is used to: perform word segmentation processing on the food status text description and then input the food status semantic coding including a word embedding layer to obtain a semantic coding feature vector of the food status text description.

[0011] Optionally, the food state semantic deviation degree calculation module includes: a position difference calculation unit, which is used to calculate the position difference between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector to obtain the food state semantic description decoding-text difference feature vector; a covariance matrix calculation unit, which is used to calculate the covariance matrix between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector, and calculate the inverse of the covariance matrix to obtain the food state semantic description decoding-text inverse covariance matrix; a decoding-text matching value calculation unit, which is used to calculate the transposed vector of the food state semantic description decoding-text difference feature vector and the product of the food state semantic description decoding-text inverse covariance matrix and the food state semantic description decoding-text difference feature vector to obtain the food state semantic description decoding-text matching value; a text semantic deviation degree calculation unit, which is used to calculate the square root of the food state semantic description decoding-text matching value to obtain the text semantic deviation degree.

[0012] Optionally, the verification result generation module is used to generate the verification result based on a comparison between the text semantic deviation and a preset threshold, wherein the verification result is used to indicate whether there is a bias between the food status text description and the food status image.

[0013] In a second aspect, the present application provides a data collaborative processing method based on blockchain, the method comprising: extracting a food state image and a food state text description of the detected food from the first block and the second block of the blockchain network; inputting the food state image into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map; inputting the food state shallow feature map and the food state semantic feature map into a salient joint perception module based on an attention mechanism to obtain a food state multi-scale joint perception salient feature map; decoding the food state multi-scale joint perception salient feature map based on the food state semantic description to obtain a food state. A semantic description decoding feature vector; semantically encoding the food status text description to obtain a food status text description semantic encoding feature vector; calculating the text semantic deviation between the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector; generating a verification result based on the comparison between the text semantic deviation and a preset threshold; wherein the food status shallow feature map and the food status semantic feature map are input into a salient joint perception module based on an attention mechanism to obtain a food status multi-scale joint perception salient feature map, including: reshaping the food status shallow feature map and the food status semantic feature map to obtain a to the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix; input the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix into the feature channel-by-channel interactive perception module to obtain the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix; input the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix into the random inactivation module to obtain the pruned food state shallow-semantic dependency matrix and the pruned food state semantic-shallow dependency matrix; input the pruned food state shallow-semantic dependency matrix with the food state semantic-shallow dependency matrix The pruned food state semantic-shallow dependency matrix is ​​matrix-multiplied with the food state shallow feature shape reshaping matrix to obtain a dependency-optimized food state shallow feature matrix; the dependency-optimized food state semantic feature matrix and the dependency-optimized food state shallow feature matrix are feature-reshaped to obtain an optimized food state shallow feature map and an optimized food state semantic feature map; the weighted sum of the optimized food state shallow feature map and the optimized food state semantic feature map is calculated to obtain the food state multi-scale joint perception salient feature map.

[0014] Optionally, the food state image is input into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map, including: inputting the food state image into a food state multi-scale image feature extractor based on a pyramid network to obtain the food state shallow feature map and the food state semantic feature map.

[0015] By adopting the above technical solution, the food state image and food state text description of the detected food are extracted from the first block and the second block of the blockchain network, and the image analysis technology based on computer vision and the text processing algorithm of natural language processing are used to perform multi-scale state analysis of the food state image and semantic encoding of the food state text description, so as to automatically judge whether there is a bias between the food state text description and the food state image according to the comparison between the language deviation between the semantic description features obtained by decoding the food state and the semantic features of the food state text description and the preset threshold. In this way, through blockchain technology, it can be ensured that the food state data cannot be tampered with once it is on the chain, and the integrity and originality of the data are maintained.

[0016] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0018] In the attached picture: Figure 1 It is a schematic diagram of the blockchain basic platform system architecture according to an exemplary embodiment.

[0019] Figure 2 It is a schematic diagram of the blockchain cloud management and control platform architecture according to an exemplary embodiment.

[0020] Figure 3 It is a schematic diagram of the overall architecture of a blockchain intelligent R&D platform according to an exemplary embodiment.

[0021] Figure 4 It is a schematic diagram of the overall architecture of blockchain compliance detection according to an exemplary embodiment.

[0022] Figure 5 It is a block diagram of a blockchain-based data collaborative processing system according to an exemplary embodiment.

[0023] Figure 6It is a flowchart of a data collaborative processing method based on blockchain according to an exemplary embodiment.

[0024] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0026] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0032] It should be understood that blockchain technology, as the cutting-edge technology of distributed data storage and transmission, has demonstrated its unique advantages and value. It achieves secure storage and reliable transmission of data by building a decentralized database system with multiple nodes working together. This distributed characteristic gives blockchain technology a high degree of tamper resistance. In the blockchain network, each data block is linked to each other through complex encryption algorithms and unique hash values ​​to form an indestructible data chain. This design ensures that once the data is written to the blockchain, it is almost impossible to tamper with or delete it. Blockchain technology has shown extremely high efficiency in ensuring the authenticity and integrity of data, and effectively prevents the risk of data tampering and forgery.

[0033] The application scope of blockchain technology is wide, not only limited to the financial field, but also shows great potential in many fields such as supply chain management, copyright protection, and identity authentication. Through the characteristics of decentralization, distribution, and immutability, blockchain technology provides a new trust mechanism and data management method for modern society, and promotes the process of digital transformation. With its unique distributed characteristics, immutability, and smart contract functions, blockchain technology provides a solid guarantee for the security and reliability of data, while reducing the risks in transactions and data processing. In the future, with the continuous development and improvement of technology, blockchain technology is expected to play an important role in more fields and promote the process of digital transformation of society.

[0034] As a cutting-edge technology, multimodal data fusion has received extensive attention and in-depth research in recent years. This technology mainly integrates and fuses data from different modalities, such as text, images, voice and video, in an efficient manner, aiming to extract richer and more comprehensive information and knowledge. Through multimodal data fusion, the complementarity and correlation between different modal data can be fully utilized, thereby improving the accuracy and efficiency of information processing. In practical applications, multimodal data fusion technology has shown great potential. First, it can overcome the limitations of single modal data in expression and understanding, making information processing more comprehensive and in-depth. For example, in the field of intelligent recognition, combining image and text information can significantly improve recognition accuracy and stability. Secondly, multimodal data fusion can also help improve the efficiency and experience of human-computer interaction. By fusing multiple modal information such as voice and images, the machine can better understand the user's intentions and needs, thereby providing more accurate and personalized services. As an innovative data processing method, multimodal data fusion technology has broad application value and prospects. It can not only improve the accuracy and efficiency of information processing, but also help promote technological innovation and development in related fields. With the continuous advancement of technology and the continuous expansion of application scenarios, I believe that multimodal data fusion will play a more important role in the future.

[0035] At a time when data security is increasingly valued, blockchain technology, with its unique advantages, provides a solid guarantee for the transmission and storage of multimodal data. Blockchain technology ensures the integrity and authenticity of data during transmission through its decentralized and distributed ledger characteristics, effectively preventing the risk of data tampering or leakage. Blockchain technology uses cryptographic principles to encrypt data, so that data security can be guaranteed even in the most complex network environment. In terms of data sharing and collaboration, blockchain technology also shows great potential. Traditional data sharing methods are often limited by the control of centralized institutions, while the decentralized characteristics of blockchain technology break this limitation, allowing different institutions or organizations to share and collaborate on multimodal data more freely and efficiently. This not only promotes the circulation and utilization of data, but also greatly improves the efficiency and accuracy of data processing. Blockchain technology has shown significant advantages in improving data security, promoting data sharing and collaboration, establishing trust mechanisms, and enhancing privacy protection. With the continuous development and improvement of technology, it is believed that blockchain technology will play a more important role in the field of multimodal data processing.

[0036] The blockchain basic platform includes autonomous and controllable blockchain technology with high performance, high availability, scalability, easy operation and maintenance, multi-compatibility, strong privacy protection, and hybrid storage, providing underlying technical support for blockchain management cloud control platform, intelligent R&D platform, compliance testing platform and other upper-level applications.

[0037] like Figure 1 As shown in the figure, the overall architecture of the blockchain basic platform consists of consensus basic services, network basic services, execution basic services, virtual machine basic services, contract basic services, storage basic services, cryptography basic services, etc. It is mainly responsible for the implementation of the basic functions of the blockchain, mainly including various consensus algorithm services responsible for the consistency of ledgers between blockchain nodes.

[0038] like Figure 2 As shown in the figure, the blockchain cloud management and control platform mainly includes infrastructure, blockchain bottom layer, management-side operation (business services and monitoring and auditing), system security, and enterprise-side blockchain services. The platform's infrastructure is supported by cloud resources provided by the cloud platform. The blockchain basic platform and the entire blockchain cloud management and control platform (management-side operation + enterprise-side blockchain services) are deployed on the cloud; the blockchain bottom layer requires support for a variety of autonomous and controllable blockchain bottom layer systems, and is compatible with the information transmission mechanism, consensus mechanism, storage mechanism, encryption algorithm, execution engine, etc. of each bottom layer; the enterprise-side blockchain service is a carrier for providing blockchain services to enterprises. Enterprises can perform resource management, alliance chain management, resource management, monitoring and operation and maintenance, and general business operations.

[0039] The blockchain intelligent R&D platform adopts a front-end and back-end separation architecture, mainly including the client and the server, and provides rich development resources. Figure 3 As shown in the figure, the blockchain intelligent R&D platform mainly includes contract templates, contract libraries, contract design patterns, project templates and development guides, etc. The client is the direct carrier for users to implement intelligent contract operations, and provides main services such as visual user management, project management, data table management and table operation management.

[0040] The entire blockchain compliance testing platform adopts the mainstream common front-end and back-end separation architecture, which mainly includes the bottom-level module, core interface and top-level module. Figure 4 The top-level module is the direct carrier of service capabilities for users, and mainly provides functions such as user management, contract file management, editor, console, operating environment simulation, and rule base expansion. The core interface is the key to connecting the core interface and the bottom-level module. Users complete the contract verification request through the console, editor, contract file management, operating environment simulation and other functions provided by the top-level module, and then the system interacts with the bottom-level module through the core interface. The bottom-level module mainly includes formal verification, rule base, contract syntax tree, unit testing, report service, data service and other functions.

[0041] Based on this, the present application proposes a data collaborative processing system based on blockchain, which extracts the food state image and food state text description of the detected food from the first block and the second block of the blockchain network, and uses the image analysis technology based on computer vision and the text processing algorithm of natural language processing to perform multi-scale analysis of the state of the food state image and the semantic encoding of the food state text description, so as to automatically judge whether there is a bias between the food state text description and the food state image according to the comparison between the language deviation degree and the preset threshold between the semantic description features obtained by decoding the food state and the semantic features of the food state text description. In this way, through blockchain technology, it can be ensured that the food state data cannot be tampered with once it is on the chain, and the integrity and originality of the data are maintained. At the same time, multi-scale analysis of the food state image and combined with text information can automatically capture the multimodal characteristics of the food and the degree of semantic difference, reduce the need for manual inspection, and greatly improve the accuracy and intelligence of food state verification.

[0042] Figure 5 is a block diagram of a data collaborative processing system based on blockchain according to an exemplary embodiment. Figure 5 As shown, the system 100 includes: a food status image text description extraction module 101, which is used to extract the food status image and food status text description of the detected food from the first block and the second block of the blockchain network.

[0043] The food state multi-scale feature extraction module 102 is used to input the food state image into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map.

[0044] The food state multi-scale salient perception module 103 is used to input the food state shallow feature map and the food state semantic feature map into a salient joint perception module based on the attention mechanism to obtain a food state multi-scale joint perception salient feature map.

[0045] The food state semantic description decoding module 104 is used to perform decoding processing on the food state multi-scale joint perceptual salient feature map based on the food state semantic description to obtain a food state semantic description decoding feature vector.

[0046] The food status text description semantic coding module 105 is used to perform semantic coding on the food status text description to obtain a food status text description semantic coding feature vector.

[0047] The food state semantic deviation calculation module 106 is used to calculate the text semantic deviation between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector.

[0048] The verification result generating module 107 is used to generate a verification result based on the comparison between the text semantic deviation degree and a preset threshold.

[0049] Specifically, in the technical solution of the present application, first, the food state image and the food state text description of the detected food are extracted from the first block and the second block of the blockchain network. Next, considering that the food state image contains feature information about different scales of food state, such as shallow texture, color and deep semantic information. Therefore, in the technical solution of the present application, the food state image is input into the food state multi-scale image feature extractor based on the pyramid network to obtain the food state shallow feature map and the food state semantic feature map. In particular, the essence of the food state multi-scale image feature extractor based on the pyramid network here is the pyramid network.

[0050] In one embodiment of the present application, the food state multi-scale feature extraction module is used to: input the food state image into a food state multi-scale image feature extractor based on a pyramid network to obtain the food state shallow feature map and the food state semantic feature map.

[0051] Then, considering that the food state shallow feature map and the food state semantic feature map respectively represent the feature information of two different scales of the food state, in order to be able to more accurately and comprehensively identify the overall state characteristics of the food, especially when the subtle differences in the food state are difficult to identify by traditional methods, in the technical solution of the present application, the food state shallow feature map and the food state semantic feature map are input into the salient joint perception module based on the attention mechanism to obtain the multi-scale joint perception salient feature map of the food state. It is worth mentioning that the salient joint perception module based on the attention mechanism strengthens the features of each scale by using dual interactive attention, thereby improving the overall feature expression ability. Specifically, first, by reshaping the food state shallow feature map and the food state semantic feature map to obtain the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix, a basis is provided for the subsequent interaction between the two. In particular, the reshaping process ensures that the dimension and shape of the feature map match the input specification of the deep learning model, while retaining the spatial structure of the feature and the integrity of the channel information. Next, the dependency between the two reshaped feature matrices is calculated separately through the feature channel-by-channel interaction perception module to reveal the intrinsic connection and mutual influence between different features on food status, which is crucial to understanding the complexity of the data, thereby generating the food status shallow-semantic dependency matrix and the food status semantic-shallow dependency matrix, providing accurate data for subsequent attention-based optimization. Then, the above two dependency matrices are input into the random dropout module for pruning to remove those feature channels that contribute little or are redundant to the model performance, thereby improving its generalization ability on new data. It should be understood that the random dropout technique selectively ignores or suppresses feature interactions through a randomized method, thereby enhancing the stability of the model in the face of data perturbations. This method of dropout pruning helps to avoid the model's excessive dependence on a specific feature combination and reduce its sensitivity to noise or outliers in the training data. Furthermore, the pruned features are matrix-multiplied with the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix respectively to strengthen the role of important or more discriminative features in the model to obtain the dependency optimized food state semantic feature matrix and the dependency optimized food state shallow feature matrix. Subsequently, the optimized features are reshaped to restore them to the form of feature maps and weighted fusion is performed to obtain the final food state multi-scale joint perception significant feature map. In this way, the model's ability to express features can be enhanced, so that the model can more accurately capture the key information of food status, thereby integrating multi-scale food status information to provide richer feature representation.

[0052] In one embodiment of the present application, the food state multi-scale salient perception module includes: a feature shape reshaping unit, which is used to perform feature shape reshaping on the food state shallow feature map and the food state semantic feature map to obtain a food state shallow feature shape reshaping matrix and a food state semantic feature shape reshaping matrix; a food state shallow-semantic channel-by-channel interactive perception unit, which is used to input the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a food state shallow-semantic dependency matrix and a food state semantic-shallow dependency matrix; a random inactivation unit, which is used to input the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix into a random inactivation module to obtain a pruned food state shallow-semantic dependency matrix and a pruned food state semantic-shallow dependency matrix ; a semantic feature matrix calculation unit, used for matrix multiplying the pruned food state shallow-semantic dependency matrix with the food state semantic feature shape reshaping matrix to obtain a dependency-optimized food state semantic feature matrix; a shallow feature matrix calculation unit, used for matrix multiplying the pruned food state semantic-shallow dependency matrix with the food state shallow feature shape reshaping matrix to obtain a dependency-optimized food state shallow feature matrix; a feature shape reshaping unit, used for feature shape reshaping the dependency-optimized food state semantic feature matrix and the dependency-optimized food state shallow feature matrix to obtain an optimized food state shallow feature map and an optimized food state semantic feature map; a weighted sum calculation unit, used for calculating the weighted sum of the optimized food state shallow feature map and the optimized food state semantic feature map to obtain the food state multi-scale joint perception salient feature map.

[0053] Furthermore, in one embodiment of the present application, the food state shallow-semantic channel-by-channel interactive perception unit is used to: calculate the matrix product between the transposed matrix of the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix to obtain a first related feature matrix; calculate the positional division of the scale of the first related feature matrix and the first related feature matrix to obtain a first related feature modulation matrix; calculate the matrix product between the transposed matrix of the food state semantic feature shape reshaping matrix and the food state shallow feature shape reshaping matrix to obtain a second related feature matrix; calculate the positional division of the scale of the second related feature matrix and the second related feature matrix to obtain a second related feature modulation matrix; input the first related feature modulation matrix and the second related feature modulation matrix into the Softmax function respectively to obtain the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix.

[0054] Specifically, the food state shallow feature map and the food state semantic feature map are input into the salient joint perception module based on the attention mechanism, and processed by the following salient joint perception formula to obtain the food state multi-scale joint perception salient feature map; wherein the salient joint perception formula is: ;in, and are respectively the shallow feature map of the food state and the semantic feature map of the food state, For the reshape operation, and are respectively the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix, is the matrix product, and is the transposed matrix of the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix, and are the scales of the first correlation feature matrix and the second correlation feature matrix, respectively. for function, and are the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix, For pruning, and are respectively the pruned food state shallow-semantic dependency matrix and the pruned food state semantic-shallow dependency matrix, and The dependency relationship is used to optimize the food state semantic feature matrix and the dependency relationship is used to optimize the food state shallow feature matrix, respectively. and are respectively the optimized food state shallow feature map and the optimized food state semantic feature map, and is the weight hyperparameter, It is a multi-scale joint perception salient feature map of the food state.

[0055] Next, considering that the food state multi-scale joint perception salient feature map also contains semantic feature information about the food state, therefore, in order to recover more detailed text information describing the food state from the image features, in the technical solution of the present application, the food state multi-scale joint perception salient feature map is input into the decoder-based food state semantic descriptor to obtain the food state semantic description decoding feature vector. That is, pre-decoding the food state multi-scale joint perception salient feature map by the decoder can extract more advanced and rich semantic information from the feature representation, thereby generating a richer and more accurate description of the food state, thereby providing a basis for the consistency of the subsequent image and text description of the food state.

[0056] In one embodiment of the present application, the food state semantic description decoding module is used to: input the food state multi-scale joint perceptual salient feature map into a decoder-based food state semantic descriptor to obtain the food state semantic description decoding feature vector.

[0057] Then, considering that the food status text description contains semantic information between food status contexts, in the technical solution of the present application, the food status text description is semantically encoded to capture and mine the semantic association information between key contexts, and obtain the semantic encoding feature vector of the food status text description.

[0058] In one embodiment of the present application, the food status text description semantic encoding module is used to: perform word segmentation processing on the food status text description and input the food status semantic encoding including a word embedding layer to obtain a semantic encoding feature vector of the food status text description.

[0059] Furthermore, in order to more accurately calculate the text semantic deviation between the two, so as to evaluate the semantic consistency between the food status text description and the food status image, that is, whether the two are consistent in describing the food status, in the technical solution of the present application, the text semantic deviation between the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector is calculated. Specifically, the text semantic deviation is obtained by respectively calculating the difference vector and the inverse matrix of the covariance matrix between the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector, and based on the difference vector and the inverse matrix.

[0060] In one embodiment of the present application, the food state semantic deviation degree calculation module includes: a position difference calculation unit, which is used to calculate the position difference between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector to obtain the food state semantic description decoding-text difference feature vector; a covariance matrix calculation unit, which is used to calculate the covariance matrix between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector, and calculate the inverse of the covariance matrix to obtain the food state semantic description decoding-text inverse covariance matrix; a decoding-text matching value calculation unit, which is used to calculate the transposed vector of the food state semantic description decoding-text difference feature vector and the product of the food state semantic description decoding-text inverse covariance matrix and the food state semantic description decoding-text difference feature vector to obtain the food state semantic description decoding-text matching value; a text semantic deviation degree calculation unit, which is used to calculate the square root of the food state semantic description decoding-text matching value to obtain the text semantic deviation degree.

[0061] Specifically, the text semantic deviation between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector is calculated using the following semantic deviation calculation formula; wherein the semantic deviation calculation formula is: ;in, and are respectively the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector, is the transpose of a vector, is the covariance matrix between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector, represents the inverse matrix of the covariance matrix between the decoded feature vector of the food state semantic description and the encoded feature vector of the food state text description semantics, is the semantic deviation of the text.

[0062] Then, based on the comparison between the text semantic deviation and the preset threshold, a verification result is generated, and the verification result is used to indicate whether there is a bias between the text description of the food status and the food status image. That is, by comparing the text semantic deviation between the decoding feature vector of the food status semantic description and the semantic encoding feature vector of the food status text description with the preset threshold, it is possible to automatically determine whether there is a bias between the text description of the food status and the food status image. In this way, through blockchain technology, it can be ensured that the food status data cannot be tampered with once it is on the chain, maintaining the integrity and originality of the data. At the same time, multi-scale analysis of the food status image and combining it with text information can automatically detect the micro and macro features of the food, reducing the need for manual inspection and greatly improving the accuracy and intelligence of food status verification.

[0063] In one embodiment of the present application, the verification result generation module is used to: generate the verification result based on the comparison between the text semantic deviation and a preset threshold, and the verification result is used to indicate whether there is a bias between the food status text description and the food status image.

[0064] Preferably, since the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector respectively express the image semantic decoding features of the food status image based on the spatial distribution attention significant joint perception of multi-scale and multi-depth image semantic features and the text semantic encoding features of the food status text description, when calculating the text semantic offset between them to map them to a common semantic space, it is expected to further improve the overall expression consistency based on the distribution of their respective feature sets, thereby improving the accuracy of the calculated text semantic offset.

[0065] In a preferred embodiment, the food state semantic offset calculation module includes: jointly optimizing the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector to obtain an optimized food state semantic decoding-text encoding cascade representation vector. Then, the optimized food state semantic decoding-text encoding cascade representation vector is split into an optimized food state semantic description decoding feature vector and an optimized food state text description semantic encoding feature vector according to the cascade of the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector, and the text semantic offset between the optimized food state semantic description decoding feature vector and the optimized food state text description semantic encoding feature vector is calculated.

[0066] Based on this, the present application jointly optimizes the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector to obtain an optimized food status semantic decoding-text encoding cascade representation vector, including the following steps: cascading the food status semantic description decoding feature vector and the food status text description semantic encoding feature vector to obtain a food status semantic decoding-text encoding cascade representation vector; calculating the sum of the absolute values ​​of each eigenvalue of the food status semantic decoding-text encoding cascade representation vector. The square root of the sum of squares ,Right now: ;in, The first one of the food state semantic decoding-text encoding cascade representation vectors is represented The eigenvalues ​​at the positions, represents the length of the food state semantic decoding-text encoding cascade representation vector, represents the sum of the absolute values ​​of the eigenvalues ​​of the food state semantic decoding-text encoding cascade representation vector, The square root of the sum of squares of the eigenvalues ​​of the food state semantic decoding-text encoding cascade representation vector.

[0067] Subtract each eigenvalue of the food state semantic decoding-text encoding cascade representation vector from After that, take the absolute value and multiply it by the length of the food state semantic decoding-text encoding cascade representation vector and The reciprocal of the product is taken to obtain the first food state semantic decoding-text encoding cascade intermediate feature vector, and each eigenvalue of the food state semantic decoding-text encoding cascade representation vector is subtracted from Then, take the absolute value and multiply it by the length The square root and The reciprocal of the product is taken to obtain the intermediate feature vector of the second food state semantic decoding-text encoding cascade, that is: ;in, The first one of the food state semantic decoding-text encoding cascade representation vectors is represented The eigenvalues ​​at the positions, represents the length of the food state semantic decoding-text encoding cascade representation vector, represents the sum of the absolute values ​​of the eigenvalues ​​of the food state semantic decoding-text encoding cascade representation vector, The square root of the sum of squares of the eigenvalues ​​of the food state semantic decoding-text encoding cascade representation vector, The first intermediate feature vector of the semantic decoding-text encoding cascade of the first food state The eigenvalues ​​at the positions, The first intermediate feature vector of the cascade of semantic decoding and text encoding of the second food state The feature value at each position.

[0068] Finally, the first food state semantic decoding-text encoding cascade intermediate feature vector is subtracted from the dot product of the second food state semantic decoding-text encoding cascade intermediate feature vector and the weighted hyperparameter to obtain the optimized food state semantic decoding-text encoding cascade representation vector.

[0069] Thus, the difference of the characteristic value of the food state semantic decoding-text encoding cascade representation vector relative to the overall feature set of the food state semantic decoding-text encoding cascade representation vector is used as the semantic change intensity information, and the phase-like conversion corresponding to the position-based intensity modulation is performed through the different and modulation representation forms, so that the aggregation enhancement of the semantic change phase perception can improve the axial aggregation receptive field along the feature aggregation direction by performing the spatial translation operation based on alternating stacking under the scale balance of the feature set of the food state semantic decoding-text encoding cascade representation vector, thereby improving the perception effect of the aggregated semantics of the food state semantic decoding-text encoding cascade representation vector on the detail semantic change, so as to improve the overall expression consistency of the food state semantic decoding-text encoding cascade representation vector, and improve the calculation accuracy of the text semantic offset between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector. In this way, through blockchain technology, it can be ensured that the food state data cannot be tampered with once it is on the chain, and the integrity and originality of the data are maintained. At the same time, multi-scale analysis of food status images and combining them with text information can automatically detect the micro and macro characteristics of food, reducing the need for manual inspection and greatly improving the accuracy and intelligence of food status verification.

[0070] In summary, the above scheme is adopted to extract the food state image and food state text description of the detected food from the first block and the second block of the blockchain network, and adopt the image analysis technology based on computer vision and the text processing algorithm of natural language processing to perform the state multi-scale analysis of the food state image and the semantic encoding of the food state text description, so as to automatically judge whether there is a bias between the food state text description and the food state image according to the comparison between the language deviation degree and the preset threshold between the semantic description features obtained by decoding the food state and the semantic features of the food state text description. In this way, through blockchain technology, it can be ensured that the food state data cannot be tampered with once it is on the chain, and the integrity and originality of the data are maintained. At the same time, the multi-scale analysis of the food state image and the combination of text information can automatically capture the multimodal characteristics of the food and the degree of semantic difference, reduce the need for manual inspection, and greatly improve the accuracy and intelligence of food state verification.

[0071] Figure 6 is a flowchart of a data collaborative processing method based on blockchain according to an exemplary embodiment. Figure 6 As shown, the method includes: step S201, extracting a food status image and a food status text description of the detected food from the first block and the second block of the blockchain network.

[0072] Step S202: input the food state image into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map.

[0073] Step S203: input the food state shallow feature map and the food state semantic feature map into a salient joint perception module based on an attention mechanism to obtain a food state multi-scale joint perception salient feature map.

[0074] Step S204: performing decoding processing on the food state multi-scale joint perceptual salient feature map based on the food state semantic description to obtain a food state semantic description decoding feature vector.

[0075] Step S205: semantically encode the food status text description to obtain a semantically encoded feature vector of the food status text description.

[0076] Step S206: Calculate the text semantic deviation between the decoded feature vector of the food state semantic description and the encoded feature vector of the food state text semantic description.

[0077] Step S207: Generate a verification result based on the comparison between the text semantic deviation degree and a preset threshold.

[0078] In one embodiment of the present application, the food state image is input into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map, including: inputting the food state image into a food state multi-scale image feature extractor based on a pyramid network to obtain the food state shallow feature map and the food state semantic feature map.

[0079] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0080] like Figure 7 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0081] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0082] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0083] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0084] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0085] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0086] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0088] The modules involved in the embodiments described in this application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the test parameter acquisition module can also be described as a "module for acquiring device test parameters corresponding to the target device".

[0089] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0090] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] In the application scenario of the blockchain-based data collaborative processing system, first, the food status image and the food status text description of the detected food are extracted from the first block and the second block of the blockchain network; then, the obtained food status image and the food status text description are input into a server deployed with the blockchain-based data collaborative processing algorithm, wherein the server can process the food status image and the food status text description based on the blockchain data collaborative processing algorithm to generate a verification result.

[0092] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0093] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0094] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely exemplary forms of implementation. With respect to the apparatus in the above-mentioned embodiments, the specific manner in which each module performs the operation has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

1. A data collaborative processing system based on blockchain, characterized in that: include: A food state image text description extraction module is used to extract the food state image and food state text description of the detected food from the first block and the second block of the blockchain network; a food state multi-scale feature extraction module is used to input the food state image into the food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map; a food state multi-scale salient perception module is used to input the food state shallow feature map and the food state semantic feature map into a salient joint perception module based on an attention mechanism to obtain a food state multi-scale joint perception salient feature map; a food state semantic description decoding module is used to decode the food state multi-scale joint perception salient feature map based on the food state a food state text description semantic encoding module, used for semantically encoding the food state text description to obtain a food state text description semantic encoding feature vector; a food state semantic deviation degree calculation module, used for calculating the text semantic deviation degree between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector; a verification result generation module, used for generating a verification result based on the comparison between the text semantic deviation degree and a preset threshold; wherein the food state multi-scale saliency perception module includes: a feature shape reshaping unit, used for reshaping the food state shallow feature map and the food state semantic The feature graph is reshaped to obtain a food state shallow feature shape reshaping matrix and a food state semantic feature shape reshaping matrix; a food state shallow-semantic channel-by-channel interactive perception unit is used to input the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a food state shallow-semantic dependency matrix and a food state semantic-shallow dependency matrix; a random inactivation unit is used to input the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix into a random inactivation module to obtain a pruned food state shallow-semantic dependency matrix and a pruned food state semantic-shallow dependency matrix. a semantic feature matrix calculation unit, used for matrix multiplying the pruned food state shallow-semantic dependency matrix with the food state semantic feature shape reshaping matrix to obtain a dependency-optimized food state semantic feature matrix; a shallow feature matrix calculation unit, used for matrix multiplying the pruned food state semantic-shallow dependency matrix with the food state shallow feature shape reshaping matrix to obtain a dependency-optimized food state shallow feature matrix; a feature shape reshaping unit, used for performing feature shape reshaping on the dependency-optimized food state semantic feature matrix and the dependency-optimized food state shallow feature matrix to obtain an optimized food state shallow feature map and an optimized food state semantic feature map;A weighted sum calculation unit, used to calculate the weighted sum of the optimized food state shallow feature map and the optimized food state semantic feature map to obtain the food state multi-scale joint perception significant feature map. ; 2. The blockchain-based data collaborative processing system according to claim 1 is characterized in that: The food state multi-scale feature extraction module is used to: input the food state image into a food state multi-scale image feature extractor based on a pyramid network to obtain the food state shallow feature map and the food state semantic feature map.

3. The blockchain-based data collaborative processing system according to claim 2 is characterized in that: The food state shallow-semantic channel-by-channel interactive perception unit is used to: calculate the matrix product between the transposed matrix of the food state shallow feature shape reshaping matrix and the food state semantic feature shape reshaping matrix to obtain a first correlation feature matrix; calculate the positional division of the scale of the first correlation feature matrix and the first correlation feature matrix to obtain a first correlation feature modulation matrix; Calculating the matrix product between the transposed matrix of the food state semantic feature shape reshaping matrix and the food state shallow feature shape reshaping matrix to obtain a second correlation feature matrix; Calculate the positional division of the scale of the second correlation feature matrix and the second correlation feature matrix to obtain a second correlation feature modulation matrix; input the first correlation feature modulation matrix and the second correlation feature modulation matrix into the Softmax function respectively to obtain the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix.

4. The blockchain-based data collaborative processing system according to claim 3 is characterized in that: The food state semantic description decoding module is used to: input the food state multi-scale joint perception salient feature map into a decoder-based food state semantic descriptor to obtain the food state semantic description decoding feature vector.

5. The blockchain-based data collaborative processing system according to claim 4 is characterized in that: The food status text description semantic coding module is used to: perform word segmentation processing on the food status text description and then input it into the food status semantic coding including the word embedding layer to obtain the food status text description semantic coding feature vector.

6. The blockchain-based data collaborative processing system according to claim 5 is characterized in that: The food state semantic deviation degree calculation module includes: a position difference calculation unit, which is used to calculate the position difference between the food state semantic description decoding feature vector and the food state text description semantic coding feature vector to obtain the food state semantic description decoding-text difference feature vector; a covariance matrix calculation unit, which is used to calculate the covariance matrix between the food state semantic description decoding feature vector and the food state text description semantic coding feature vector, and calculate the inverse of the covariance matrix to obtain the food state semantic description decoding-text inverse covariance matrix; a decoding-text matching value calculation unit, which is used to calculate the transposed vector of the food state semantic description decoding-text difference feature vector and the product of the food state semantic description decoding-text inverse covariance matrix and the food state semantic description decoding-text difference feature vector to obtain the food state semantic description decoding-text matching value; and a text semantic deviation degree calculation unit, which is used to calculate the square root of the food state semantic description decoding-text matching value to obtain the text semantic deviation degree.

7. The blockchain-based data collaborative processing system according to claim 6 is characterized in that: The verification result generation module is used to generate the verification result based on the comparison between the text semantic deviation degree and a preset threshold value, and the verification result is used to indicate whether there is a bias between the food state text description and the food state image.

8. A data collaborative processing method based on blockchain, characterized in that: include: Extract the food state image and food state text description of the detected food from the first block and the second block of the blockchain network; input the food state image into the food state multi-scale image feature extractor to obtain the food state shallow feature map and the food state semantic feature map; input the food state shallow feature map and the food state semantic feature map into the salient joint perception module based on the attention mechanism to obtain the food state multi-scale joint perception salient feature map; perform decoding processing on the food state multi-scale joint perception salient feature map based on the food state semantic description to obtain the food state semantic description decoding feature vector; perform semantic decoding processing on the food state text description encoding to obtain a semantic encoding feature vector of a food state text description; calculating a text semantic deviation between the food state semantic description decoding feature vector and the food state text description semantic encoding feature vector; generating a verification result based on a comparison between the text semantic deviation and a preset threshold; wherein the food state shallow feature map and the food state semantic feature map are input into a salient joint perception module based on an attention mechanism to obtain a multi-scale joint perception salient feature map of the food state, including: reshaping the food state shallow feature map and the food state semantic feature map to obtain a food state shallow feature shape reshaping matrix and a food state shallow feature shape reshaping matrix. The food state semantic feature shape reshaping matrix is ​​input into the feature channel-by-channel interactive perception module to obtain the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix; the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix are input into the random inactivation module to obtain the pruned food state shallow-semantic dependency matrix and the pruned food state semantic-shallow dependency matrix; the pruned food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix are input into the random inactivation module to obtain the pruned food state shallow-semantic dependency matrix and the pruned food state semantic-shallow dependency matrix; the pruned food state shallow-semantic dependency matrix and the food state semantic feature shape reshaping matrix are input into the feature channel-by-channel interactive perception module to obtain the food state shallow-semantic dependency matrix and the food state semantic-shallow dependency matrix The pruned food state semantic-shallow dependency matrix is ​​matrix-multiplied with the food state shallow feature shape reshaping matrix to obtain the dependency optimized food state shallow feature matrix; the dependency optimized food state semantic feature matrix and the dependency optimized food state shallow feature matrix are feature reshaped to obtain an optimized food state shallow feature map and an optimized food state semantic feature map; the weighted sum of the optimized food state shallow feature map and the optimized food state semantic feature map is calculated to obtain the food state multi-scale joint perception salient feature map.

9. The data collaborative processing method based on blockchain according to claim 8 is characterized in that: The food state image is input into a food state multi-scale image feature extractor to obtain a food state shallow feature map and a food state semantic feature map, including: the food state image is input into a food state multi-scale image feature extractor based on a pyramid network to obtain the food state shallow feature map and the food state semantic feature map.

Images