Multimodal data processing method and system based on blockchain technology

Abstract

The present application discloses a multimodal data processing method and system of blockchain technology, which uses artificial intelligence-based text processing and image analysis technology to perform semantic coding of the search query text input and local area semantic analysis of the first candidate image, so as to intelligently determine whether to return the first candidate image based on the query matching results between the search query semantic coding features and the local area semantics of each first candidate image. In this way, the image matching the user's search text input semantics can be automatically output, so that the multimodal data processing and query method can deeply understand the semantic matching between the user query and the multimedia content, thereby providing a more comprehensive query understanding, thereby improving the diversity and intelligence level of multimodal data processing and retrieval.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a multimodal data processing method and system of blockchain technology. Background Art

[0002] With the rapid development of the Internet and mobile communication technology, we are currently experiencing an era of explosive growth in information. Multimedia content, such as images, videos, and texts, has not only increased in quantity, but also has become increasingly diverse in form and source. Users' needs for information retrieval have also become more complex and personalized, and they expect to be able to quickly and accurately find information that is highly relevant to their needs.

[0003] However, when processing user queries, traditional retrieval often faces the problem of single-modal dependence, such as being able to output relevant text information only through text retrieval, which limits their potential in understanding complex queries and providing rich content. Specifically, if only text data is relied upon for input and output of retrieval, and visual information in certain images or video content cannot be matched, this limitation leads to poor performance when faced with the need to retrieve and output image data information. In addition, traditional retrieval is usually based on keywords, but user queries may contain implicit intent or emotional colors. If only simple keyword matching is used to accurately capture them, the context may not be effectively utilized, reducing the relevance and reliability of the retrieval.

[0004] Therefore, an optimized data processing method is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a multimodal data processing method and system of blockchain technology, which uses artificial intelligence-based text processing and image analysis technology to perform semantic coding of the retrieval query text input and local area semantic analysis of the first candidate image, so as to intelligently determine whether to return the first candidate image based on the query matching results between the retrieval query semantic coding features and the local area semantics of each first candidate image. In this way, the image that matches the user's retrieval text input semantics can be automatically output, so that the multimodal data processing and query method can deeply understand the semantic matching between the user query and the multimedia content, so as to provide a more comprehensive query understanding, thereby improving the diversity and intelligence level of multimodal data processing and retrieval.

[0006] According to one aspect of the present application, a multimodal data processing method of blockchain technology is provided, which includes: obtaining a user's retrieval query text input; semantically encoding the retrieval query text input to obtain a retrieval query text input semantic encoding feature vector; extracting a first candidate image from a first block of the blockchain network; performing image segmentation processing on the first candidate image and then extracting semantic features of the candidate image local area to obtain a sequence of semantic feature vectors of the first candidate image local area; inputting the sequence of semantic feature vectors of the first candidate image local area into a feature association optimization network based on endogenous correlation semantic measurement to obtain a sequence of optimized semantic feature vectors of the first candidate image local area; using the semantic encoding feature vector of the retrieval query text input as a query feature vector, and inputting the query feature vector and the sequence of the first candidate image local area optimized semantic feature vector into a fine-grained feature adaptive decoupling query response engine to obtain an input-candidate image query response adaptive matching result representation matrix as an input-candidate image query response adaptive matching result representation feature; based on the input-candidate image query response adaptive matching result representation feature, obtaining a retrieval result, the retrieval result being used to indicate whether to return the first candidate image as a query result.

[0007] According to another aspect of the present application, a multimodal data processing system of blockchain technology is provided, which includes: a retrieval query text acquisition module, used to obtain a user's retrieval query text input; a semantic encoding module, used to semantically encode the retrieval query text input to obtain a retrieval query text input semantic encoding feature vector; a first candidate image extraction module, used to extract a first candidate image from a first block of the blockchain network; a candidate image local area semantic feature extraction module, used to perform image segmentation processing on the first candidate image and then perform semantic feature extraction on the candidate image local area to obtain a sequence of semantic feature vectors of the first candidate image local area; a feature association optimization module, used to convert the sequence input of the first candidate image local area semantic feature vector into a feature vector based on endogenous phase correlation. A feature association optimization network based on relevant semantic metrics is used to obtain a sequence of optimized semantic feature vectors of the local area of ​​the first candidate image; a fine-grained feature adaptive decoupling query response module is used to use the semantic encoding feature vector of the retrieval query text input as the query feature vector, and input the query feature vector and the sequence of the local area optimized semantic feature vector of the first candidate image into the fine-grained feature adaptive decoupling query response engine to obtain an input-candidate image query response adaptive matching result representation matrix as an input-candidate image query response adaptive matching result representation feature; a retrieval result generation module is used to obtain a retrieval result based on the input-candidate image query response adaptive matching result representation feature, and the retrieval result is used to indicate whether to return the first candidate image as the query result.

[0008] Compared with the prior art, the multimodal data processing method and system of blockchain technology provided by the present application uses text processing and image analysis technology based on artificial intelligence to perform semantic coding of the search query text input and semantic analysis of the local area of ​​the first candidate image, so as to intelligently determine whether to return the first candidate image based on the query matching results between the search query semantic coding features and the semantics of the local areas of each first candidate image. In this way, the image matching the user's search text input semantics can be automatically output, so that the multimodal data processing and query method can deeply understand the semantic matching between the user query and the multimedia content, so as to provide a more comprehensive query understanding, thereby improving the diversity and intelligence level of multimodal data processing and retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 This is a flowchart of a multimodal data processing method of blockchain technology according to an embodiment of the present application.

[0011] Figure 2 This is a data flow diagram of a multimodal data processing method based on blockchain technology according to an embodiment of the present application.

[0012] Figure 3 It is a block diagram of a multimodal data processing system based on blockchain technology according to an embodiment of the present application.

[0013] Figure 4 This is a system architecture diagram of the blockchain infrastructure platform according to an embodiment of the present application.

[0014] Figure 5 This is an architecture diagram of the blockchain cloud management and control platform according to an embodiment of the present application.

[0015] Figure 6 This is a diagram of the overall architecture of the blockchain intelligent R&D platform according to an embodiment of the present application.

[0016] Figure 7 This is a diagram of the overall architecture of blockchain compliance detection according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0018] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0019] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0020] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0021] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0022] When processing user queries, traditional retrieval often faces the problem of single-modal dependence, such as being able to output relevant text information only through text retrieval, which limits their potential in understanding complex queries and providing rich content. Specifically, if only text data is relied upon for input and output of retrieval, and the visual information in certain images or video content cannot be matched, this limitation leads to poor performance when faced with the need to retrieve and output image data information. In addition, traditional retrieval is usually based on keywords, but user queries may contain implicit intentions or emotional colors. If only simple keyword matching is used to accurately capture them, the context may not be effectively utilized, reducing the relevance and reliability of the retrieval. Therefore, an optimized data processing method is desired.

[0023] It should be understood that blockchain technology achieves secure storage and reliable transmission of data by building a decentralized database system with multiple nodes working together. This distributed feature gives blockchain technology a high degree of tamper resistance. In the blockchain network, each data block is linked to each other through complex encryption algorithms and unique hash values ​​to form an indestructible data chain. This design ensures that once the data is written to the blockchain, it is almost impossible to tamper with or delete it. Blockchain technology has shown extremely high efficiency in ensuring the authenticity and integrity of data, and effectively prevents the risk of data tampering and forgery.

[0024] In the technical solution of the present application, a multimodal data processing method of blockchain technology is proposed. Figure 1 This is a flowchart of a multimodal data processing method of blockchain technology according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a multimodal data processing method of blockchain technology according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the multimodal data processing method of the blockchain technology of the embodiment of the present application, the steps include: S1, obtaining the user's search query text input; S2, semantically encoding the search query text input to obtain the search query text input semantic encoding feature vector; S3, extracting the first candidate image from the first block of the blockchain network; S4, performing image block processing on the first candidate image and then extracting the semantic features of the local area of ​​the candidate image to obtain a sequence of semantic feature vectors of the local area of ​​the first candidate image; S5, inputting the sequence of semantic feature vectors of the local area of ​​the first candidate image into a feature association optimization network based on endogenous correlation semantic measurement to obtain the first candidate image. Obtain a sequence of optimized semantic feature vectors of the local area of ​​the first candidate image; S6, using the semantic encoding feature vector of the retrieval query text input as the query feature vector, and inputting the query feature vector and the sequence of the optimized semantic feature vectors of the local area of ​​the first candidate image into the fine-grained feature adaptive decoupling query response engine to obtain an input-alternative image query response adaptive matching result representation matrix as an input-alternative image query response adaptive matching result representation feature; S7, based on the input-alternative image query response adaptive matching result representation feature, obtain a retrieval result, and the retrieval result is used to indicate whether to return the first candidate image as a query result.

[0025] In particular, S1 and S2 obtain the search query text input of the user; and semantically encode the search query text input to obtain a search query text input semantic encoding feature vector. Considering that the search query text contains the semantic information of the user's intention and the query content, in order to better understand and analyze the semantic information in the user's search query text, in the technical solution of the present application, the search query text input is semantically encoded to obtain a search query text input semantic encoding feature vector.

[0026] In particular, S3 and S4 extract the first candidate image from the first block of the blockchain network; and perform image segmentation processing on the first candidate image and then perform semantic feature extraction of the local area of ​​the candidate image to obtain a sequence of semantic feature vectors of the local area of ​​the first candidate image. In a specific example of the present application, the first candidate image is segmented and then subjected to a ViT model including an embedding layer to obtain a sequence of semantic feature vectors of the local area of ​​the first candidate image. Considering that different local areas in the first candidate image contain different semantic feature information, and there is a semantic association relationship between the contexts between each local area. The ViT model including an embedding layer can capture the semantic relationship between different image blocks and realize the understanding of each local area. Therefore, in the technical solution of the present application, the first candidate image is segmented and then subjected to a ViT model including an embedding layer to obtain a sequence of semantic feature vectors of the local area of ​​the first candidate image.

[0027] In particular, in S5, the sequence of the first candidate image local area semantic feature vectors is input into a feature association optimization network based on an endogenous correlation semantic metric to obtain a sequence of the first candidate image local area optimized semantic feature vectors. Considering that there are different semantic dependencies and associations based on the entire global semantics between the individual first candidate image local area semantic features in the sequence of the first candidate image local area semantic feature vectors. Therefore, in the technical solution of the present application, the sequence of the first candidate image local area semantic feature vectors is input into a feature association optimization network based on an endogenous correlation semantic metric to obtain a sequence of the first candidate image local area optimized semantic feature vectors. It is worth mentioning that the feature association optimization network based on an endogenous correlation semantic metric is a network that focuses on improving the quality of feature representation by quantifying and optimizing the semantic associations between feature vectors.

[0028] In an embodiment of the present application, the sequence of the first candidate image local region semantic feature vectors is input into a feature association optimization network based on the intrinsic correlation semantic metric to obtain a sequence of the first candidate image local region optimized semantic feature vectors, including: first, calculating the semantic association score vector between any two first candidate image local region semantic feature vectors in the sequence of the first candidate image local region semantic feature vectors to obtain a set of first candidate image local region semantic association score vectors; by calculating the score vector between any two vectors in the sequence of the first candidate image local region semantic feature vectors, the semantic similarity or difference between the two is quantified. Then, based on the set of the first candidate image local region semantic association score vectors, the global representation vector of the intrinsic correlation of the first candidate image local region sequence is determined; the global information is calculated based on the sharing amount of each, so as to integrate and consider the semantic structure information of the entire first candidate image local region semantic feature vector, and provide a basis for subsequent association optimization. Furthermore, based on the global representation vector of the endogenous correlation of the first candidate image local region sequence, the association optimization factor of each first candidate image local region semantic feature vector in the sequence of the first candidate image local region semantic feature vector is calculated to obtain a sequence of first candidate image local region semantic association optimization factors; that is, based on the global information, each first candidate image local region semantic feature vector is subjected to association optimization to obtain an association value, so as to enhance the expression capability of each local region feature vector for the image semantic content, so as to more accurately capture the semantic similarity or difference information between the local features and the global features. Then, the sequence of the first candidate image local region semantic association optimization factors is input into the image processing unit. An activation function is used to obtain a sequence of semantic association optimization weight factors of the local area of ​​the first candidate image; in the technical solution of the present application, each association value is input into the Softmax activation function to convert each factor into a weight form of a probability distribution, so that the local area with a higher association optimization factor obtains a larger weight in the final probability distribution, thereby enhancing the distinguishability of the feature. Subsequently, each first candidate image local area semantic association optimization weight factor in the sequence of the first candidate image local area semantic association optimization weight factor is used as a weight to weight each first candidate image local area semantic feature vector in the sequence of the first candidate image local area semantic feature vector to obtain a sequence of the first candidate image local area optimized semantic feature vector.

[0029] Among them, the process of calculating the semantic association score vector between any two first candidate image local area semantic feature vectors in the sequence of the first candidate image local area semantic feature vectors to obtain a set of first candidate image local area semantic association score vectors includes: cascading any two first candidate image local area semantic feature vectors in the sequence of the first candidate image local area semantic feature vectors to obtain a set of first candidate image local area semantic cascade feature vectors; multiplying each first candidate image local area semantic cascade feature vector in the set of the first candidate image local area semantic cascade feature vectors with the first candidate image semantic weight matrix and then adding them positionally with the first candidate image semantic bias vector to obtain a set of the first candidate image local area semantic association score vectors; and, based on the set of the first candidate image local area semantic association score vectors, determining the global representation vector of the endogenous correlation of the first candidate image local area sequence, including: calculating the mean vector of the set of the first candidate image local area semantic association score vectors as the global representation vector of the endogenous correlation of the first candidate image local area sequence. More specifically, based on the global representation vector of the endogenous correlation of the local area sequence of the first candidate image, the process of calculating the association optimization factor of each first candidate image local area semantic feature vector in the sequence of the local area semantic feature vector of the first candidate image to obtain a sequence of the local area semantic association optimization factors of the first candidate image includes: multiplying the global representation vector of the endogenous correlation of the local area sequence of the first candidate image with the first candidate image global weight matrix to obtain the first candidate image global representation weight vector; multiplying each first candidate image local area semantic feature vector in the sequence of the local area semantic feature vector of the first candidate image with the first candidate image local weight matrix to obtain a set of first candidate image local representation weight vectors; adding each first candidate image local representation weight vector in the set of the first candidate image local representation weight vectors with the first candidate image global representation weight vector and the first candidate image bias vector by position to obtain a set of first candidate image local area semantic association vectors; inputting the set of the first candidate image local area semantic association vectors into the Sigmoid function to obtain a sequence of the local area semantic association optimization factors of the first candidate image.

[0030] In summary, in the above embodiment, the sequence of the first candidate image local area semantic feature vectors is input into a feature association optimization network based on an endogenous correlation semantic metric to obtain a sequence of the first candidate image local area optimized semantic feature vectors, including: inputting the sequence of the first candidate image local area semantic feature vectors into a feature association optimization network based on an endogenous correlation semantic metric, and processing them with the following feature association optimization formula to obtain a sequence of the first candidate image local area optimized semantic feature vectors; wherein the feature aggregation formula is: ;in, is a sequence of semantic feature vectors of the local region of the first candidate image, represents the number of feature vectors in the sequence of semantic feature vectors of the local region of the first candidate image, and are respectively the first candidate image local area semantic feature vectors in the sequence and The first candidate image local region semantic feature vector, For cascade processing operations, is the first candidate image semantic weight matrix, is the first candidate image semantic bias vector, is the first candidate image local region semantic association score vector in the set The first candidate image local region semantic association score vector, is the number of vectors in the set of semantic association score vectors of the local region of the first candidate image, is the global representation vector of the endogenous correlation of the local region sequence of the first candidate image, and are the first candidate image global weight matrix and the first candidate image local weight matrix, respectively. is the first candidate image bias vector, yes function, yes function, The first candidate image local area semantic association optimization weight factor in the sequence The first candidate image local region semantic association optimization weight factor, The first candidate image local area is optimized for the first candidate image local area in the sequence of semantic feature vectors The first candidate image local region optimizes the semantic feature vector.

[0031] In particular, in S6, the semantically encoded feature vector of the retrieval query text input is used as the query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into the fine-grained feature adaptive decoupling query response engine to obtain the input-candidate image query response adaptive matching result representation matrix as the input-candidate image query response adaptive matching result representation feature. In order to be able to better query and match the sequence of the first candidate image local area optimized semantic feature vector based on the retrieval query text input semantic encoding feature vector, thereby improving the accuracy of the query result, in the technical solution of the present application, the semantically encoded feature vector of the retrieval query text input is used as the query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into the fine-grained feature adaptive decoupling query response engine to obtain the input-candidate image query response adaptive matching result representation matrix. It is worth mentioning that the fine-grained feature adaptive decoupling query response engine improves the matching accuracy between the query and library feature vectors by fine-grained decoupling and adaptive query matching of features.

[0032] In an embodiment of the present application, the semantically encoded feature vector of the retrieval query text input is used as a query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into a fine-grained feature adaptive decoupling query response engine to obtain an input-alternative image query response adaptive matching result representation matrix as an input-alternative image query response adaptive matching result representation feature, including: first, fine-grained decoupling of the sequence of the first candidate image local area optimized semantic feature vector is performed to obtain a set of subsequences of the first candidate image local area optimized semantic feature vector, thereby dividing the entire serialization into smaller subsequences to better characterize and capture the local semantic feature information and details in the local area of ​​the first candidate image. Next, each subsequence of the first candidate image local region optimization semantic feature vector in the set of subsequences of the first candidate image local region optimization semantic feature vector is input into a subsequence semantic enhancement encoder based on the Bi-LSTM model to obtain a set of first candidate image local region optimization semantic fine-grained decoupled local association feature vectors; that is, each decoupled subsequence is input into a subsequence semantic enhancement encoder based on the Bi-LSTM model for context encoding between different semantic spans to strengthen the semantic expression between each fine-grained information, so that the obtained fine-grained decoupled local association feature vector has stronger semantic association. Further, the covariance matrix between the query feature vector and each first candidate image local region optimization semantic fine-grained decoupled local association feature vector in the set of the first candidate image local region optimization semantic fine-grained decoupled local association feature vector is calculated to obtain a set of query first candidate image local region response covariance matrices. In particular, the covariance matrix is ​​a statistical tool for measuring the linear relationship and semantic matching degree between the query feature and the first candidate image local region optimization semantic feature. Then, the matrix traces of each query first candidate image local area response covariance matrix in the set of query first candidate image local area response covariance matrices are calculated to obtain a set of query first candidate image local area response trace measurement factors to reflect the overall characteristics between semantics. In particular, the matrix trace is the sum of each element on the matrix diagonal. The set of query first candidate image local area response trace measurement factors is input into the Sigmoid activation function to obtain a set of query first candidate image local area response trace attention weights; here, each query first candidate image local area response trace measurement factor is probabilistically processed to adaptively adjust the contribution between each feature, highlight the key feature information, and obtain the query first candidate image local area response trace attention weights.Finally, using each query first candidate image local area response trace attention weight in the set of query first candidate image local area response trace attention weights as a weight, the weighted sum of the set of query first candidate image local area response covariance matrices is calculated to obtain the input-alternative image query response adaptive matching result representation matrix.

[0033] Among them, calculating the matrix trace of each query first alternative image local area response covariance matrix in the set of query first alternative image local area response covariance matrices to obtain a set of query first alternative image local area response trace measurement factors includes: respectively calculating the sum of each element on the diagonal of each query first alternative image local area response covariance matrix in the set of query first alternative image local area response covariance matrices to obtain the set of query first alternative image local area response trace measurement factors.

[0034] In summary, in the above embodiment, the semantic encoding feature vector of the retrieval query text input is used as the query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into the fine-grained feature adaptive decoupling query response engine to obtain the input-alternative image query response adaptive matching result representation matrix as the input-alternative image query response adaptive matching result representation feature, including: using the semantic encoding feature vector of the retrieval query text input as the query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into the fine-grained feature adaptive decoupling query response engine, and processing is performed with the following query response formula to obtain the input-alternative image query response adaptive matching result representation matrix; wherein, the query response formula is: ;in, each first candidate image local region optimized semantic feature vector in the sequence representing the first candidate image local region optimized semantic feature vector, To decouple operations at a fine-grained level, A set of subsequences representing the optimized semantic feature vectors of the local region of the first candidate image, A subsequence of each first candidate image local region optimized semantic feature vector in a set of subsequences of the first candidate image local region optimized semantic feature vectors, represents the Bi-LSTM encoder, is each first candidate image local region optimized semantic fine-grained decoupling local association feature vector in the set of the first candidate image local region optimized semantic fine-grained decoupling local association feature vectors, is the query feature vector, represents the operation of calculating the covariance matrix, is each query first candidate image local area response covariance matrix in the set of query first candidate image local area response covariance matrices, represents the calculation matrix trace, is each query first candidate image local area response trace metric factor in the set of query first candidate image local area response trace metric factors, yes function, is the attention weight of each local region response trace of the first candidate image in the set of attention weights of the local region response trace of the first candidate image in the query, is the input-candidate image query response adaptive matching result representation matrix.

[0035] In particular, the S7 obtains a retrieval result based on the input-alternative image query response adaptive matching result representation feature, and the retrieval result is used to indicate whether to return the first alternative image as a query result. In an embodiment of the present application, the input-alternative image query response adaptive matching result representation matrix is ​​input into a retrieval result generator based on a classifier to obtain the retrieval result. That is, the input-alternative image query response adaptive matching result representation feature obtained by adaptively matching the query response between the sequence of the semantic encoding feature vector of the retrieval query text input and the local area optimized semantic feature vector of the first alternative image is used for classification processing to intelligently determine whether to return the first alternative image. In this way, the image matching the user's retrieval text input semantics can be automatically output, so that the multimodal data processing and query method can deeply understand the semantic matching between the user query and the multimedia content, thereby providing a more comprehensive query understanding, thereby improving the diversity and intelligence level of multimodal data processing and retrieval.

[0036] Preferably, since the sequence of the retrieval query text input semantic encoding feature vector and the first candidate image local area optimized semantic feature vector respectively expresses the text semantic encoding features of the retrieval query text and the local source image semantic context-associated image semantic features of the first candidate image, the global image semantic feature domain associated features based on the image semantic distribution related measure under the local image semantic feature domain, when performing fine-grained feature adaptive decoupling query response on it, considering the fine-grained correspondence differences of cross-modal data in different feature domains, it is expected to further improve its overall expression effect based on differentiated local distribution, thereby improving the accuracy of the retrieval results obtained by the retrieval result generator based on the classifier of the input-alternative image query response adaptive matching result representation matrix input.

[0037] In a preferred embodiment, inputting the input-alternative image query response adaptive matching result representation matrix into a retrieval result generator based on a classifier to obtain retrieval results includes: optimizing the input-alternative image query response adaptive matching result representation matrix to obtain an optimized input-alternative image query response adaptive matching result representation matrix; inputting the optimized input-alternative image query response adaptive matching result representation matrix into a retrieval result generator based on a classifier to obtain retrieval results.

[0038] Based on this, the present application optimizes the input-alternative image query response adaptive matching result representation matrix to obtain an optimized input-alternative image query response adaptive matching result representation matrix, including the following steps: calculating the sum of the absolute values ​​of each eigenvalue of the input-alternative image query response adaptive matching result representation matrix The square root of the sum of squares ,Right now: ;in, The first one represents the input-candidate image query response adaptive matching result representation matrix The eigenvalues ​​at the positions, The feature matrix scale representing the input-candidate image query response adaptive matching result representation matrix, represents the sum of the absolute values ​​of the eigenvalues ​​of the input-candidate image query response adaptive matching result representation matrix, The square root of the sum of squares of the eigenvalues ​​of the input-candidate image query response adaptive matching result representation matrix is ​​represented.

[0039] Subtract each eigenvalue of the input-candidate image query response adaptive matching result representation matrix from After that, take the absolute value and multiply it by the feature matrix scale of the input-alternative image query response adaptive matching result representation matrix and The reciprocal of the product is taken to obtain the first input-candidate image query response adaptive matching result intermediate feature matrix, and each eigenvalue of the input-candidate image query response adaptive matching result representation matrix is ​​subtracted from After that, take the absolute value and multiply it by the feature matrix scale The square root of The inverse of the product is taken to obtain the intermediate feature matrix of the second input-alternative image query response adaptive matching result, that is: ;in, The first one represents the input-candidate image query response adaptive matching result representation matrix The eigenvalues ​​at the positions, The feature matrix scale representing the input-candidate image query response adaptive matching result representation matrix, represents the sum of the absolute values ​​of the eigenvalues ​​of the input-candidate image query response adaptive matching result representation matrix, represents the square root of the sum of squares of the eigenvalues ​​of the input-candidate image query response adaptive matching result representation matrix, represents the first input-candidate image query response adaptive matching result intermediate feature matrix The eigenvalues ​​at the positions, represents the intermediate feature matrix of the second input-alternative image query response adaptive matching result The feature value at each position.

[0040] Finally, the first input-alternative image query response adaptive matching result intermediate feature matrix is ​​subtracted from the dot product of the second input-alternative image query response adaptive matching result intermediate feature matrix and the weighted hyperparameter to obtain an optimized input-alternative image query response adaptive matching result representation matrix.

[0041] Thus, by taking the difference of the difference and modulation representation of the eigenvalue of the input-candidate image query response adaptive matching result representation matrix relative to the overall feature set of the input-candidate image query response adaptive matching result representation matrix as the semantic change intensity information, and performing the phase-like transformation corresponding to the position-based intensity modulation through the different and modulation representation forms, so as to perform the spatial translation operation based on alternating stacking under the feature set scale balance of the input-candidate image query response adaptive matching result representation matrix, so as to make the aggregation enhancement of the semantic change phase perception able to improve the axial aggregation receptive field along the feature aggregation direction, thereby improving the perception effect of the aggregated semantics of the input-candidate image query response adaptive matching result representation matrix on the detail semantic change, so as to improve the overall expression effect of the input-candidate image query response adaptive matching result representation matrix, and improve the accuracy of the retrieval results obtained by the retrieval result generator based on the classifier. In this way, the image matching the user's retrieval text input semantics can be automatically outputted, so that the multimodal data processing and query method can deeply understand the semantic matching between the user query and the multimedia content, thereby providing a more comprehensive query understanding, thereby improving the diversity and intelligence level of multimodal data processing and retrieval.

[0042] In summary, the multimodal data processing method of the blockchain technology according to the embodiment of the present application is explained, which uses artificial intelligence-based text processing and image analysis technology to perform semantic coding of the search query text input and local area semantic analysis of the first candidate image, so as to intelligently determine whether to return the first candidate image based on the query matching results between the search query semantic coding features and the local area semantics of each first candidate image. In this way, the image that matches the user's search text input semantics can be automatically output, so that the multimodal data processing and query method can deeply understand the semantic matching between the user query and the multimedia content, thereby providing a more comprehensive query understanding, thereby improving the diversity and intelligence level of multimodal data processing and retrieval.

[0043] Furthermore, a multimodal data processing system of blockchain technology is also provided.

[0044] Figure 3 : is a block diagram of a multimodal data processing system of blockchain technology according to an embodiment of the present application. Figure 3 As shown, the multimodal data processing system 300 of the blockchain technology according to the embodiment of the present application includes: a search query text acquisition module 310, which is used to obtain the user's search query text input; a semantic encoding module 320, which is used to semantically encode the search query text input to obtain a search query text input semantic encoding feature vector; a first candidate image extraction module 330, which is used to extract a first candidate image from a first block of the blockchain network; a candidate image local area semantic feature extraction module 340, which is used to perform image segmentation processing on the first candidate image and then perform candidate image local area semantic feature extraction to obtain a sequence of first candidate image local area semantic feature vectors; a feature association optimization module 350, which is used to input the sequence of the first candidate image local area semantic feature vectors based on A feature association optimization network of endogenous correlation semantic measurement is used to obtain a sequence of optimized semantic feature vectors of the local area of ​​the first candidate image; a fine-grained feature adaptive decoupling query response module 360 ​​is used to use the semantic encoding feature vector of the retrieval query text input as the query feature vector, and input the query feature vector and the sequence of the first candidate image local area optimized semantic feature vector into the fine-grained feature adaptive decoupling query response engine to obtain an input-candidate image query response adaptive matching result representation matrix as an input-candidate image query response adaptive matching result representation feature; a retrieval result generation module 370 is used to obtain a retrieval result based on the input-candidate image query response adaptive matching result representation feature, and the retrieval result is used to indicate whether to return the first candidate image as a query result.

[0045] As described above, the multimodal data processing system 300 of the blockchain technology according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a multimodal data processing algorithm of the blockchain technology. In one possible implementation, the multimodal data processing system 300 of the blockchain technology according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the multimodal data processing system 300 of the blockchain technology can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the multimodal data processing system 300 of the blockchain technology can also be one of the many hardware modules of the wireless terminal.

[0046] Alternatively, in another example, the multimodal data processing system 300 of the blockchain technology and the wireless terminal may also be separate devices, and the multimodal data processing system 300 of the blockchain technology may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0047] In the example of this application, the blockchain basic platform includes domestically produced, independent and controllable blockchain technology with high performance, high availability, scalability, easy operation and maintenance, multi-compatibility, strong privacy protection, and hybrid storage, providing underlying technical support for blockchain management cloud control platform, intelligent R&D platform, compliance testing platform and other upper-level applications.

[0048] 1. Blockchain basic platform: Figure 4 As shown in the figure, the overall architecture of the blockchain basic platform consists of consensus modules, network service modules, smart contract modules, and other services. It is mainly responsible for the implementation of the basic functions of the blockchain, including various consensus algorithm services responsible for the consistency of ledgers between blockchain nodes.

[0049] 2. Blockchain cloud management and control platform: This project builds a blockchain cloud management and control sub-platform, creating a two-tier management model for management and users, mainly including the blockchain cloud management and control subsystem on the government side and the blockchain operation management subsystem on the enterprise side. The management department plays the role of a service-oriented government by providing blockchain business service capabilities; enterprises directly obtain standard blockchain service capabilities through the operating system, which can help enterprises focus on core business development.

[0050] like Figure 5 As shown in the figure, the blockchain cloud control platform mainly includes infrastructure, blockchain bottom layer, government-side operation management (business services and monitoring audit), system security, and enterprise-side blockchain services. The blockchain basic platform and the entire blockchain cloud control platform (government-side operation management + enterprise-side blockchain services) are deployed on the cloud.

[0051] The underlying blockchain system is required to support a variety of domestically produced, independently controllable underlying blockchain systems, and be compatible with the underlying information transmission mechanism, consensus mechanism, storage mechanism, encryption algorithm, execution engine, etc. The enterprise-side blockchain service is a carrier for providing blockchain services to enterprises. Enterprises can perform resource management, alliance chain management, resource management, monitoring and operation, and general business operations.

[0052] 3. Blockchain intelligent R&D platform: The blockchain intelligent R&D platform adopts a front-end and back-end separation architecture, mainly including the client and the server, and provides rich development resources.

[0053] like Figure 6 As shown in the figure, the blockchain intelligent R&D platform mainly includes contract templates, contract libraries, contract design patterns, project templates and development guides, etc. The client is the direct carrier for users to implement intelligent contract operations, and provides main services such as visual user management, project management, data table management and table operation management.

[0054] 4. Blockchain compliance testing platform: The entire blockchain compliance testing platform adopts the mainstream and general front-end and back-end separation architecture, which mainly includes underlying modules, core interfaces and top-level modules.

[0055] like Figure 7 As shown in the figure, the top-level module is the direct carrier of service capabilities for users, mainly providing functions such as user management, contract file management, editor, console, operating environment simulation and rule base expansion. The core interface is the key to connecting the core interface and the bottom-level module. Users complete the contract verification request through the console, editor, contract file management, operating environment simulation and other functions provided by the top-level module, and then the system interacts with the bottom-level module through the core interface. The bottom-level module mainly includes formal verification, rule base, contract syntax tree, unit testing, report service, data service and other functions.

[0056] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal data processing method of blockchain technology, characterized in that: The method comprises: obtaining a search query text input by a user; semantically encoding the search query text input to obtain a search query text input semantic encoding feature vector; extracting a first candidate image from a first block of a blockchain network; performing image segmentation processing on the first candidate image and then extracting semantic features of a local region of the candidate image to obtain a sequence of semantic feature vectors of a local region of the first candidate image; inputting the sequence of semantic feature vectors of the local region of the first candidate image into a feature association optimization network based on an endogenous correlation semantic metric to obtain a sequence of optimized semantic feature vectors of the local region of the first candidate image; using the semantic encoding feature vector of the search query text input as a query feature vector, and inputting the query feature vector and the sequence of the optimized semantic feature vectors of the local region of the first candidate image into a fine-grained feature adaptive decoupling query response engine to obtain an input-candidate image query response adaptive matching result representation matrix as an input-candidate image query response adaptive matching result representation feature; Obtaining a retrieval result based on the input-candidate image query response adaptive matching result representation feature, wherein the retrieval result is used to indicate whether to return the first candidate image as a query result; Among them, the semantic encoding feature vector of the retrieval query text input is used as the query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into the fine-grained feature adaptive decoupling query response engine to obtain the input-alternative image query response adaptive matching result representation matrix, including: fine-grained decoupling of the sequence of the first candidate image local area optimized semantic feature vector to obtain a set of subsequences of the first candidate image local area optimized semantic feature vector; each subsequence of the first candidate image local area optimized semantic feature vector in the set of subsequences of the first candidate image local area optimized semantic feature vector is input into the subsequence semantic reinforcement encoder based on the Bi-LSTM model to obtain a set of first candidate image local area optimized semantic fine-grained decoupling local association feature vectors; calculating the query feature vector and the first candidate image local area optimized semantic fine-grained decoupling local association feature vector. The covariance matrix between the semantic fine-grained decoupled local correlation feature vectors of each first candidate image local area in the set is optimized to obtain a set of query first candidate image local area response covariance matrices; the matrix traces of each query first candidate image local area response covariance matrix in the set of query first candidate image local area response covariance matrices are calculated to obtain a set of query first candidate image local area response trace measurement factors; the set of query first candidate image local area response trace measurement factors is input into a Sigmoid activation function to obtain a set of query first candidate image local area response trace attention weights; using each query first candidate image local area response trace attention weight in the set of query first candidate image local area response trace attention weights as a weight, the weighted sum of the set of query first candidate image local area response covariance matrices is calculated to obtain the input-alternative image query response adaptive matching result representation matrix.

2. The multimodal data processing method of blockchain technology according to claim 1 is characterized in that: The method comprises: performing image segmentation processing on the first candidate image and then extracting semantic features of local regions of the candidate image to obtain a sequence of semantic feature vectors of local regions of the first candidate image, comprising: performing image segmentation processing on the first candidate image and then extracting semantic features of local regions of the candidate image through a ViT model including an embedding layer to obtain a sequence of semantic feature vectors of local regions of the first candidate image.

3. The multimodal data processing method of blockchain technology according to claim 2 is characterized in that: Inputting the sequence of the first candidate image local area semantic feature vectors into a feature association optimization network based on the endogenous correlation semantic metric to obtain a sequence of the first candidate image local area optimized semantic feature vectors, including: calculating the semantic association score vector between any two first candidate image local area semantic feature vectors in the sequence of the first candidate image local area semantic feature vectors to obtain a set of first candidate image local area semantic association score vectors; determining the endogenous correlation global representation vector of the first candidate image local area sequence based on the set of the first candidate image local area semantic association score vectors; calculating the association optimization factor of each first candidate image local area semantic feature vector in the sequence of the first candidate image local area semantic feature vector based on the endogenous correlation global representation vector of the first candidate image local area sequence to obtain a sequence of first candidate image local area semantic association optimization factors; inputting the sequence of the first candidate image local area semantic association optimization factors into a feature association optimization network based on the endogenous correlation semantic metric of the first candidate image local area semantic feature vectors. An activation function is provided to obtain a sequence of first candidate image local area semantic association optimization weight factors; each first candidate image local area semantic association optimization weight factor in the sequence of the first candidate image local area semantic association optimization weight factors is used as a weight, and each first candidate image local area semantic feature vector in the sequence of the first candidate image local area semantic feature vector is weighted respectively to obtain a sequence of the first candidate image local area optimized semantic feature vectors.

4. The multimodal data processing method of blockchain technology according to claim 3 is characterized in that: Calculate the semantic association score vector between any two first candidate image local area semantic feature vectors in the sequence of the first candidate image local area semantic feature vectors to obtain a set of first candidate image local area semantic association score vectors, including: cascading any two first candidate image local area semantic feature vectors in the sequence of the first candidate image local area semantic feature vectors to obtain a set of first candidate image local area semantic cascade feature vectors; multiply each first candidate image local area semantic cascade feature vector in the set of the first candidate image local area semantic cascade feature vectors by the first candidate image semantic weight matrix, and then positionally add the first candidate image semantic bias vector to obtain the set of the first candidate image local area semantic association score vectors.

5. The multimodal data processing method of blockchain technology according to claim 4 is characterized in that: Based on the set of semantic association score vectors of local regions of the first candidate images, a global representation vector of endogenous correlation of a sequence of local regions of the first candidate images is determined, including: calculating a mean vector of the set of semantic association score vectors of local regions of the first candidate images as a global representation vector of endogenous correlation of a sequence of local regions of the first candidate images.

6. The multimodal data processing method of blockchain technology according to claim 5 is characterized in that: Based on the global representation vector of the endogenous correlation of the local area sequence of the first candidate image, the association optimization factor of each first candidate image local area semantic feature vector in the sequence of the local area semantic feature vector of the first candidate image is calculated to obtain a sequence of the local area semantic association optimization factors of the first candidate image, including: multiplying the global representation vector of the endogenous correlation of the local area sequence of the first candidate image with the first candidate image global weight matrix to obtain the first candidate image global representation weight vector; multiplying each first candidate image local area semantic feature vector in the sequence of the local area semantic feature vector of the first candidate image with the first candidate image local weight matrix to obtain a set of first candidate image local representation weight vectors; adding each first candidate image local representation weight vector in the set of the first candidate image local representation weight vectors to the first candidate image global representation weight vector and the first candidate image bias vector by position to obtain a set of first candidate image local area semantic association vectors; inputting the set of the first candidate image local area semantic association vectors into a Sigmoid function to obtain a sequence of the local area semantic association optimization factors of the first candidate image.

7. The multimodal data processing method of blockchain technology according to claim 6 is characterized in that: Calculating the matrix traces of each query first alternative image local area response covariance matrix in the set of query first alternative image local area response covariance matrices to obtain a set of query first alternative image local area response trace measurement factors, including: respectively calculating the sum of each element on the diagonal of each query first alternative image local area response covariance matrix in the set of query first alternative image local area response covariance matrix to obtain the set of query first alternative image local area response trace measurement factors.

8. The multimodal data processing method of blockchain technology according to claim 7 is characterized in that: Obtaining a retrieval result based on the input-candidate image query response adaptive matching result representation feature includes: inputting the input-candidate image query response adaptive matching result representation matrix into a classifier-based retrieval result generator to obtain the retrieval result.

9. A multimodal data processing system based on blockchain technology, characterized in that: include: A search query text acquisition module is used to obtain the search query text input by the user; A semantic encoding module, used for semantically encoding the search query text input to obtain a semantic encoding feature vector of the search query text input; A first candidate image extraction module is used to extract a first candidate image from a first block of a blockchain network; a candidate image local area semantic feature extraction module is used to extract the semantic features of the candidate image local area after performing image segmentation processing on the first candidate image to obtain a sequence of semantic feature vectors of the first candidate image local area; A feature association optimization module, used for inputting the sequence of semantic feature vectors of the local region of the first candidate image into a feature association optimization network based on endogenous correlation semantic measurement to obtain a sequence of optimized semantic feature vectors of the local region of the first candidate image; A fine-grained feature adaptive decoupled query response module is used to use the semantic encoding feature vector of the retrieval query text input as the query feature vector, and input the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector into the fine-grained feature adaptive decoupled query response engine to obtain an input-candidate image query response adaptive matching result representation matrix as an input-candidate image query response adaptive matching result representation feature; a retrieval result generation module is used to obtain a retrieval result based on the input-candidate image query response adaptive matching result representation feature, and the retrieval result is used to indicate whether to return the first candidate image as a query result; Among them, the semantic encoding feature vector of the retrieval query text input is used as the query feature vector, and the sequence of the query feature vector and the first candidate image local area optimized semantic feature vector is input into the fine-grained feature adaptive decoupling query response engine to obtain the input-alternative image query response adaptive matching result representation matrix, including: fine-grained decoupling of the sequence of the first candidate image local area optimized semantic feature vector to obtain a set of subsequences of the first candidate image local area optimized semantic feature vector; each subsequence of the first candidate image local area optimized semantic feature vector in the set of subsequences of the first candidate image local area optimized semantic feature vector is input into the subsequence semantic reinforcement encoder based on the Bi-LSTM model to obtain a set of first candidate image local area optimized semantic fine-grained decoupling local association feature vectors; calculating the query feature vector and the first candidate image local area optimized semantic fine-grained decoupling local association feature vector. The covariance matrix between the semantic fine-grained decoupled local correlation feature vectors of each first candidate image local area in the set is optimized to obtain a set of query first candidate image local area response covariance matrices; the matrix traces of each query first candidate image local area response covariance matrix in the set of query first candidate image local area response covariance matrices are calculated to obtain a set of query first candidate image local area response trace measurement factors; the set of query first candidate image local area response trace measurement factors is input into a Sigmoid activation function to obtain a set of query first candidate image local area response trace attention weights; using each query first candidate image local area response trace attention weight in the set of query first candidate image local area response trace attention weights as a weight, the weighted sum of the set of query first candidate image local area response covariance matrices is calculated to obtain the input-alternative image query response adaptive matching result representation matrix.

Images