A Sino-Tibetan multimodal image and text processing method and processing system

By extracting and interacting the characteristics of Chinese and Tibetan graphic information, combining multi-gradient segmentation and image copy reorganization, the semantic correlation problem in Chinese and Tibetan graphic interaction is solved, and more realistic and accurate graphic processing results are achieved, and the results of enhanced Supports cross-lingual and cultural exchanges.

CN118709147BActive Publication Date: 2024-11-12BEIJING WANGZHI TIANYUAN BIG DATA TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411201040.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-12
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

In the Chinese-Tibetan graphic interaction processing, due to the different semantic representation methods of image data and text data, semantic correlations between cross-modals are difficult to establish, resulting in the processing results deviating from the real semantic information.

Method used

By obtaining Chinese and Tibetan graphic information, extracting image and text features, and performing feature interaction, building interaction trajectory and interactive feature quantities, combining multi-gradient segmentation and image copy reorganization, enhancing semantic correlation capabilities, and outputting interactive processing confidence to improve The authenticity of the processing results.

Benefits of technology

The semantic correlation between cross-modal interaction between Chinese and Tibetan graphic and text is realized, the authenticity and accuracy of processing results are improved, and the communication support between language and cross-cultural are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118709147B_ABST
    Figure CN118709147B_ABST
Patent Text Reader

Abstract

The present application provides a Sino-Tibetan multimodal image and text processing method and processing system, which extracts the image features of the Sino-Tibetan image modality and the text features of the Sino-Tibetan text modality in the Sino-Tibetan image and text information to determine the first Sino-Tibetan image and text interaction feature information, and determines the first interaction feature quantity according to the first Sino-Tibetan image and text interaction feature information; the image of the Sino-Tibetan image modality is segmented and reorganized to obtain a Sino-Tibetan image copy, and the second Sino-Tibetan image and text interaction feature information is determined according to the image copy features and text features of the Sino-Tibetan image copy, and the second interaction feature quantity is determined according to the second Sino-Tibetan image and text interaction feature information; the interaction result of the Sino-Tibetan image and text interaction is output based on the first interaction feature quantity and the second interaction feature quantity. The above scheme outputs the interaction result based on the first interaction feature quantity and the second interaction feature quantity, which can realize the semantic association of the cross-modal interaction of Sino-Tibetan images and texts, thereby improving the authenticity of the cross-modal interaction processing results of Sino-Tibetan images and texts.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image and text processing, and more specifically, to a Sino-Tibetan multimodal image and text processing method and processing system. Background Art

[0002] Image and text processing technology is crucial in the information age. It includes image processing technology, text processing technology, etc., and is widely used in digital publishing, advertising design, education, medical care, entertainment and other fields. Its core lies in the efficient collection, processing, analysis and display of image and text data to meet various application needs. With the rapid development of computer technology, artificial intelligence and big data technology, image and text processing technology is also constantly innovating. Among them, image processing technology covers image acquisition, compression, storage, enhancement, analysis and recognition. Deep learning is also widely used in image processing, such as convolutional neural network (CNN), which significantly improves the accuracy and speed of image and text recognition. In addition, text processing technology involves text collection, analysis, generation and retrieval, including character recognition (OCR), natural language processing (NLP) and text mining. NLP can achieve tasks such as grammatical analysis, sentiment analysis and topic recognition through machine learning and deep learning. In addition, multimodal fusion technology comprehensively processes information in multiple modes such as images, texts and audios, providing comprehensive information understanding and analysis.

[0003] In the prior art, the implementation principle of image and text processing technology includes: image acquisition, preprocessing, feature extraction, text acquisition, optical character recognition, natural language processing and multimodal data fusion and other steps. Among them, through image processing, the system can identify key information in the image, and text processing uses optical character recognition and natural language processing technology to convert text data into editable text for semantic analysis and emotion recognition. In addition, multimodal fusion technology comprehensively utilizes data of multiple modalities such as images and texts to provide more comprehensive information analysis and decision support; however, in the interactive processing of Sino-Tibetan images and texts, there are differences in the semantic representation methods between different modal data, resulting in a semantic gap problem. For example, image data is represented in the form of pixels or feature vectors, while text data is represented in the form of words or sentences. This difference makes it difficult to establish and understand the semantic association between cross-modal data, thereby causing the processing results of Sino-Tibetan multimodal interactive processing to deviate from the real semantic information. Therefore, how to achieve the semantic association of Sino-Tibetan image and text cross-modal interaction, so as to improve the authenticity of the results of Sino-Tibetan image and text cross-modal interaction processing has become a difficult problem faced by the industry. Summary of the invention

[0004] The present application provides a Sino-Tibetan multimodal image and text processing method and processing system, which can realize the semantic association of Sino-Tibetan image and text cross-modal interaction.

[0005] In a first aspect, the present application provides a Sino-Tibetan multimodal image and text processing method, comprising the following steps:

[0006] Obtain the Chinese-Tibetan graphic information to be processed;

[0007] Extracting image features of the Sino-Tibetan image modality and text features of the Sino-Tibetan text modality from the Sino-Tibetan image-text information according to different modality extractors;

[0008] Performing feature interaction on the image feature and the text feature to obtain first Chinese-Tibetan image-text interaction feature information, constructing a first interaction trajectory of Chinese-Tibetan image-text according to the first Chinese-Tibetan image-text interaction feature information, and then obtaining a first interaction feature quantity;

[0009] The image of the Sino-Tibetan image modality is segmented according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks, all the segmented image blocks are reorganized according to the frame ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and image copy features of the Sino-Tibetan image copy are extracted according to a corresponding extractor;

[0010] Performing feature interaction on the image copy feature and the text feature to obtain second Chinese-Tibetan image-text interaction feature information, constructing a second interaction trajectory of the Chinese-Tibetan image-text according to the second Chinese-Tibetan image-text interaction feature information, and then obtaining a second interaction feature quantity;

[0011] The interactive processing confidence in the process of Sino-Tibetan image-text interactive processing is determined according to the first interactive feature quantity and the second interactive feature quantity, and the interactive result of the Sino-Tibetan image-text interaction is output based on the interactive processing confidence quantity.

[0012] In some embodiments, performing feature interaction on the image feature and the text feature to obtain the first Chinese-Tibetan image-text interaction feature information specifically includes:

[0013] Determining interactive impact parameters of the image features;

[0014] Determining a first interactive influence parameter of the text feature;

[0015] Dimensionally aligning the image feature with the text feature;

[0016] The first Sino-Tibetan image-text interaction feature information is determined according to the interaction influence parameter of the image feature, the first interaction influence parameter of the text feature, and the dimensionally aligned image feature and text feature.

[0017] In some embodiments, determining the interaction influence parameter of the image feature specifically includes:

[0018] Obtaining the dimension size of the image feature and the text feature after dimension alignment;

[0019] Obtaining the image features and the transposed text features;

[0020] The interactive influence parameter of the image feature is determined according to the image feature, the transposed text feature and the dimension size.

[0021] In some embodiments, constructing a first interaction trajectory of Chinese-Tibetan images and texts according to the first Chinese-Tibetan image-text interaction feature information, and then obtaining a first interaction feature quantity specifically includes:

[0022] Obtain all first Sino-Tibetan image-text interaction feature values ​​in the first Sino-Tibetan image-text interaction feature information;

[0023] Connecting all the first interaction feature values ​​of the Sino-Tibetan images and texts to obtain the first interaction trajectory of the Sino-Tibetan images and texts;

[0024] Obtain all peak points and trough points from the first interaction trajectory;

[0025] A first interaction feature quantity is determined according to all peak points and valley points obtained from the first interaction trajectory.

[0026] In some embodiments, segmenting the image of the Sino-Tibetan image modality according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks specifically includes:

[0027] Obtaining a preset multi-gradient segmentation scale, wherein the multi-gradient segmentation scale includes a horizontal segmentation scale and a vertical segmentation scale;

[0028] The images of the Sino-Tibetan image modality are cut one by one according to the horizontal cutting scale and the vertical cutting scale to obtain a plurality of cut image blocks.

[0029] In some embodiments, reorganizing all the segmented image blocks according to the aspect ratio of the image in the Sino-Tibetan image modality to obtain the Sino-Tibetan image copy specifically includes:

[0030] Determining the aspect ratio of the image in the Sino-Tibetan image modality;

[0031] Determining a maximum disturbance distance of the segmented image blocks;

[0032] All the segmented image blocks are randomly perturbed based on the frame ratio and the maximum perturbation distance, and then the perturbed segmented image blocks are reorganized to obtain a Sino-Tibetan image copy.

[0033] In some embodiments, the Chinese-Tibetan image and text information to be processed is obtained through a Chinese-Tibetan image and text database.

[0034] In a second aspect, the present application provides a Sino-Tibetan multimodal image and text processing system, comprising:

[0035] An acquisition module is used to acquire Chinese and Tibetan graphic information to be processed;

[0036] A processing module, used for extracting image features of Sino-Tibetan image modality and text features of Sino-Tibetan text modality from the Sino-Tibetan image-text information according to different modality extractors;

[0037] The processing module is further used to perform feature interaction on the image feature and the text feature to obtain first Sino-Tibetan image-text interaction feature information, construct a first interaction trajectory of the Sino-Tibetan image-text according to the first Sino-Tibetan image-text interaction feature information, and then obtain a first interaction feature quantity;

[0038] The processing module is further used to segment the image of the Sino-Tibetan image modality according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks, reorganize all the segmented image blocks according to the frame ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and extract image copy features of the Sino-Tibetan image copy according to the corresponding extractor;

[0039] The processing module is further used to perform feature interaction on the image copy feature and the text feature to obtain second Chinese-Tibetan image-text interaction feature information, construct a second interaction trajectory of the Chinese-Tibetan image-text according to the second Chinese-Tibetan image-text interaction feature information, and then obtain a second interaction feature value;

[0040] An execution module is used to determine the interaction processing confidence in the process of Chinese-Tibetan image-text interaction according to the first interaction feature and the second interaction feature, and output the interaction result of the Chinese-Tibetan image-text interaction based on the interaction processing confidence.

[0041] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned Sino-Tibetan multimodal image and text processing method.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned Sino-Tibetan multimodal image and text processing method is implemented.

[0043] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:

[0044] In the Sino-Tibetan multimodal image and text processing method and processing system provided by the present application, first, the Sino-Tibetan image and text information to be processed is obtained; secondly, the image features of the Sino-Tibetan image modality and the text features of the Sino-Tibetan text modality in the Sino-Tibetan image and text information are correspondingly extracted according to different modality extractors; further, the image features and the text features are subjected to feature interaction to obtain first Sino-Tibetan image and text interaction feature information, and a first interaction trajectory of the Sino-Tibetan image and text is constructed according to the first Sino-Tibetan image and text interaction feature information, thereby obtaining a first interaction feature quantity; further, the image of the Sino-Tibetan image modality is segmented according to a preset multi-gradient segmentation scale to obtain a plurality of segmentation blocks. , reorganize all the segmented blocks according to the aspect ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and extract the image copy features of the Sino-Tibetan image copy according to the corresponding extractor; then, perform feature interaction on the image copy features and the text features to obtain second Sino-Tibetan image-text interaction feature information, and construct a second interaction trajectory of Sino-Tibetan image and text according to the second Sino-Tibetan image-text interaction feature information, and then obtain a second interaction feature quantity; finally, determine the interaction processing confidence in the Sino-Tibetan image-text interaction processing process according to the first interaction feature quantity and the second interaction feature quantity, and output the interaction result of the Sino-Tibetan image-text interaction based on the interaction processing confidence.

[0045] It can be seen that the present application firstly extracts the image features of the Sino-Tibetan image modality and the text features of the Sino-Tibetan text modality in the Sino-Tibetan image and text information through the pre-trained image feature extractor and text feature extractor, so as to more effectively understand the Sino-Tibetan image content and the Sino-Tibetan text expression, thereby realizing the semantic understanding and analysis between the Sino-Tibetan image and the text; secondly, the Sino-Tibetan image content and the Sino-Tibetan text expression are subjected to feature interaction and the first interactive feature quantity is determined, so as to analyze the model's ability to understand and process complex data, and then identify the ability to establish and understand semantic associations across modalities; then, based on the disordered The image copy features and text features of the reorganized Sino-Tibetan image copies determine the second interaction feature quantity, so as to further identify the ability to establish and understand semantic associations across modalities, thereby avoiding the deviation of the processing results of Sino-Tibetan multimodal interaction processing from the real semantic information; finally, the interaction processing confidence in the Sino-Tibetan image-text interaction processing is determined according to the first interaction feature quantity and the second interaction feature quantity, and the interaction result of the Sino-Tibetan image-text interaction is output based on the interaction processing confidence degree; in summary, the semantic association of Sino-Tibetan image-text cross-modal interaction can be realized, thereby improving the authenticity of the Sino-Tibetan image-text cross-modal interaction processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is an exemplary flow chart of a Sino-Tibetan multimodal image and text processing method according to some embodiments of the present application;

[0047] Figure 2 is an exemplary flow chart for determining first Sino-Tibetan graphic-text interaction feature information according to some embodiments of the present application;

[0048] Figure 3 is an exemplary flow chart of determining image copy features according to some embodiments of the present application;

[0049] Figure 4 is a schematic diagram of exemplary hardware and / or software of a Sino-Tibetan multimodal image and text processing system according to some embodiments of the present application;

[0050] Figure 5 It is a structural schematic diagram of a computer device for implementing a Sino-Tibetan multimodal image and text processing method according to some embodiments of the present application. DETAILED DESCRIPTION

[0051] The core of the present application is to first obtain the Chinese-Tibetan image and text information to be processed; secondly, extract the image features of the Chinese-Tibetan image modality and the text features of the Chinese-Tibetan text modality in the Chinese-Tibetan image and text information according to different modality extractors; further, perform feature interaction on the image features and the text features to obtain first Chinese-Tibetan image and text interaction feature information, construct a first interaction trajectory of the Chinese-Tibetan image and text according to the first Chinese-Tibetan image and text interaction feature information, and then obtain a first interaction feature quantity; further, perform segmentation processing on the image of the Chinese-Tibetan image modality according to a preset multi-gradient segmentation scale to obtain a plurality of segmented blocks, reorganize all the segmented blocks according to the aspect ratio of the image of the Chinese-Tibetan image modality, and obtain to the Sino-Tibetan image copy, and extract the image copy features of the Sino-Tibetan image copy according to the corresponding extractor; then, perform feature interaction on the image copy features and the text features to obtain second Sino-Tibetan image-text interaction feature information, and construct a second interaction trajectory of the Sino-Tibetan image and text according to the second Sino-Tibetan image-text interaction feature information, and then obtain a second interaction feature quantity; finally, determine the interaction processing confidence in the Sino-Tibetan image-text interaction processing process according to the first interaction feature quantity and the second interaction feature quantity, and output the interaction result of the Sino-Tibetan image-text interaction based on the interaction processing confidence, so as to realize the semantic association of the cross-modal interaction of the Sino-Tibetan image and text, thereby improving the authenticity of the cross-modal interaction processing result of the Sino-Tibetan image and text.

[0052] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a Sino-Tibetan multimodal image and text processing method according to some embodiments of the present application. The Sino-Tibetan multimodal image and text processing method 100 mainly includes the following steps:

[0053] In step 101, Chinese-Tibetan graphic information to be processed is obtained.

[0054] In specific implementation, the Sino-Tibetan image and text information to be processed can be obtained through the Sino-Tibetan image and text database, the Sino-Tibetan image and text information represents images and text content that are mainly expressed in Sino-Tibetan (i.e., the languages ​​of Chinese and Tibetan), the Sino-Tibetan image and text information includes Sino-Tibetan image information and Sino-Tibetan text information, the Sino-Tibetan image information represents Sino-Tibetan related content presented in the form of images, such as pictures containing Sino-Tibetan characters, Sino-Tibetan culture, and scenery of Sino-Tibetan areas, etc., and the Sino-Tibetan text information represents Sino-Tibetan related content presented in the form of text, such as Sino-Tibetan articles, news, books, blogs, forum posts, etc.

[0055] It should be noted that the Sino-Tibetan image and text database in this embodiment refers to a database containing Sino-Tibetan images and text information. The establishment of the Sino-Tibetan image and text database aims to collect, organize and store Sino-Tibetan image and text information for subsequent retrieval, analysis and application.

[0056] It should also be noted that the Sino-Tibetan image information and Sino-Tibetan text information are related in content, that is, the content presented by the Sino-Tibetan image can be explained or supplemented by the Sino-Tibetan text, and vice versa. For example, in a news report, the picture usually echoes the text description. In addition, the Sino-Tibetan image information and Sino-Tibetan text information can also complement each other's information. Sino-Tibetan images can provide intuitive Sino-Tibetan visual information, while Sino-Tibetan text can provide more detailed and abstract Sino-Tibetan semantic information. The combination of the two can enrich the expression and transmission of information.

[0057] In step 102, image features of the Sino-Tibetan image modality and text features of the Sino-Tibetan text modality in the Sino-Tibetan image-text information are correspondingly extracted according to different modality extractors.

[0058] It should be noted that the modal extractor in this application refers to a tool or model used to extract features from different types of modal data (such as images, text, audio, video, etc.). In multimodal data processing tasks, data of different modalities have different representations and characteristics, so a special feature extractor is required to process and extract their respective features.

[0059] It should also be noted that the Sino-Tibetan image modality in the present application refers to images related to the Sino-Tibetan languages ​​in the Sino-Tibetan graphic and text information, namely, Sino-Tibetan images. In addition, the Sino-Tibetan text modality in the present application refers to Chinese and Tibetan languages ​​presented in text form in the Sino-Tibetan graphic and text information.

[0060] In some embodiments, before extracting the image features of the Sino-Tibetan image modality and the text features of the Sino-Tibetan text modality in the Sino-Tibetan image and text information according to different modality extractors, it also includes: preprocessing the Sino-Tibetan image modality and the Sino-Tibetan text modality in the Sino-Tibetan image and text information respectively.

[0061] In specific implementation, the Sino-Tibetan image modality is preprocessed, for example, the nearest neighbor interpolation method in the prior art can be used to scale the Sino-Tibetan image modality. In addition, in other embodiments, the Sino-Tibetan image modality can also be preprocessed by operations such as cropping and normalization, which are not limited here. The Sino-Tibetan text modality is preprocessed, for example, the maximum matching method in the prior art can be used to preprocess the Sino-Tibetan text modality. In addition, in other embodiments, the Sino-Tibetan text modality can also be preprocessed by operations such as word segmentation, stop word removal, and word vectorization, which are not limited here.

[0062] It should be noted that the purpose of preprocessing the Sino-Tibetan image modality and the Sino-Tibetan text modality in the Sino-Tibetan graphic information is to adapt to the input requirements of each modality extractor.

[0063] In some embodiments, the following steps may be specifically adopted to extract the image features of the Sino-Tibetan image modality and the text features of the Sino-Tibetan text modality in the Sino-Tibetan graphic information according to different modality extractors, namely:

[0064] Call the pre-trained image feature extractor and text feature extractor;

[0065] Extracting features of the Sino-Tibetan image modality in the Sino-Tibetan image-text information according to the image feature extractor to obtain image features;

[0066] The text feature extractor is used to extract features of the Sino-Tibetan text modality in the Sino-Tibetan graphic information to obtain text features.

[0067] In the specific implementation, the pre-trained image feature extractor and text feature extractor are directly called by a computer, wherein the training of the image feature extractor is based on the Sino-Tibetan image samples in the Sino-Tibetan image and text library, and the training of the text feature extractor is based on the Sino-Tibetan text samples in the Sino-Tibetan image and text library.

[0068] In a specific implementation, a convolutional neural network may be used as an image feature extractor, and the convolutional neural network may be used to perform feature extraction on the Sino-Tibetan image modality in the Sino-Tibetan image-text information to obtain image features. For example, the obtained image features are: [0.234, -0.567, 0.123, 0.456, -0.789, ..., 0.876], wherein each number represents the value of the Sino-Tibetan image modality in a certain feature dimension, and these values ​​together describe the content of the image, which may be features such as the edge, texture, and shape of the Sino-Tibetan language. In addition, in other embodiments, other image feature extractors may be used to perform feature extraction on the Sino-Tibetan image modality in the Sino-Tibetan image-text information, which is not limited here.

[0069] In a specific implementation, a word embedding model may be used as a text feature extractor, and feature extraction is performed on the Sino-Tibetan text modality in the Sino-Tibetan graphic information according to the word embedding model to obtain text features. For example, the obtained text features are: [0.2, 0.1, -0.4, ..., 0.6, -0.3, 0.8], wherein each number in the text feature represents the value of the text on a certain feature dimension, and these values ​​are learned by the text feature extractor during the training process, and specifically represent various features of the text (such as theme, emotion, syntactic structure, etc.), which are used together to describe and represent the content of the text. In addition, in other embodiments, other text feature extractors may be used to extract features from the Sino-Tibetan text modality in the Sino-Tibetan graphic information, which is not limited here.

[0070] It should be noted that the image features in the present application represent feature vectors extracted from the Sino-Tibetan image modality for describing and representing the content of the Sino-Tibetan image, such as texture feature vectors and structural feature vectors for describing the Sino-Tibetan image. In Sino-Tibetan image and text processing, the extraction of image features is aimed at more effectively understanding the content of the Sino-Tibetan image, including objects, scenes, cultural elements, etc. in the image, thereby achieving semantic understanding and analysis of the Sino-Tibetan image, thereby more accurately understanding the meaning expressed by the image, and providing a basis for subsequent natural language processing and semantic analysis.

[0071] It should also be noted that the text feature representation in this application is a feature vector or feature representation used to describe and represent the content of the Sino-Tibetan text, such as the word embedding vector of the Sino-Tibetan text, which maps each word in the text to a continuous vector in a low-dimensional space to represent the semantic information of the word. The purpose of extracting text features is to provide a basis for multimodal image and text processing in Sino-Tibetan languages, so that the text modality in the Sino-Tibetan image and text can be integrated and interacted with the image modality. By combining text features with image features, joint analysis and processing of images and texts can be achieved, providing support for multimodal intelligent applications.

[0072] In step 103, feature interaction is performed on the image features and the text features to obtain first Sino-Tibetan image-text interaction feature information, and a first interaction trajectory of the Sino-Tibetan image-text is constructed according to the first Sino-Tibetan image-text interaction feature information to obtain a first interaction feature quantity.

[0073] In some embodiments, reference Figure 2 As shown, this figure is an exemplary flow chart for determining the first Sino-Tibetan image-text interaction feature information according to some embodiments of the present application. In this embodiment, the image feature and the text feature are subjected to feature interaction, and the first Sino-Tibetan image-text interaction feature information can be obtained by the following steps:

[0074] First, in 1031, the interactive influence parameter of the image feature is determined;

[0075] Next, in 1032, a first interactive influence parameter of the text feature is determined;

[0076] Then, in 1033, the image feature and the text feature are dimensionally aligned;

[0077] Finally, in 1034, first Sino-Tibetan image-text interaction feature information is determined based on the interaction influence parameter of the image feature, the first interaction influence parameter of the text feature, and the dimensionally aligned image feature and text feature.

[0078] In some embodiments, the interactive influence parameter of the image feature may be determined by the following steps, namely:

[0079] Obtaining the dimension size of the image feature and the text feature after dimension alignment;

[0080] Obtaining the image features and the transposed text features;

[0081] The interactive influence parameter of the image feature is determined according to the image feature, the transposed text feature and the dimension size.

[0082] In a specific implementation, the interactive influence parameter of the image feature is determined according to the image feature, the transposed text feature and the dimensional size, that is, the image feature is multiplied by the transposed text feature, and the interactive influence parameter of the image feature is obtained by dividing the product by the dimensional size. In addition, in other embodiments, other calculation methods may be used to calculate the interactive influence parameter of the image feature, which is not limited here.

[0083] It should be noted that the interactive influence parameter of the image feature in this embodiment represents the influence parameter or weight parameter of the image feature in the process of feature interaction between the image feature and the text feature. This parameter can adjust the relative importance of text and image in the process of feature interaction, thereby improving the nature and quality of the final feature representation.

[0084] In some embodiments, the first interactive influence parameter of the text feature may be determined by the following steps, namely:

[0085] Obtaining the dimension size of the image feature and the text feature after dimension alignment;

[0086] Obtaining the text features and the transposed image features;

[0087] A first interactive influencing parameter of the text feature is determined according to the text feature, the transposed image feature and the dimension size.

[0088] In a specific implementation, the first interactive influence parameter of the text feature is determined according to the text feature, the transposed image feature and the dimensional size, that is, the text feature is multiplied by the transposed image feature, and the product calculation result is divided by the dimensional size to obtain the first interactive influence parameter of the text feature. In addition, in other embodiments, other calculation methods can also be used to calculate the first interactive influence parameter of the text feature, which is not limited here.

[0089] It should be noted that the first interactive influence parameter of the text feature in this embodiment represents the influence parameter or weight parameter of the text feature in the process of feature interaction between the image feature and the text feature. This parameter can adjust the relative importance of text and image in the process of feature interaction, thereby improving the nature and quality of the final feature representation.

[0090] In specific implementation, the image features and the text features can be dimensionally aligned through linear transformation in the prior art. For example, the image features and the text features can be projected into a common dimension respectively through two layers of linear transformation, thereby achieving dimensional alignment, which will not be repeated here.

[0091] In a specific implementation, the first Sino-Tibetan image-text interaction feature information is determined based on the interaction influence parameter of the image feature, the first interaction influence parameter of the text feature, and the image feature and text feature after dimension alignment, that is: first, the interaction influence parameter of the image feature is multiplied by the image feature, and secondly, the first interaction influence parameter of the text feature is multiplied by the text feature, and finally, the new feature obtained after the two product calculations is element-wise multiplied to obtain the first Sino-Tibetan image-text interaction feature information.

[0092] It should be noted that, in this embodiment, element-level multiplication represents the process of multiplying the corresponding elements of the two new features calculated one by one, and the first Sino-Tibetan image-text interaction feature information in this application represents the first Sino-Tibetan image-text comprehensive feature obtained after the interaction of Sino-Tibetan image features and Sino-Tibetan text features. The first Sino-Tibetan image-text interaction feature information includes multiple Sino-Tibetan image-text first interaction feature values ​​(that is, each element in the first Sino-Tibetan interaction feature), and the Sino-Tibetan image-text first interaction feature value represents the element value obtained after the interaction of Sino-Tibetan image features and Sino-Tibetan text features. The first interactive feature value of Sino-Tibetan images and texts is used to measure the semantic correlation and representational similarity between Sino-Tibetan images and Sino-Tibetan texts. That is, in Sino-Tibetan multimodal image processing, the Sino-Tibetan image-text interactive feature information integrates the features of Sino-Tibetan texts and images, and can better understand the image content and the corresponding text description to capture the semantic correlation between text and images. The determination of the Sino-Tibetan image-text interactive feature information can not only analyze the model's ability to understand and process complex data, but also enhance the model's robustness and generalization ability, promote cross-language and cross-cultural communication, and support richer application scenarios.

[0093] In some embodiments, the first interaction trajectory of the Chinese-Tibetan image and text is constructed according to the first Chinese-Tibetan image and text interaction feature information, and then the first interaction feature value is obtained by specifically adopting the following steps, namely:

[0094] Obtain all first Sino-Tibetan image-text interaction feature values ​​in the first Sino-Tibetan image-text interaction feature information;

[0095] Connecting all the first interaction feature values ​​of the Sino-Tibetan images and texts to obtain the first interaction trajectory of the Sino-Tibetan images and texts;

[0096] Obtain all peak points and trough points from the first interaction trajectory;

[0097] A first interaction feature quantity is determined according to all peak points and valley points obtained from the first interaction trajectory.

[0098] In specific implementation, all the first interactive feature values ​​of Sino-Tibetan images and texts are connected to obtain the first interactive trajectory of Sino-Tibetan images and texts, that is, all the first interactive feature values ​​of Sino-Tibetan images and texts are connected by curves in chronological order to obtain the first interactive trajectory of Sino-Tibetan images and texts. In this embodiment, the first interactive trajectory represents a curve obtained by visually connecting the first interactive feature values ​​of Sino-Tibetan images and texts in the first Sino-Tibetan image-text interactive feature information. The determination of the first interactive trajectory is to analyze the changing trend of the first interactive feature values ​​of Sino-Tibetan images and texts and explore the correlation between images and texts, so as to better understand the interactive behavior between Sino-Tibetan images and texts, and provide strong support for subsequent data analysis and application.

[0099] In a specific implementation, all peak points and trough points are obtained from the first interaction trajectory, that is, starting from the second data point in the first interaction trajectory, for each data point, the value of the data point is differenced with the values ​​of the adjacent previous and next data points. If the values ​​corresponding to the previous and next difference calculation results are positive and negative respectively, then the data point is a peak point; if the values ​​corresponding to the previous and next difference calculation results are negative and positive respectively, then the data point is a trough point; in other cases, it is not regarded as a peak point or a trough point.

[0100] It should be noted that, in this embodiment, the peak point represents the dividing point from an upward trend to a downward trend in the first interaction trajectory, and the trough point represents the dividing point from a downward trend to an upward trend in the first interaction trajectory. The determination of the peak point and the trough point is to effectively identify the variability of the semantic association between the image and the text after the current Sino-Tibetan image modality and the Sino-Tibetan text modality interact.

[0101] In a specific implementation, the first interaction feature is determined based on all the peak points and trough points obtained from the first interaction trajectory, that is: first, the first interaction feature values ​​of the Chinese-Tibetan image and text corresponding to all the peak points are summed to obtain a first summation result; secondly, the first interaction feature values ​​of the Chinese-Tibetan image and text corresponding to all the trough points are summed to obtain a second summation result; finally, the ratio of the second summation result to the first summation result is used as the first interaction feature.

[0102] It should be noted that, in the present application, the first interactive feature quantity represents the trend change amount of the first interactive feature value of Sino-Tibetan image and text in the first Sino-Tibetan image and text interactive feature information. The closer the first interactive feature quantity is to 1, the smoother the trend change of the first interactive feature value of Sino-Tibetan image and text in the first Sino-Tibetan image and text interactive feature information, indicating that the semantic association between the image and the text after the interaction between the Sino-Tibetan image modality and the Sino-Tibetan text modality is greater. The further the first interactive feature quantity is from 1, the steeper the trend change of the first interactive feature value of Sino-Tibetan image and text in the first Sino-Tibetan image and text interactive feature information, indicating that the semantic association between the image and the text after the interaction between the Sino-Tibetan image modality and the Sino-Tibetan text modality is smaller.

[0103] In step 104, the image of the Sino-Tibetan image modality is segmented according to a preset multi-gradient segmentation scale to obtain a plurality of segmented blocks, all the segmented blocks are reorganized according to the aspect ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and image copy features of the Sino-Tibetan image copy are extracted according to the corresponding extractor.

[0104] In some embodiments, the image of the Sino-Tibetan image modality is segmented according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks, which may be specifically performed by the following steps, namely:

[0105] Obtaining a preset multi-gradient segmentation scale, wherein the multi-gradient segmentation scale includes a horizontal segmentation scale and a vertical segmentation scale;

[0106] The images of the Sino-Tibetan image modality are cut one by one according to the horizontal cutting scale and the vertical cutting scale to obtain a plurality of cut image blocks.

[0107] It should be noted that the multi-gradient segmentation scale preset in the present embodiment is set based on the horizontal scale, vertical scale and actual needs of the image of the Sino-Tibetan image modality. For example, the image of the Sino-Tibetan image modality is evenly divided according to the horizontal scale to obtain a preset horizontal segmentation scale, and the image of the Sino-Tibetan image modality is evenly divided according to the vertical scale to obtain a preset vertical segmentation scale. In addition, in other embodiments, other setting methods may be used to set the horizontal segmentation scale and the vertical segmentation scale, which are not limited here.

[0108] It should also be noted that the segmented image blocks in this embodiment represent sub-image blocks obtained by segmenting the image of the Sino-Tibetan image modality, and different regions in the image may have different semantics and features. By segmenting the image into multiple small image blocks, the local information of the image can be better captured, so that the image processing algorithm can analyze and understand the image content more finely.

[0109] In some embodiments, reference Figure 3 As shown, this figure is an exemplary flow chart of determining a Sino-Tibetan image copy according to some embodiments of the present application. In this embodiment, all the segmented blocks are reorganized by the frame ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, which can be achieved by the following steps:

[0110] First, in step 1041, the aspect ratio of the image in the Sino-Tibetan image modality is determined;

[0111] Then, in step 1042, the maximum disturbance distance of the segmented image block is determined;

[0112] Finally, in step 1043, all the segmented image blocks are randomly perturbed based on the frame ratio and the maximum perturbation distance, and then the perturbed segmented image blocks are reorganized to obtain a Sino-Tibetan image copy.

[0113] In a specific implementation, the aspect ratio of the image in the Sino-Tibetan image modality is determined. The aspect ratio can be obtained by calculating the resolution of the image in the Sino-Tibetan image modality (determined by the ratio of the width to the length of the image in the Sino-Tibetan image modality). It should be noted that, in this embodiment, the aspect ratio represents the overall size scale of the image in the Sino-Tibetan image modality.

[0114] It should be noted that all the segmented blocks are reorganized based on the frame ratio to obtain a new Sino-Tibetan image copy. The reorganization process needs to ensure seamless connection between the blocks, which can be achieved through image processing algorithms (such as smooth transition, edge fusion, etc.).

[0115] In a specific implementation, the maximum disturbance distance of the segmented image block is determined. The maximum disturbance distance can be determined by the difference between the maximum continuous feature number and the minimum continuous feature number of the segmented image block. The maximum continuous feature number represents the maximum number of segmented image blocks with continuous similar features after image segmentation of the Sino-Tibetan image modality. The maximum continuous feature number is used to measure the maximum segmentation degree of image similarity features of the Sino-Tibetan image modality. The minimum continuous feature number represents the minimum number of segmented image blocks with continuous similar features after image segmentation of the Sino-Tibetan image modality. The minimum continuous feature number is used to measure the minimum segmentation degree of image similarity features of the Sino-Tibetan image modality.

[0116] It should be noted that the maximum perturbation distance in this embodiment represents the range within which the segmented blocks can be randomly exchanged or moved, and the maximum perturbation distance determines the maximum range within which each block can be moved, thereby introducing appropriate confusion while ensuring a certain regularity of the Sino-Tibetan image modality.

[0117] It should also be noted that the Sino-Tibetan image copy in the present application represents a new Sino-Tibetan image obtained by segmenting and perturbing the image of the Sino-Tibetan image modality and then reorganizing it. In the Sino-Tibetan multimodal image and text processing, perturbing and reorganizing the image of the Sino-Tibetan image modality can enable more robust features to be learned in the process of Sino-Tibetan semantic understanding. In addition, the local and global features of the Sino-Tibetan image can be better learned. By disrupting the local areas of the image of the Sino-Tibetan image modality, the model needs to re-identify and understand these features in different contexts, thereby enhancing its ability to understand the image details and overall structure.

[0118] In some embodiments, the following steps may be specifically used to extract the image copy features of the Sino-Tibetan image copy according to the corresponding extractor, namely:

[0119] Call the image copy feature extractor of the pre-trained Sino-Tibetan image copy;

[0120] The image copy feature extractor is used to extract features of the Sino-Tibetan image copy to obtain image copy features of the Sino-Tibetan image copy.

[0121] In the specific implementation, a pre-trained image copy feature extractor is directly called by a computer, wherein the training of the image copy feature extractor is based on segmented and reorganized Sino-Tibetan image samples in a Sino-Tibetan image and text library.

[0122] In a specific implementation, a convolutional neural network may be used as an image copy feature extractor, and feature extraction is performed on the Sino-Tibetan image copy according to the convolutional neural network to obtain image copy features. For example, the obtained image copy features are: [0.124, -0.357, 0.213, 0.446, -0.867, ..., 0.877], wherein each number represents the value of the Sino-Tibetan image copy on a certain feature dimension, and these values ​​together describe the content of the Sino-Tibetan image copy, which may be features such as the edge, texture, and shape of the Sino-Tibetan language. In addition, in other embodiments, other image copy feature extractors may be used to extract features from the Sino-Tibetan image copy, which is not limited here.

[0123] It should be noted that the image copy features in the present application represent feature vectors extracted from the Sino-Tibetan image copies for describing and representing the content of the reorganized Sino-Tibetan image, such as texture feature vectors and structural feature vectors for describing the reorganized Sino-Tibetan image. In Sino-Tibetan image and text processing, the extraction of image copy features is aimed at more effectively understanding the local features of the Sino-Tibetan image, including edge features, intermediate features, etc. in the image, thereby achieving a deeper semantic understanding and analysis of the Sino-Tibetan image, and more accurately understanding the meaning expressed by the image.

[0124] In step 105, feature interaction is performed on the image copy feature and the text feature to obtain second Sino-Tibetan image-text interaction feature information, and a second interaction trajectory of the Sino-Tibetan image-text is constructed based on the second Sino-Tibetan image-text interaction feature information to obtain a second interaction feature quantity.

[0125] In some embodiments, the following steps may be used to perform feature interaction on the image copy feature and the text feature to obtain the second Chinese-Tibetan image-text interaction feature information, namely:

[0126] Determining interactive influencing parameters of the image copy features;

[0127] Determining a second interactive influence parameter of the text feature;

[0128] Dimensionally aligning the image copy features with the text features;

[0129] The second Sino-Tibetan image-text interaction feature information is determined according to the interaction influence parameter of the image copy feature, the second interaction influence parameter of the text feature, and the dimensionally aligned image copy feature and text feature.

[0130] In some embodiments, the interactive influencing parameter of the image copy feature may be determined by the following steps, namely:

[0131] Obtaining the dimension size of the image copy feature after dimension alignment with the text feature;

[0132] Obtaining the image copy features and the transposed text features;

[0133] The interactive influence parameter of the image copy feature is determined according to the image copy feature, the transposed text feature and the dimension size.

[0134] In a specific implementation, the interactive influence parameter of the image copy feature is determined according to the image copy feature, the transposed text feature and the dimensional size, that is, the image copy feature is multiplied by the transposed text feature, and the interactive influence parameter of the image copy feature is obtained by dividing the product calculation result by the dimensional size. In addition, in other embodiments, other calculation methods can also be used to calculate the interactive influence parameter of the image copy feature, which is not limited here.

[0135] It should be noted that the interactive influence parameter of the image copy feature in this embodiment represents the influence parameter or weight parameter of the image copy feature on the feature interaction during the feature interaction between the image copy feature and the text feature. This parameter can adjust the relative importance of text and image in the feature interaction process, thereby improving the nature and quality of the final feature representation.

[0136] In some embodiments, the second interactive influence parameter of the text feature may be determined by the following steps, namely:

[0137] Obtaining the dimension size of the image copy feature after dimension alignment with the text feature;

[0138] Obtaining the text features and the transposed image copy features;

[0139] A second interaction influencing parameter of the text feature is determined according to the text feature, the transposed image copy feature, and the dimension size.

[0140] In a specific implementation, the second interactive influence parameter of the image copy feature is determined according to the text feature, the transposed image copy feature and the dimensional size, that is, the text feature is multiplied by the transposed image copy feature, and the product calculation result is divided by the dimensional size to obtain the second interactive influence parameter of the text feature. In addition, in other embodiments, other calculation methods can also be used to calculate the second interactive influence parameter of the text feature, which is not limited here.

[0141] It should be noted that the second interaction influence parameter of the text feature in this embodiment represents the influence parameter or weight parameter of the text feature on the feature interaction during the feature interaction between the image copy feature and the text feature. This parameter can adjust the relative importance of text and image in the feature interaction process, thereby improving the nature and quality of the final feature representation.

[0142] In specific implementation, the image copy features and the text features can be dimensionally aligned through linear transformation in the prior art. For example, the image copy features and the text features can be projected into a common dimension respectively through two layers of linear transformation, thereby achieving dimensional alignment, which will not be repeated here.

[0143] In a specific implementation, the second Chinese-Tibetan image-text interaction feature information is determined based on the interaction influence parameter of the image copy feature, the second interaction influence parameter of the text feature, and the dimensionally aligned image copy feature and text feature, that is: first, the interaction influence parameter of the image copy feature is multiplied by the image copy feature, and secondly, the second interaction influence parameter of the text feature is multiplied by the text feature, and finally, the new feature obtained after the two product calculations is element-wise multiplied to obtain the second Chinese-Tibetan image-text interaction feature information.

[0144] It should be noted that, in this embodiment, element-level multiplication represents the process of multiplying the corresponding elements of the two new features calculated one by one. In this application, the second Sino-Tibetan image-text interaction feature information represents the representation of the second Sino-Tibetan image-text comprehensive feature obtained after the Sino-Tibetan image copy feature and the Sino-Tibetan text feature interact. The second Sino-Tibetan image-text interaction feature information includes multiple Sino-Tibetan image-text second interaction feature values ​​(that is, each element in the second Sino-Tibetan interaction feature). The Sino-Tibetan image-text second interaction feature value represents the element value obtained after the Sino-Tibetan image copy feature and the Sino-Tibetan text feature interact. The second interactive feature value of Sino-Tibetan images and texts is used to measure the semantic relevance and representational similarity between Sino-Tibetan images and Sino-Tibetan texts, that is, in Sino-Tibetan multimodal image processing, the Sino-Tibetan image-text interactive feature information integrates the features of Sino-Tibetan texts and images, so that the image content and the corresponding text description can be better understood to capture the semantic relevance between text and images. The determination of the Sino-Tibetan image-text interactive feature information can further analyze the model's ability to understand and process complex data, and also enhance the model's robustness and generalization ability, promote cross-language and cross-cultural communication, and support richer application scenarios.

[0145] In some embodiments, the second interaction trajectory of the Chinese-Tibetan image and text is constructed according to the second Chinese-Tibetan image and text interaction feature information, and then the second interaction feature value is obtained by specifically adopting the following steps, namely:

[0146] Obtain all second Sino-Tibetan image-text interaction feature values ​​in the second Sino-Tibetan image-text interaction feature information;

[0147] Connecting all the second interaction feature values ​​of the Sino-Tibetan images and texts to obtain the second interaction trajectory of the Sino-Tibetan images and texts;

[0148] Obtain all peak points and trough points from the second interaction trajectory;

[0149] A second interaction feature quantity is determined according to all peak points and valley points obtained in the second interaction trajectory.

[0150] In the specific implementation, all the second interaction characteristic values ​​of the Sino-Tibetan image and text are connected to obtain the second interaction trajectory of the Sino-Tibetan image and text, that is: all the second interaction characteristic values ​​of the Sino-Tibetan image and text are connected by a curve in sequence to obtain the second interaction trajectory of the Sino-Tibetan image and text. In this embodiment, the second interaction trajectory represents a curve obtained by visually connecting the second interaction characteristic values ​​of the Sino-Tibetan image and text in the second Sino-Tibetan image and text interaction feature information. The determination of the second interaction trajectory is to analyze the changing trend of the second interaction characteristic values ​​of the Sino-Tibetan image and text and explore the correlation between the images and texts, so as to better understand the interaction behavior between the Sino-Tibetan image and text, and provide strong support for subsequent data analysis and application.

[0151] In a specific implementation, all peak points and trough points are obtained from the second interactive trajectory, that is, starting from the second data point in the second interactive trajectory, for each data point, the value of the data point is differenced with the values ​​of the adjacent previous and next data points. If the values ​​corresponding to the previous and next difference calculation results are positive and negative respectively, then the data point is a peak point; if the values ​​corresponding to the previous and next difference calculation results are negative and positive respectively, then the data point is a trough point; in other cases, it is not regarded as a peak point or a trough point.

[0152] It should be noted that, the peak point in the second interaction trajectory in this embodiment represents the dividing point from an upward trend to a downward trend in the second interaction trajectory, and the trough point in the second interaction trajectory in this embodiment represents the dividing point from a downward trend to an upward trend in the second interaction trajectory. The determination of the peak point and the trough point is to effectively identify the variability of the semantic association between the image and the text after the current Sino-Tibetan image modality and the Sino-Tibetan text modality interact.

[0153] In specific implementation, the second interaction feature is determined based on all peak points and trough points obtained in the second interaction trajectory, that is: first, the second interaction feature values ​​of the Chinese-Tibetan image and text corresponding to all peak points are summed to obtain a first summation result; secondly, the second interaction feature values ​​of the Chinese-Tibetan image and text corresponding to all trough points are summed to obtain a second summation result; finally, the ratio of the first summation result to the second summation result is used as the second interaction feature.

[0154] It should be noted that the second interaction feature quantity in the present application is used to measure the trend change amount of the second interaction feature value of the Sino-Tibetan image and text in the second Sino-Tibetan image and text interaction feature information. The closer the second interaction feature quantity is to 1, the smoother the trend change of the second interaction feature value of the Sino-Tibetan image and text in the second Sino-Tibetan image and text interaction feature information, indicating that the greater the semantic association between the image and the text after the interaction between the image of the Sino-Tibetan image modality and the text of the Sino-Tibetan text modality. The further the second interaction feature quantity is from 1, the steeper the trend change of the second interaction feature value of the Sino-Tibetan image and text in the second Sino-Tibetan image and text interaction feature information, indicating that the smaller the semantic association between the image and the text after the interaction between the image of the Sino-Tibetan image modality and the text of the Sino-Tibetan text modality.

[0155] In step 106, the interaction processing confidence in the Chinese-Tibetan image-text interaction process is determined according to the first interaction feature and the second interaction feature, and the interaction result of the Chinese-Tibetan image-text interaction is output based on the interaction processing confidence.

[0156] In some embodiments, the following steps may be used to determine the interactive processing confidence level in the Chinese-Tibetan text-graphic interactive processing process according to the first interactive feature quantity and the second interactive feature quantity, namely:

[0157] An absolute difference is calculated between the first interaction feature and the second interaction feature, and the inverse of the absolute difference calculation result is taken to obtain the interaction processing confidence in the Chinese-Tibetan image-text interaction processing process.

[0158] It should be noted that, in other embodiments, other calculation methods may be used to calculate the interactive processing confidence level in the Chinese-Tibetan text-graphic interactive processing process, which is not limited here.

[0159] It should also be noted that the interaction processing confidence in the present application indicates the degree of credibility of the results of the Sino-Tibetan image-text interaction processing, that is, the greater the interaction processing confidence, the greater the credibility of the results of the Sino-Tibetan image-text interaction processing; and the smaller the interaction processing confidence, the lower the credibility of the results of the Sino-Tibetan image-text interaction processing.

[0160] In some embodiments, the following steps may be used to output the interaction result of the Chinese-Tibetan text-graphic interaction based on the interaction processing confidence level, namely:

[0161] The interaction processing confidence is compared with the preset interaction processing confidence. When the interaction processing confidence is greater than the preset interaction processing confidence, the Sino-Tibetan multimodal image and text processing system outputs the first Sino-Tibetan image and text interaction feature information as the interaction result of the Sino-Tibetan image and text interaction. When the interaction processing confidence is less than or equal to the preset interaction processing confidence, the interaction result of the Sino-Tibetan image and text interaction is not output.

[0162] It should be noted that in the Sino-Tibetan multimodal image and text processing, the local texture features, local shape features, etc. of the Sino-Tibetan images will not change. Even if the Sino-Tibetan images are segmented and then disorderly reorganized, the local texture features, local shape features, etc. in the reorganized images are the same. Moreover, the interactive features obtained by the feature interaction between the Sino-Tibetan images and the Sino-Tibetan texts have a strong global similarity with the interactive features obtained by the feature interaction between the Sino-Tibetan image copies and the Sino-Tibetan texts. Among them, since the first Sino-Tibetan image and text interactive feature information is obtained by weighted interaction between image features and text features, the first Sino-Tibetan image and text interactive feature information will contain the original The global structure and semantic features of Sino-Tibetan images, even if the original Sino-Tibetan images are segmented and reorganized in random order, the features within each block remain unchanged, so the overall global features will also be retained. When the reorganized Sino-Tibetan image interacts with the text again, since the same original image features and text features are used, the second Sino-Tibetan image-text interaction feature information will also remain consistent with the first Sino-Tibetan image-text interaction feature information. Furthermore, by determining the interaction processing confidence in the Sino-Tibetan image-text interaction process, the effectiveness and authenticity of the current interaction model for Sino-Tibetan multimodal image-text interaction can be effectively identified, thereby outputting the interaction results of the Sino-Tibetan image-text interaction.

[0163] It should be noted that the preset interaction processing confidence in this embodiment represents a preset standard or indicator for identifying the interaction credibility of Chinese-Tibetan images and texts during the Chinese-Tibetan image-text interaction process. This indicator is used to quantify and evaluate the degree of interaction credibility of Chinese-Tibetan images and texts. The preset scoring threshold is pre-set and can be set specifically according to actual needs, which will not be repeated here.

[0164] In the specific implementation, the interaction processing confidence is compared with the preset interaction processing confidence. When the interaction processing confidence is greater than the preset interaction processing confidence, it indicates that the current Chinese-Tibetan image and text have good interaction processing capabilities. Therefore, the information obtained by performing feature interaction on the Chinese-Tibetan image and text is more effective and reliable. Then, the Chinese-Tibetan multimodal image and text processing system can output the first Chinese-Tibetan image and text interaction feature information as the interaction result of the Chinese-Tibetan image and text interaction. When the interaction processing confidence is less than or equal to the preset interaction processing confidence, it indicates that the current Chinese-Tibetan image and text have poor interaction processing capabilities. Therefore, the effectiveness and reliability of the information obtained by the interaction on the Chinese-Tibetan image and text are worse. In this case, the interaction processing result information is not output, and the interaction processing can be improved to obtain a reliable interaction result.

[0165] In addition, in another aspect of the present application, in some embodiments, the present application provides a Sino-Tibetan multimodal image and text processing system, referring to Figure 4, which is a schematic diagram of exemplary hardware and / or software of a Sino-Tibetan multimodal image and text processing system according to some embodiments of the present application. The Sino-Tibetan multimodal image and text processing system 200 includes: an acquisition module 201, a processing module 202 and an execution module 203, which are described as follows:

[0166] Acquisition module 201, in this application, acquisition module 201 is mainly used to acquire Chinese-Tibetan graphic information to be processed;

[0167] Processing module 202, in the present application, processing module 202 is mainly used to extract image features of Sino-Tibetan image modality and text features of Sino-Tibetan text modality in the Sino-Tibetan graphic information according to different modality extractors;

[0168] The processing module 202 is further used to perform feature interaction on the image feature and the text feature to obtain first Sino-Tibetan image-text interaction feature information, construct a first interaction trajectory of the Sino-Tibetan image-text according to the first Sino-Tibetan image-text interaction feature information, and then obtain a first interaction feature value;

[0169] The processing module 202 is further used to segment the image of the Sino-Tibetan image modality according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks, reorganize all the segmented image blocks according to the frame ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and extract image copy features of the Sino-Tibetan image copy according to the corresponding extractor;

[0170] In addition, the processing module 202 is further used to perform feature interaction on the image copy feature and the text feature to obtain second Chinese-Tibetan image-text interaction feature information, construct a second interaction trajectory of the Chinese-Tibetan image-text according to the second Chinese-Tibetan image-text interaction feature information, and then obtain a second interaction feature value;

[0171] Execution module 203, in the present application, execution module 203 is mainly used to determine the interaction processing confidence in the process of Chinese-Tibetan image-text interaction processing according to the first interaction feature and the second interaction feature, and output the interaction result of the Chinese-Tibetan image-text interaction based on the interaction processing confidence.

[0172] In addition, the present application also provides a computer device, which includes a memory and a processor, the memory stores a code, and the processor is configured to obtain the code and execute the above-mentioned Sino-Tibetan multimodal image and text processing method.

[0173] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device using a Sino-Tibetan multimodal image and text processing method according to some embodiments of the present application. The Sino-Tibetan multimodal image and text processing method in the above embodiment can be performed by Figure 5 The computer device 300 shown in the figure is implemented, and the computer device 300 includes at least one processor 301, a communication bus 302, a memory 303 and at least one communication interface 304.

[0174] The processor 301 may be a general-purpose central processing unit (CPU), or an application-specific integrated circuit (ASIC) or one or more processors for controlling the execution of the Chinese-Tibetan multimodal image and text processing method in the present application.

[0175] The communication bus 302 may be used to transmit information between the above-mentioned components.

[0176] The memory 303 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.

[0177] The memory 303 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the program code stored in the memory 303. The program code may include one or more software modules. The determination of the Sino-Tibetan multimodal image and text processing method in the above embodiment can be implemented by the processor 301 and one or more software modules in the program code in the memory 303.

[0178] The communication interface 304 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0179] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0180] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0181] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned Sino-Tibetan multimodal image and text processing method is implemented.

[0182] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0183] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A Sino-Tibetan multimodal image and text processing method, characterized in that: The steps include: Obtain the Chinese-Tibetan graphic information to be processed; Extracting image features of the Sino-Tibetan image modality and text features of the Sino-Tibetan text modality from the Sino-Tibetan image-text information according to different modality extractors; Performing feature interaction on the image feature and the text feature to obtain first Chinese-Tibetan image-text interaction feature information, constructing a first interaction trajectory of Chinese-Tibetan image-text according to the first Chinese-Tibetan image-text interaction feature information, and then obtaining a first interaction feature quantity; The image of the Sino-Tibetan image modality is segmented according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks, all the segmented image blocks are reorganized according to the frame ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and image copy features of the Sino-Tibetan image copy are extracted according to a corresponding extractor; Performing feature interaction on the image copy feature and the text feature to obtain second Chinese-Tibetan image-text interaction feature information, constructing a second interaction trajectory of the Chinese-Tibetan image-text according to the second Chinese-Tibetan image-text interaction feature information, and then obtaining a second interaction feature quantity; Determining an interaction processing confidence in the process of Chinese-Tibetan image-text interaction processing according to the first interaction feature quantity and the second interaction feature quantity, and outputting an interaction result of the Chinese-Tibetan image-text interaction based on the interaction processing confidence quantity; The step of performing feature interaction on the image feature and the text feature to obtain the first Chinese-Tibetan image-text interaction feature information specifically includes: Determining interactive impact parameters of the image features; Determining a first interactive influence parameter of the text feature; Dimensionally aligning the image feature with the text feature; Determine first Sino-Tibetan image-text interaction feature information according to the interaction influence parameter of the image feature, the first interaction influence parameter of the text feature, and the dimensionally aligned image feature and text feature; The interactive influencing parameters of the image features are determined as follows: Obtaining the dimension size of the image feature and the text feature after dimension alignment; Obtaining the image features and the transposed text features; Determine the interactive influence parameter of the image feature according to the image feature, the transposed text feature and the dimension size; Wherein, determining the first interactive influencing parameter of the text feature specifically includes: Obtaining the dimension size of the image feature and the text feature after dimension alignment; Obtaining the text features and the transposed image features; Determine a first interactive influence parameter of the text feature according to the text feature, the transposed image feature and the dimension size; The feature interaction between the image copy feature and the text feature to obtain the second Chinese-Tibetan image-text interaction feature information specifically includes: Determining interactive influencing parameters of the image copy features; Determining a second interactive influence parameter of the text feature; Dimensionally aligning the image copy features with the text features; Determine second Sino-Tibetan image-text interaction feature information according to the interaction influence parameter of the image copy feature, the second interaction influence parameter of the text feature, and the dimensionally aligned image copy feature and text feature; The interactive influencing parameters for determining the image copy features specifically include: Obtaining the dimension size of the image copy feature after dimension alignment with the text feature; Obtaining the image copy features and the transposed text features; Determine the interactive influence parameter of the image copy feature according to the image copy feature, the transposed text feature and the dimension size; Wherein, determining the second interactive influencing parameter of the text feature specifically includes: Obtaining the dimension size of the image copy feature after dimension alignment with the text feature; Obtaining the text features and the transposed image copy features; A second interaction influencing parameter of the text feature is determined according to the text feature, the transposed image copy feature, and the dimension size.

2. The method according to claim 1, characterized in that Constructing a first interaction trajectory of the Chinese-Tibetan image and text according to the first Chinese-Tibetan image and text interaction feature information, and then obtaining a first interaction feature quantity specifically includes: Obtain all first Sino-Tibetan image-text interaction feature values ​​in the first Sino-Tibetan image-text interaction feature information; Connecting all the first interaction feature values ​​of the Sino-Tibetan images and texts to obtain the first interaction trajectory of the Sino-Tibetan images and texts; Obtain all peak points and trough points from the first interaction trajectory; A first interaction feature quantity is determined according to all peak points and valley points obtained from the first interaction trajectory.

3. The method according to claim 1, characterized in that The image of the Sino-Tibetan image modality is segmented according to a preset multi-gradient segmentation scale to obtain a plurality of segmentation blocks, specifically including: Obtaining a preset multi-gradient segmentation scale, wherein the multi-gradient segmentation scale includes a horizontal segmentation scale and a vertical segmentation scale; The images of the Sino-Tibetan image modality are cut one by one according to the horizontal cutting scale and the vertical cutting scale to obtain a plurality of cut image blocks.

4. The method according to claim 1, characterized in that Reorganizing all the segmented image blocks according to the aspect ratio of the image in the Sino-Tibetan image modality to obtain the Sino-Tibetan image copy specifically includes: Determining the aspect ratio of the image in the Sino-Tibetan image modality; Determining a maximum disturbance distance of the segmented image blocks; All the segmented image blocks are randomly perturbed based on the frame ratio and the maximum perturbation distance, and then the perturbed segmented image blocks are reorganized to obtain a Sino-Tibetan image copy.

5. The method according to claim 1, characterized in that The Sino-Tibetan image and text information to be processed is obtained through the Sino-Tibetan image and text database.

6. A Sino-Tibetan multimodal image and text processing system, which uses the method of any one of claims 1 to 5 to perform image and text processing, characterized in that: The Sino-Tibetan multimodal image and text processing system includes: An acquisition module is used to acquire Chinese and Tibetan graphic information to be processed; A processing module, used for extracting image features of Sino-Tibetan image modality and text features of Sino-Tibetan text modality from the Sino-Tibetan image-text information according to different modality extractors; The processing module is further used to perform feature interaction on the image feature and the text feature to obtain first Sino-Tibetan image-text interaction feature information, construct a first interaction trajectory of the Sino-Tibetan image-text according to the first Sino-Tibetan image-text interaction feature information, and then obtain a first interaction feature quantity; The processing module is further used to segment the image of the Sino-Tibetan image modality according to a preset multi-gradient segmentation scale to obtain a plurality of segmented image blocks, reorganize all the segmented image blocks according to the frame ratio of the image of the Sino-Tibetan image modality to obtain a Sino-Tibetan image copy, and extract image copy features of the Sino-Tibetan image copy according to the corresponding extractor; The processing module is further used to perform feature interaction on the image copy feature and the text feature to obtain second Chinese-Tibetan image-text interaction feature information, construct a second interaction trajectory of the Chinese-Tibetan image-text according to the second Chinese-Tibetan image-text interaction feature information, and then obtain a second interaction feature value; An execution module is used to determine the interaction processing confidence in the process of Chinese-Tibetan image-text interaction according to the first interaction feature and the second interaction feature, and output the interaction result of the Chinese-Tibetan image-text interaction based on the interaction processing confidence.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores codes, and the processor is configured to obtain the codes and execute the Sino-Tibetan multimodal image and text processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the Sino-Tibetan multimodal image and text processing method as described in any one of claims 1 to 5 is implemented.