Text data determination method, electronic equipment, storage medium and program product

Abstract

The invention discloses a text data determination method, electronic equipment, a storage medium and a program product, and relates to the technical field of data processing, a text data determination model is utilized to perform feature extraction on a view image set of multiple views, integrate information of a time dimension and a space dimension to obtain an aggregation feature, and a matching result is generated based on the aggregation feature. The matching text provides text description information corresponding to the aggregated features, and the matching score quantifies the matching accuracy. The problem of multi-modal joint characterization and capture feature relevance is solved, and the cross-modal characterization capability and the cross-modal matching accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a text data determination method, electronic equipment, storage medium and program product. Background Art

[0002] The cross-modal retrieval task aims to improve the understanding and analysis capabilities of complex scenarios by integrating multimodal data (such as images and text). In the medical field, this method combines medical images with text descriptions and extracts features from them to support tasks such as text-assisted diagnosis and multimodal diagnosis report generation, thereby improving the accuracy and efficiency of clinical decision-making.

[0003] However, related technologies perform poorly in the joint representation of multimodal data and find it difficult to effectively capture the feature correlations between data. Summary of the invention

[0004] The present application provides a text data determination method, an electronic device, a storage medium, and a program product to at least solve the problems of the related art in the joint representation of multimodal data and the establishment of feature association.

[0005] The present application provides a text data determination method, comprising:

[0006] Acquire data to be matched, wherein the data to be matched includes a plurality of view image sets, and any of the view image sets includes image data collected at a plurality of stages under the view;

[0007] For any view image set, the view image set is input into a text data determination model, the aggregate features corresponding to the view image set are determined, and a matching result is determined based on the aggregate features, wherein the matching result includes a matching text and a matching score that matches the view image set.

[0008] The present application also provides a text data determination device, comprising:

[0009] An acquisition module, used to acquire data to be matched, wherein the data to be matched includes a plurality of view image sets, and any of the view image sets includes image data collected at a plurality of stages under the view;

[0010] A determination module is used to input any view image set into a text data determination model, determine the aggregate features corresponding to the view image set, and determine a matching result based on the aggregate features, wherein the matching result includes a matching text and a matching score that matches the view image set.

[0011] The present application also provides an electronic device, comprising: a memory, a transceiver and a processor, wherein the memory is used to store a computer program; the transceiver is used to send and receive data under the control of the processor; and the processor is used to implement the steps of any one of the above-mentioned text data determination methods when executing the computer program.

[0012] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned text data determination methods are implemented.

[0013] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned text data determination methods when executed by a processor.

[0014] Through this application, a text data determination model is used to extract visual features from a set of view images of different views, and aggregate features are obtained, which can capture rich visual information that a single perspective cannot provide, enhance the comprehensiveness and diversity of features, and effectively integrate information in the time and space dimensions; the matching results generated based on these aggregate features include matching text and matching scores that match the view image set. The matching text provides descriptive information corresponding to the aggregate features, and the matching score quantifies the accuracy and reliability of the match. Therefore, the method of this application effectively solves the shortcomings of related technologies in the joint representation of multimodal data, as well as the problem of difficulty in effectively capturing the feature correlation between data, and improves cross-modal representation capabilities and the accuracy of cross-modal matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] Figure 1 A schematic diagram of an application scenario of a text data determination method provided in an embodiment of the present application;

[0017] Figure 2 A flowchart of a text data determination method provided in an embodiment of the present application Figure 1 ;

[0018] Figure 3 Schematic diagram of the training process of the text data determination model provided in the embodiment of the present application Figure 1 ;

[0019] Figure 4 Schematic diagram of the training process of the text data determination model provided in the embodiment of the present application Figure 2 ;

[0020] Figure 5A schematic diagram of a framework of a text data determination method provided in an embodiment of the present application;

[0021] Figure 6 Schematic diagram of the process of determining text data provided in the embodiment of the present application Figure 2 ;

[0022] Figure 7 A schematic diagram of the structure of a text data determination device provided in an embodiment of the present application;

[0023] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0024] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] The cross-modal retrieval task aims to improve the understanding and analysis capabilities of complex scenes by integrating multimodal data (such as images and text). Taking the application of cross-modal retrieval in the medical field as an example, this method combines medical images with text descriptions and extracts features from them to support tasks such as text-assisted diagnosis and multimodal diagnosis report generation, thereby improving the accuracy and efficiency of clinical decision-making. However, related technologies have neglected the focus on multi-view image data and cannot effectively capture rich information from different perspectives; in addition, medical data not only contains spatial information, but also involves dynamic changes in the time dimension, such as the distribution of contrast agents in the body over time. Due to the limitations of these problems, it is impossible to effectively capture the correlation between features, which limits the joint representation capabilities of multimodal data and restricts the performance of related technologies in medical applications.

[0028] In order to capture richer visual information and capture the dynamic changes of medical data, view image sets from multiple views, such as axial, sagittal and coronal view image sets, can be used. Each set contains medical image data collected at multiple stages under this view, so as to comprehensively capture rich visual information from different perspectives and retain dynamic changes in time. In order to effectively capture the association between features, these view image sets can be input into the text data determination model to extract aggregate features, which can represent the association of data in time and space dimensions. To further enhance the joint representation ability of multimodal data, matching results can be generated based on aggregate features, including matching text and matching scores that match the view image sets. The matching text provides a clinical description corresponding to the image content, while the matching score quantifies the accuracy and reliability of the match, which not only improves the interpretability and credibility of the model output, but also improves the performance of the model in actual medical applications.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the medical text data determination method depends, the specific application environment architecture or specific hardware architecture is described herein. Figure 1 , Figure 1 A schematic diagram of an application scenario of a text data determination method provided in an embodiment of the present application. Figure 1As shown, the data request device 101 and the request processing device 102 are included. The data request device 101 may be a local terminal device of the user, and the user may send a text data query request to the request processing device 102 through the data request device 101. The request processing device 102 may be a remote terminal device, and the request processing device 102 may process the text data query request and determine a matching result, and encapsulate the matching result in a text data query response and return it to the data request device 101.

[0031] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0032] Figure 2 A flowchart of a text data determination method provided in an embodiment of the present application Figure 1 ,like Figure 2 As shown, an embodiment of the present application provides a text data determination method, and the method is described in detail as follows:

[0033] S201: Acquire data to be matched, where the data to be matched includes multiple view image sets, and any view image set includes image data collected at multiple stages under the view.

[0034] The data to be matched refers to the data that needs to be matched using the model, and the view refers to a specific perspective.

[0035] In this embodiment, the multiple view image sets may include an axial plane view image set, a sagittal plane view image set, and a coronal plane view image set. Among them, the axial plane view image set refers to a view image set collected from the horizontal direction (parallel to the ground), which is often used to display the structure of the cross section of the human body. The sagittal plane view image set refers to a view image set collected from the front-to-back direction (from front to back, or from back to front), which is often used to display the structure of the side of the human body. The coronal plane view image set refers to a view image set collected from the left-to-right direction (from left to right, or from right to left), which is often used to display the structure of the front or back of the human body.

[0036] The stage can refer to the image acquisition at different time points in the medical imaging diagnosis process, or it can refer to the image acquisition under different contrast agent injection conditions. Images at different stages can provide different types of clinical information.

[0037] In this embodiment, the image data may be collected by X-ray imaging, magnetic resonance imaging (MRI), computed tomography (CT), etc., which is not limited in this embodiment.

[0038] For example, taking the axial view image set of the brain obtained by CT scanning as an example, the view image set can include three stages, namely the axial contrast-free stage, the axial arterial phase stage and the axial perfusion phase. Among them, the axial contrast-free stage refers to the image acquisition without the injection of any contrast agent. This image is mainly used to observe the basic structure and density changes of the tissue; the axial arterial phase stage refers to the image acquisition after the contrast agent is injected and the contrast agent enters the arterial system. At this stage, the distribution of blood vessels and blood flow can be observed, which can be used to evaluate vascular diseases; the axial perfusion phase refers to the image acquisition when the contrast agent further diffuses and affects tissue perfusion. At this stage, the blood perfusion of the tissue can be observed, which can be used to evaluate the blood flow and functional status of the tissue.

[0039] S202: For any view image set, input the view image set into the text data determination model, determine the aggregate features corresponding to the view image set, and determine the matching results based on the aggregate features, where the matching results include matching text and matching scores that match the view image set.

[0040] The text data determination model refers to a model used to extract image features and determine text data matching the features.

[0041] Aggregate features can be comprehensive features extracted from multiple images, and aggregate features can represent the common characteristics of images in a set of view images. For example, aggregate features can represent key information such as the shape, size, location, and change status of the lesion area.

[0042] The matching text may be a diagnosis report, a description of a disease condition, or other forms of medical documents related to the aggregated features.

[0043] The matching score is a quantitative indicator that can be used to quantify the relevance or credibility of the current matching text and the aggregated features.

[0044] As an optional implementation, the text data determination model may include a three-dimensional visual transformer.

[0045] The text data determination model can determine the aggregate features corresponding to the view image set through the following steps: input the images of each stage in the view image set into the three-dimensional visual transformer, and extract the labeled features corresponding to the images of each stage; use the text data determination model and determine the global features corresponding to each stage based on the labeled features corresponding to each stage, the global features are the features corresponding to the classification labels in the labeled features, and the classification labels are learnable vectors; use the text data determination model and preset the global features corresponding to each stage to obtain the aggregate features corresponding to the view image set.

[0046] A 3D visual transformer refers to a deep learning model for processing 3D data structures, where the 3D data may be data in spatial and temporal dimensions.

[0047] For example, in medical image analysis, some diseases or pathological conditions show significant changes at different time points. Therefore, in order to monitor the development of lesions or the effect of treatment, medical images usually need to be collected at multiple time points. By modeling the time dimension, these dynamic changes can be effectively tracked, thereby providing more accurate diagnosis and treatment evaluation. 3D visual transformers can capture this multi-dimensional information and achieve effective modeling of temporal and spatial information.

[0048] Tokenization is the process of converting input data (such as an image) into a series of discrete tags. Tokenized features are a collection of these tags, representing different parts of the image.

[0049] The global feature may be a feature representing the overall characteristics of the image at this stage.

[0050] In this embodiment, the tokenization features of each stage are determined The process can be expressed as:

[0051]

[0052]

[0053] in, It is a three-dimensional visual transformer; is a collection of view images, A collection of images for a view The images of each stage; Indicates the number of markers, that is, how many patches the image is divided into; represents the embedding size of each token, that is, the feature vector dimension of each token.

[0054] Specifically, the images at each stage is split into multiple patches, each of which is converted into a token In addition, At the beginning of the corresponding tag sequence, a special classification tag (CLS) can be added. This tag does not correspond to a specific image patch, but serves as a learnable vector to capture the global information of the entire tag sequence. The 3D visual transformer uses a multi-layer self-attention mechanism to classify the tag sequence. Each tag (including classification tags) is converted into a new feature vector, and the classification tags can integrate information from the entire tag sequence to generate a comprehensive global feature representation , .

[0055] This embodiment uses a three-dimensional visual transformer to extract aggregate features from a set of view images. The three-dimensional visual transformer can simultaneously capture the complex relationships between images in spatial and temporal dimensions, thereby more comprehensively understanding the semantic information of the set of view images. Secondly, by extracting the labeled features of images at each stage and focusing on the global features corresponding to the classification labels, the three-dimensional visual transformer can effectively summarize key high-level information and reduce the interference of redundant data. The method of pre-processing global features to generate aggregate features is also conducive to improving the ability to model image data across stages.

[0056] As an optional implementation, the global features corresponding to each stage may be preset in the following manner to obtain aggregate features corresponding to the view image set:

[0057] According to the preset mask, the global features corresponding to each stage are average pooled to obtain the first aggregated feature; the first aggregated feature is linearly transformed to obtain the first transformed feature; the first transformed feature is normalized to obtain the aggregated feature.

[0058] The preset mask can be a binary matrix or vector to indicate which features should be retained or ignored. During feature processing, the mask can be used to selectively focus on certain specific parts of the features and ignore other parts of the features.

[0059] Mean pooling refers to a dimensionality reduction operation. It helps reduce the data dimension and retain key information by averaging different parts of the input data to generate a more compact representation.

[0060] Linear transformation is a mathematical operation. Linear transformation can transform input data into a new representation through matrix multiplication. In linear transformation, a weight matrix (i.e., transformation matrix) can be used to achieve this transformation.

[0061] Normalization is a data preprocessing technique that adjusts data of different scales to the same range (such as between 0 and 1) to better compare and analyze features.

[0062] In this embodiment, the first aggregate feature can be expressed as:

[0063]

[0064] in, is the number of stages of the view image set, and the default mask is , For the The mask of each stage, Representative The phase is the effective phase. Representative The first stage is the mask stage.

[0065] The first transformation feature can be expressed as:

[0066]

[0067] in, is the weight matrix.

[0068] The aggregated features can be expressed as:

[0069]

[0070] in, is the Euclidean norm of the first transformed feature.

[0071] In this embodiment, through the mean pooling operation combined with the preset mask selection mechanism, the global features of the effective stage can be screened out, and irrelevant or redundant information can be ignored, which is conducive to reducing the data dimension and retaining key semantic content, and improving the efficiency of feature calculation. Linear changes can improve the flexibility of feature representation, and normalization adjusts the features to a uniform scale range, eliminating the dimensional differences between features at different stages, and is also conducive to enhancing the stability of aggregated features.

[0072] The text data determination method provided in this embodiment uses multiple view image sets, each set containing image data of multiple stages, to comprehensively capture rich visual information from different perspectives and retain dynamic changes in time. These view image sets are input into the text data determination model to extract aggregate features, which can represent the correlation of image data in time and space dimensions. The matching results generated based on the aggregate features include matching text corresponding to the image content and matching scores that quantify the matching accuracy and reliability, thereby enhancing the joint representation capability of multimodal data.

[0073] Based on the above embodiments, Figure 3The illustrated embodiment provides a detailed description of the training process of the medical text data determination model in the embodiment of the present application.

[0074] Figure 3 Schematic diagram of the training process of the text data determination model provided in the embodiment of the present application Figure 1 . See Figure 3 , the method may include:

[0075] S301: Acquire initial data, where the initial data includes a plurality of initial view image sets and initial text data corresponding to each initial view image set.

[0076] Initial data may refer to data that has not been selected and preprocessed, including image and text information. Such data may be unprocessed or unscreened, and may contain noise, redundant information, or irrelevant information.

[0077] In this embodiment, the initial data may include a plurality of initial view image sets and text description information corresponding to the initial view image sets, that is, initial text data.

[0078] S302: Determine target data according to the initial data, where the target data includes target image data and target text data.

[0079] The target data may refer to data selected and processed from the initial data.

[0080] As an optional implementation, the target data can be determined from the initial data in the following manner: using a preset data selection strategy and based on an initial view image set to determine the target image data; using a preset data enhancement strategy to enhance the initial text data to obtain the target text data.

[0081] The preset data selection strategy refers to a predefined method or rule for selecting a suitable image from the initial view image set as the target image data. The preset data selection strategy can be based on a variety of criteria, such as image quality, label information, the presence or absence of specific features, etc.

[0082] A preset data enhancement strategy refers to a predefined method or rule for enhancing the initial text data to make it richer and more diverse. The preset data enhancement strategy can include operations such as data cleaning, expansion, and conversion.

[0083] In this embodiment, the target data is determined from the initial data through a preset data selection strategy and a preset data enhancement strategy. The preset data selection strategy can efficiently screen out qualified target image data based on clear standards (such as image quality, label information or specific features), avoiding noise interference caused by blind selection, and is conducive to improving the relevance of image data to specific business needs. The preset data enhancement strategy is conducive to enriching the diversity of text data by performing enhancement operations on the initial text data, and also makes up for the possible deficiencies of the original text data. In addition, the use of a "preset" approach can also make the process of determining target data more standardized and controllable.

[0084] As an optional implementation, the target image data can be determined in the following manner: obtain a preset stage number, and for any initial view image set, obtain the corresponding initial stage number; compare the preset stage number with the initial stage number to obtain a comparison result; if the preset stage number is equal to the initial stage number, determine the image in the initial view image set as the target image data; if the initial stage number is greater than the preset stage number, select the first image data from the initial view image set so that the stage number of the first image data forming the set is equal to the preset stage number, and determine the first image data as the target image data; if the initial stage number is less than the preset stage number, fill the initial view image set with a zero tensor to obtain the second image data, so that the stage number of the second image data forming the set is equal to the preset stage number, and determine the second image data as the target image data.

[0085] The preset number of stages refers to a preset value, which indicates the number of stages that each view image set is expected to contain.

[0086] The initial stage number refers to the number of stages actually included in the initial view image set.

[0087] Zero tensor padding is a data processing technology that adds zero-value tensors (i.e., all-zero matrices) to the insufficient parts of the initial view image set, so that the number of stages of the view image set can reach the preset number without adding additional noise.

[0088] Specifically, for each initial view image set, the number of stages actually contained therein is counted, and then compared with the preset number of stages to determine whether the initial number of stages is equal to, greater than, or less than the preset number of stages. If the number of stages in the initial view image set is exactly equal to the preset number of stages, no modification is required, and these images are directly used as target image data; if the number of stages in the initial view image set is greater than the preset number of stages, a part of images is selected from the initial view image set so that the number of stages of the set formed by these images is equal to the preset number of stages, and the selected images are used as target image data; if the number of stages in the initial view image set is less than the preset number of stages, the insufficient part in the initial view image set is filled with zero tensors until its number of stages is equal to the preset number of stages, and the filled view image set is used as the target image data.

[0089] In this embodiment, when selecting images at each stage from the initial view image set, the images may also be evaluated so that the selected images can contain richer information to better support the training of the model.

[0090] For example, in order to retain the timing information in the view image set, the images may be filtered in the following manner:

[0091] For example, a uniform sampling strategy can be used to uniformly select images of each stage in the time dimension to ensure that each stage has representative image data; or a key image selection strategy can be used to select key images based on certain predefined criteria (such as significant image changes, occurrence of specific events, etc.), which may contain important stage information; sliding window technology can also be used to select images within a certain time interval to ensure stage continuity in the temporal dimension.

[0092] For example, in order to improve the diversity of the view image set, the images may be screened in the following manner:

[0093] For example, a clustering algorithm can be used to divide the images in the initial view image set into multiple clusters, and the most representative image can be selected from each cluster; or the feature distance between images can be calculated, and those images with a large distance between feature vectors can be selected to ensure the differences between the selected images; or images with different features or perspectives can be selected based on professional knowledge or label information.

[0094] In this embodiment, by comparing the preset stage number with the initial stage number, the view image set in different situations can be flexibly processed to determine the target image data. The preset stage number is clearly set as a unified standard to ensure the consistency of all target image data in the stage number, providing standardized input for model training. In the case where the initial stage number is greater than the preset stage number, the first image data is formed by selectively extracting part of the image, avoiding the interference of redundant information while retaining key data; and when the initial stage number is less than the preset stage number, the zero tensor filling technology is used to fill the insufficient part without introducing additional noise, effectively solving the problem of missing data and ensuring the integrity of the data structure.

[0095] As an optional implementation, the initial text data can be enhanced in the following manner to obtain target text data: the initial text data is segmented to obtain an initial text sequence, the initial text sequence contains at least one segmented word; for any segmented word, the synonym corresponding to the segmented word is obtained from a predefined dictionary; for the initial text sequence, the segmented words are replaced with synonyms to obtain a first text sequence; for the initial text sequence, the position of each segmented word is exchanged according to a preset exchange strategy to obtain a second text sequence; the initial text sequence, the first text sequence, and the second text sequence are determined as target text data.

[0096] Word segmentation refers to the process of segmenting continuous text into discrete vocabulary units. In Chinese, this usually involves segmenting sentences into words; in English, it involves segmenting sentences into words or subwords. Each word segmentation represents a vocabulary unit in the original text data.

[0097] A predefined dictionary can be a predefined vocabulary. Each vocabulary can have a corresponding synonym list. A predefined dictionary can be built by domain experts or automatically generated by natural language processing tools.

[0098] Synonyms are words with the same or similar meanings. In the text enhancement process, using synonyms to replace original words can increase the diversity of the text without changing its core meaning.

[0099] The preset exchange strategy refers to a predefined rule or method for determining how to exchange the positions of each word segment in the initial text sequence. The preset exchange strategy may include randomly exchanging adjacent words, etc.

[0100] The target text data may be a collection of text data obtained after enhancement processing.

[0101] Exemplarily, the initial text data can be segmented into multiple word segments, a list of synonyms for each word segment is obtained from a predefined dictionary, and each word segment in the initial text sequence is replaced with a synonym to generate a first text sequence; the position of each word segment in the initial text sequence is exchanged according to a preset exchange strategy to generate a second text sequence; the initial text sequence, the first text sequence, and the second text sequence can be combined together as target text data.

[0102] Enhancing the initial text data can increase the diversity and richness of the text data, and is also beneficial to improving the effectiveness and robustness of model training.

[0103] In this embodiment, by performing word segmentation processing on the initial text data and combining it with synonym replacement in the predefined dictionary, the expression of the initial text data can be effectively expanded, the semantic diversity can be increased, and the model can better understand and adapt to different language forms. The position of the word segmentation is adjusted using a preset exchange strategy, which further enriches the sentence structure of the text and enhances the robustness of the model to word order changes. The initial text sequence, the first text sequence (synonym replacement result) and the second text sequence (position exchange result) are used as target text data together, which not only retains the original information, but also introduces a variety of variants, providing richer samples for model training, and is also conducive to improving the generalization ability of the model.

[0104] S303: Construct a data set according to the target image data and the target text data.

[0105] A data set refers to a data set used to train and evaluate a model, including images and corresponding text data. In this embodiment, the data set can be in the form of:

[0106]

[0107] in, is the preset number of stages, is the target image data, The target text data.

[0108] S304: According to a preset loss function, the text data determination model is trained using the data set to obtain a text data determination model that is trained to convergence. The preset loss function includes an image-to-text loss function and a text-to-image loss function.

[0109] The preset loss function is a function used to measure the difference between the model results and the true label. In multimodal tasks, the loss function can contain multiple parts.

[0110] The image-to-text loss function refers to the loss function that measures the difference between the text description generated from the image and the actual text description; the text-to-image loss function refers to the loss function that measures the difference between the image features generated from the text and the actual image features.

[0111] Training to convergence means updating the model parameters through multiple iterations until the model performance no longer improves significantly.

[0112] In this embodiment, the data set can be used to train the text data determination model, and the preset loss function can be used to optimize the model until the model reaches a convergence state.

[0113] The text data determination method provided in this embodiment screens and enhances the initial view image set and initial text data, uses a preset number of stages to ensure the representativeness and information richness of the target image data, and fills the insufficient parts with zero tensors; by implementing technologies such as word segmentation, synonym replacement, and position exchange on the initial text data, the diversity and richness of the target text data are enhanced, and the robustness of the model is improved. Combining the image-to-text and text-to-image loss functions is conducive to the effective alignment and matching of image and text data, so that the model can better understand the intrinsic connection between cross-modal data.

[0114] Based on the above embodiments, Figure 4 The illustrated embodiment further illustrates the training process of the text data determination model in the embodiment of the present application.

[0115] Figure 4 Schematic diagram of the training process of the text data determination model provided in the embodiment of the present application Figure 2 Please attend Figure 4 , the method may include:

[0116] S401: Input the target image data in the data set into the three-dimensional visual transformer to obtain the training image aggregation features corresponding to the target image data.

[0117] The execution process of S401 can refer to S202 and will not be repeated here.

[0118] S402: Input the target text data in the data set into the bidirectional transformer encoder to obtain the training text features corresponding to the target text data.

[0119] A bidirectional transformer encoder is a natural language processing model that generates word representations by using a bidirectional encoder with a transformer architecture, which can simultaneously capture the forward and backward context information of each token in the input sequence.

[0120] As an optional implementation, the training text features corresponding to the target text data can be obtained through the following steps: inputting the training target text data into a bidirectional transformer encoder to obtain a hidden state sequence; determining the global text features based on the hidden state sequence, the global text features being the features corresponding to the classification labels in the hidden state sequence; normalizing the global text features to determine the training text features.

[0121] The hidden state sequence refers to the data representation learned by the intermediate layers of the model when processing the input data. For text data, the hidden state sequence can contain the internal representation of each word or token after it passes through the encoder.

[0122] The global text feature may be a feature representing the overall characteristics of the target text data.

[0123] In this embodiment, the process of obtaining the hidden state sequence can be expressed as:

[0124]

[0125]

[0126] in, For bidirectional converter encoder, is a sequence of text tags, , is the maximum sequence length of the text token sequence, For the The token of a word or subword unit, is the embedding size of each token, i.e., the feature vector dimension of each token.

[0127] Specifically, the target text data can be converted into a sequence of text tags In addition, similar to the 3D visual transformer, a special classification tag can be added at the beginning of the text tag sequence. This tag does not correspond to a specific subword unit, but serves as a learnable vector to integrate the global information of the entire text tag sequence. This generates a comprehensive global text feature .

[0128] In order to better align the global text features with the training image aggregate features, the global text features can be normalized to determine the training text features. The training text features can be expressed as:

[0129]

[0130] in, is the Euclidean norm of the global text feature.

[0131] In this embodiment, the target text data is processed using a bidirectional transformer encoder, which can fully capture contextual information and generate a semantically rich hidden state sequence, thereby more accurately expressing the text content. By extracting the features corresponding to the classification tags in the hidden state sequence as global text features, the key information of the entire text tag sequence is integrated to form a compact and comprehensive representation, which effectively reflects the overall characteristics of the target text data. Normalizing the global text features not only eliminates the differences in feature vectors at different scales, but also enhances their alignment capabilities with the aggregated features of the training images.

[0132] S403: Calculate the similarity between the training image aggregate features and the training text features.

[0133] Similarity can be the degree of proximity between two objects in a certain feature or attribute, and is an indicator to measure the similarity between the two.

[0134] As an optional implementation, the similarity between the training image aggregate features and the training text features may be calculated in the following manner: the training image aggregate features are transposed to obtain transposed aggregate features; and the transposed aggregate features and the training text features are dot-producted to obtain similarity.

[0135] Transposition is a basic concept in matrix or vector operations, which refers to the operation of swapping the rows and columns of a matrix.

[0136] The dot product operation is a basic operation between two vectors.

[0137] In this embodiment, the similarity can be expressed as:

[0138]

[0139] in, Representative The aggregated features of the training images and The similarity between the features of the training texts, For the Transposed aggregate features, For the training text features, Calculates the dot product.

[0140] In this embodiment, the transposition process converts the aggregated features of the training image into a form suitable for the dot product operation, which simplifies the calculation process while retaining the complete information of the feature vector. Using the dot product operation as a similarity calculation method can directly reflect the directional consistency of the two feature vectors in the high-dimensional space. The larger the value, the closer the two are, thus providing an intuitive and efficient similarity measurement method.

[0141] S404: Determine a comparison loss value according to the similarity, the image-to-text loss function, and the text-to-image loss function.

[0142] As an optional implementation, the contrast loss value can be determined in the following manner: determine a first contrast loss value based on similarity and an image-to-text loss function; determine a second contrast loss value based on similarity and a text-to-image loss function; determine a contrast loss value based on the first contrast loss value and the second contrast loss value.

[0143] In this embodiment, the loss function of image to text can be expressed as:

[0144]

[0145] in, is the temperature parameter, which is used to control the smoothness of the normalized exponential function (softmax). Representative The training image aggregate features and The similarity between the training text features, is an exponential transformation.

[0146] Correspondingly, the first contrast loss value can be determined according to the image-to-text loss function.

[0147] The loss function for text to image can be expressed as:

[0148]

[0149] in, For the training text features and The similarity between the aggregated features of the training images, For the training text features and The similarity between the aggregated features of the training images.

[0150] Correspondingly, the second contrast loss value can be determined according to the text-to-image loss function.

[0151] The contrast loss value is determined according to the first contrast loss value and the second contrast loss value, where the contrast loss value is:

[0152]

[0153] In this embodiment, the first contrast loss value and the second contrast loss value are calculated separately, which can comprehensively measure the matching degree between the image and the text from a bidirectional perspective, ensuring that the model not only focuses on the prediction accuracy of the image to the text, but also pays attention to the mapping consistency of the text to the image. Integrating the two to determine the final contrast loss value effectively balances the relationship between the image and the text in the feature space, avoiding the deviation problem that may be caused by one-way optimization. This method not only improves the alignment ability of the model in multimodal tasks, but also enhances its robustness and generalization performance.

[0154] S405: adjusting the training parameters in the text data determination model based on the contrast loss value until the contrast loss value reaches a minimum, and determining the text data determination model with the minimum contrast loss value as the text data determination model.

[0155] Training parameters are the variables in the model that need to be optimized through the training process. These training parameters determine how the model maps inputs to outputs. For example, in a neural network, these parameters can be weights and bias terms.

[0156] In this embodiment, the text data determination model continuously adjusts its internal parameters according to the change of the contrast loss value to minimize the contrast loss value. When the contrast loss value reaches the minimum, it means that the difference between the prediction result of the medical text data determination model and the true label has been minimized as much as possible, and the text data determination model at this time can be determined as the text data determination model.

[0157] Figure 5 A schematic diagram of a framework of a text data determination method provided in an embodiment of the present application. Figure 5 It can be seen that the text data determination method provided in the embodiment of the present application can receive three-dimensional medical imaging data and diagnostic text data as input, extract image features and text features respectively, model the correlation relationship between the features, and optimize the model using contrast loss. Figure 5 The specific execution process of the text data determination method shown can be found in the above embodiment and will not be described in detail here.

[0158] The text data determination method provided in this embodiment realizes the joint representation and fusion of two different types of data, image and text, by inputting the target image data into the three-dimensional visual transformer to obtain the training image aggregation features, and inputting the target text data into the bidirectional transformer encoder to obtain the training text features. The similarity is calculated based on the transposition and dot product operations, and the contrast loss values ​​of image to text and text to image are calculated respectively according to the similarity, so as to further optimize the model parameters and realize the effective alignment and matching of image and text features.

[0159] Based on the above embodiments, Figure 6The embodiments shown are used to further illustrate the technical solution of the present application.

[0160] Figure 6 Schematic diagram of the process of determining text data provided in the embodiment of the present application Figure 2 . See Figure 6 , the method may include:

[0161] S601: receiving a text data query request sent by a user terminal.

[0162] The user terminal refers to a device for a user to interact with the device where the text data query method in this embodiment is located. The user can input a text data query request through the user terminal and receive the corresponding query result.

[0163] S602: Acquire the data to be matched according to the text data query request.

[0164] Exemplarily, the text data query request may include a view image set collected by the user, and the view image set may be regarded as the data to be matched. This embodiment does not limit the content of the data query request and the method of obtaining the data to be matched.

[0165] S603: Input the data to be matched into the text data determination model to determine the matching result.

[0166] The execution process of S603 may refer to S202 and will not be described in detail here.

[0167] S604: Send a text data query response to the user terminal, where the text data query response includes a matching result.

[0168] For example, the matching result output by the text data determination model may be encapsulated into a text data query response and sent to a user terminal for the user to view.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0170] Figure 7 This is a schematic diagram of the structure of the text data determination device provided in the embodiment of the present application. Figure 7 As shown, the text data determination device includes: an acquisition module 11 and a determination module 12.

[0171] The acquisition module 11 is used to acquire the data to be matched.

[0172] The data to be matched includes multiple view image sets, and any view image set includes image data collected at multiple stages under the view.

[0173] The determination module 12 is used for inputting the view image set into the text data determination model for any view image set, determining the aggregated features corresponding to the view image set, and determining the matching result according to the aggregated features.

[0174] The matching result includes the matching text and the matching score that matches the view image set.

[0175] In one implementation, the determination module 12 is specifically configured to:

[0176] The images of each stage in the view image set are input into the three-dimensional visual transformer to extract the tokenized features corresponding to the images of each stage; the model is determined by using text data and the global features corresponding to each stage are determined based on the tokenized features corresponding to each stage; the model is determined by using text data and the global features corresponding to each stage are preset to obtain the aggregated features corresponding to the view image set.

[0177] Among them, the global feature is the feature corresponding to the classification label in the tokenized feature, and the classification label is a learnable vector.

[0178] Optionally, the text data determination device further includes a processing module, which is specifically used to:

[0179] According to the preset mask, the global features corresponding to each stage are average pooled to obtain the first aggregated feature; the first aggregated feature is linearly transformed to obtain the first transformed feature; the first transformed feature is normalized to obtain the aggregated feature.

[0180] In one implementation, the acquisition module 11 is further configured to:

[0181] Get initial data.

[0182] The initial data includes a plurality of initial view image sets and initial text data corresponding to each initial view image set.

[0183] In one embodiment, the determination module 12 is further configured to:

[0184] According to the initial data, target data is determined, and the target data includes target image data and target text data; according to the target image data and target text data, a data set is constructed.

[0185] Optionally, the text data determination device further includes a training module, and the training module is specifically used for:

[0186] According to a preset loss function, the text data determination model is trained using the data set to obtain a text data determination model that is trained to convergence.

[0187] Among them, the preset loss function includes the loss function from image to text and the loss function from text to image.

[0188] In one implementation, the determination module 12 is specifically configured to:

[0189] A preset data selection strategy is adopted and based on an initial view image set, target image data is determined; a preset data enhancement strategy is adopted to enhance the initial text data to obtain target text data.

[0190] Optionally, the text data determination device further includes a comparison module, the comparison module being specifically configured to:

[0191] Obtain a preset stage number, and for any initial view image set, obtain the corresponding initial stage number; compare the preset stage number with the initial stage number to obtain a comparison result; if the preset stage number is equal to the initial stage number, determine the image in the initial view image set as the target image data; if the initial stage number is greater than the preset stage number, select the first image data from the initial view image set so that the stage number of the first image data forming the set is equal to the preset stage number, and determine the first image data as the target image data; if the initial stage number is less than the preset stage number, fill the initial view image set with a zero tensor to obtain the second image data, so that the stage number of the second image data forming the set is equal to the preset stage number, and determine the second image data as the target image data.

[0192] In one embodiment, the processing module is further configured to:

[0193] Perform word segmentation processing on initial text data to obtain an initial text sequence, wherein the initial text sequence contains at least one word segmentation; for any word segmentation, obtain a synonym corresponding to the word segmentation from a predefined dictionary; for the initial text sequence, use the synonym to replace the word segmentation to obtain a first text sequence; for the initial text sequence, according to a preset exchange strategy, exchange the position of each word segmentation to obtain a second text sequence; determine the initial text sequence, the first text sequence, and the second text sequence as target text data.

[0194] In one embodiment, the training module is specifically used to:

[0195] Input the target image data in the data set into the three-dimensional visual transformer to obtain the training image aggregation features corresponding to the target image data; input the target text data in the data set into the bidirectional transformer encoder to obtain the training text features corresponding to the target text data; calculate the similarity between the training image aggregation features and the training text features; determine the contrast loss value based on the similarity, the first loss function and the second loss function; adjust the training parameters in the text data determination model based on the contrast loss value until the contrast loss value is minimized, and determine the text data determination model when the contrast loss value is minimized as the text data determination model.

[0196] In one embodiment, the training module is further used to:

[0197] The training target text data is input into the bidirectional transformer encoder to obtain a hidden state sequence; based on the hidden state sequence, a global text feature is determined, and the global text feature is a feature corresponding to a classification label in the hidden state sequence; the global text feature is normalized to determine the training text feature.

[0198] In one embodiment, the training module is further used to:

[0199] The aggregated features of the training image are transposed to obtain the transposed aggregated features; the dot product of the transposed aggregated features and the training text features is calculated to obtain the similarity.

[0200] In one embodiment, the training module is further used to:

[0201] According to the similarity and the image-to-text loss function, a first contrast loss value is determined; according to the similarity and the text-to-image loss function, a second contrast loss value is determined; according to the first contrast loss value and the second contrast loss value, a contrast loss value is determined.

[0202] Optionally, the text data determination device further includes a receiving module, and the receiving module is specifically used to:

[0203] Receive a text data query request sent by a user terminal.

[0204] The acquisition module 11 is further used for:

[0205] According to the text data query request, the data to be matched is obtained.

[0206] Optionally, the text data determination device further includes a sending module, and the sending module is specifically used to:

[0207] Send a text data query response to the user terminal.

[0208] The text data query response includes matching results.

[0209] The description of the features in the embodiment corresponding to the text data determination device can refer to the relevant description of the embodiment corresponding to the text data determination method, which will not be repeated here.

[0210] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device provided in this embodiment includes at least one processor 23 and a memory 21. Optionally, the electronic device also includes a transceiver 22. The processor 23, the memory 21 and the transceiver 22 are connected via a bus.

[0211] In a specific implementation process, at least one processor 23 executes the computer-executable instructions stored in the memory 21, so that at least one processor 23 executes the above-mentioned text data determination method embodiment.

[0212] The specific implementation process of the processor 23 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0213] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0214] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0215] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0216] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned text data determination method embodiments when running.

[0217] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0218] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned text data determination method embodiments are implemented.

[0219] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned text data determination method embodiments are implemented.

[0220] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0221] The above is a detailed introduction to a text data determination method, electronic device, storage medium and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for determining text data, characterized in that: include: Acquire data to be matched, wherein the data to be matched includes a plurality of view image sets, and any of the view image sets includes image data collected at a plurality of stages under the view; For any view image set, the view image set is input into a text data determination model, the aggregate features corresponding to the view image set are determined, and a matching result is determined based on the aggregate features, wherein the matching result includes a matching text and a matching score that matches the view image set.

2. The text data determination method according to claim 1, characterized in that: The text data determination model includes a three-dimensional visual transformer; The step of inputting the view image set into a text data determination model to determine aggregate features corresponding to the view image set includes: Inputting images of each stage in the view image set into a three-dimensional visual transformer, and extracting tokenized features corresponding to the images of each stage; Using text data to determine the model and determining the global features corresponding to each stage according to the tokenized features corresponding to each stage, the global features are features corresponding to the classification marks in the tokenized features, and the classification marks are learnable vectors; The text data is used to determine the model and the global features corresponding to each stage are preset to obtain the aggregate features corresponding to the view image set.

3. The text data determination method according to claim 2, characterized in that: The pre-processing of the global features corresponding to each stage to obtain the aggregate features corresponding to the view image set includes: According to the preset mask, the global features corresponding to each stage are processed by mean pooling to obtain a first aggregated feature; Performing a linear transformation on the first aggregated feature to obtain a first transformed feature; The first transformed features are normalized to obtain the aggregated features.

4. The text data determination method according to claim 1, characterized in that: The training step of the text data determination model includes: Acquire initial data, the initial data comprising a plurality of initial view image sets and initial text data corresponding to each of the initial view image sets; Determine target data according to the initial data, wherein the target data includes target image data and target text data; Constructing a data set according to the target image data and the target text data; According to a preset loss function, the text data determination model is trained using the data set to obtain a text data determination model that is trained to convergence, wherein the preset loss function includes an image-to-text loss function and a text-to-image loss function.

5. The text data determination method according to claim 4, characterized in that: Determining target data according to the initial data includes: Determine the target image data by adopting a preset data selection strategy and based on the initial view image set; The initial text data is enhanced using a preset data enhancement strategy to obtain the target text data.

6. The text data determination method according to claim 5, characterized in that: The adopting a preset data selection strategy and determining the target image data based on the initial view image set includes: Obtaining a preset stage number, and for any of the initial view image sets, obtaining a corresponding initial stage number; Comparing the preset number of stages with the initial number of stages to obtain a comparison result; If the preset stage number is equal to the initial stage number, determining an image in the initial view image set as the target image data; If the initial stage number is greater than the preset stage number, selecting first image data from the initial view image set so that the stage number of the first image data forming the set is equal to the preset stage number, and determining the first image data as the target image data; If the initial stage number is less than the preset stage number, the initial view image set is filled with zero tensors to obtain second image data, so that the stage number of the second image data forming the set is equal to the preset stage number, and the second image data is determined as the target image data.

7. The text data determination method according to claim 5, characterized in that: The method of adopting a preset data enhancement strategy to enhance the initial text data to obtain the target text data includes: Performing word segmentation processing on the initial text data to obtain an initial text sequence, wherein the initial text sequence contains at least one word segmentation; For any of the participles, obtaining a synonym corresponding to the participle from a predefined dictionary; For the initial text sequence, the segmented words are replaced by the synonyms to obtain a first text sequence; For the initial text sequence, according to a preset exchange strategy, the position of each of the word segments is exchanged to obtain a second text sequence; The initial text sequence, the first text sequence and the second text sequence are determined as the target text data.

8. The text data determination method according to claim 4, characterized in that: The step of training the text data determination model using the data set according to a preset loss function includes: Inputting the target image data in the data set into the three-dimensional visual transformer to obtain the training image aggregation features corresponding to the target image data; Inputting the target text data in the data set into a bidirectional transformer encoder to obtain training text features corresponding to the target text data; Calculating the similarity between the training image aggregate features and the training text features; Determine a comparison loss value according to the similarity, the first loss function and the second loss function; The training parameters in the text data determination model are adjusted based on the contrast loss value until the contrast loss value reaches a minimum, and the text data determination model when the contrast loss value reaches a minimum is determined as the text data determination model.

9. The text data determination method according to claim 8, characterized in that: The step of inputting the training target text data in the data set into a bidirectional transformer encoder to obtain training text features corresponding to the training target text data includes: Inputting the training target text data into the bidirectional transformer encoder to obtain a hidden state sequence; Determine a global text feature according to the hidden state sequence, wherein the global text feature is a feature corresponding to a classification mark in the hidden state sequence; The global text features are normalized to determine the training text features.

10. The text data determination method according to claim 8, characterized in that: The calculating the similarity between the training image aggregate feature and the training text feature includes: Transposing the aggregated features of the training image to obtain transposed aggregated features; The transposed aggregation feature and the training text feature are subjected to a dot product calculation to obtain the similarity.

11. The text data determination method according to claim 8, characterized in that: Determining the contrast loss value according to the similarity, the image-to-text loss function, and the text-to-image loss function includes: Determining a first contrast loss value according to the similarity and the image-to-text loss function; Determining a second contrast loss value according to the similarity and the text-to-image loss function; The contrast loss value is determined according to the first contrast loss value and the second contrast loss value.

12. The text data determination method according to claim 1, characterized in that: Also includes: Receiving a text data query request sent by a user terminal; The obtaining of the data to be matched includes: According to the text data query request, obtaining the data to be matched; After determining the matching result according to the aggregated features, the method further includes: A text data query response is sent to the user terminal, wherein the text data query response includes the matching result.

13. An electronic device, characterized in that: include: Memory, transceivers and processors, The memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; The processor is used to implement the steps of the text data determination method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the text data determination method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the text data determination method according to any one of claims 1 to 12 are implemented.

Images