Image and text matching processing method and electronic equipment

Abstract

The invention provides an image and text matching processing method and electronic equipment, and relates to the technical field of computers, the method comprises the following steps: obtaining an image data set; performing stage filling, multi-view feature extraction and multi-stage feature aggregation processing on the image to obtain an aggregated memory item; extracting a text embedding feature corresponding to the text; performing alignment processing on the aggregated memory items and the embedded features, so as to determine matching values corresponding to the image set and the text according to the aligned memory items and the aligned text embedded features; according to the matching value corresponding to the image set and the text, the matching degree corresponding to the image set and the text is optimized, so that accurate alignment of description matching of the image and the corresponding text is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image and text matching processing method and an electronic device. Background Art

[0002] Images can be medical images, traffic images or other images. Taking medical images as an example, it is a study of the interaction between media such as X-rays and the human body, showing the structure, density and other characteristics of human internal tissues and organs in the form of images. At the same time, the above images and corresponding texts complement each other. Images are used to intuitively represent the image content, while texts are used to describe the corresponding image content.

[0003] In the related art, the image and the corresponding text are integrated to complete the matching of the image and the corresponding text. However, in the related art, it is difficult to cope with complex scenes by integrating the image and the corresponding text, which easily causes the separation problem of matching the image and the corresponding text. Summary of the invention

[0004] The embodiments of the present application provide an image and text matching processing method and an electronic device to cope with complex scenes and achieve the effect of precise alignment of the image and the corresponding text description.

[0005] In a first aspect, an embodiment of the present application provides a method for matching an image and a text, comprising:

[0006] Acquire an image data set, wherein the image data set includes one or more image samples, the image samples include an image set and text, and the image set includes images of one or more stages;

[0007] Performing stage filling processing on the image according to a preset threshold value to obtain a filled image set;

[0008] Performing multiple viewing angle feature extraction processes on the images of each stage in the filled image set to obtain an extracted image set;

[0009] Inputting the extracted image set into a pre-generated gating model to perform multi-stage feature aggregation processing to obtain aggregated memory items;

[0010] Extracting text embedding features corresponding to the text;

[0011] Aligning the aggregated memory items and the embedded features to obtain aligned memory items and aligned text embedded features;

[0012] Determine a matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features;

[0013] According to the matching values ​​of the image set and the text, the matching degree of the image set and the text is optimized.

[0014] In a second aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0017] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementations of the first aspect.

[0018] In a fourth aspect, a computer program product comprises a computer program, wherein when the computer program is executed by a processor, it is used to implement the above first aspect and / or various possible implementation modes of the first aspect.

[0019] The image and text matching processing method and electronic device provided in the embodiments of the present application, the image and text matching processing method provided in the embodiments, by acquiring an image data set, wherein the image data set includes one or more image samples, the image samples include an image set and text, and the image set includes images of one or more stages; performing stage filling processing on the image according to a preset threshold to obtain a filled image set; performing multiple perspective feature extraction processing on the images of each stage in the filled image set to obtain an extracted image set; inputting the extracted image set into a pre-generated gated model for multi-stage feature aggregation processing to obtain an aggregated memory item; extracting text embedding features corresponding to the text; aligning the aggregated memory item and the embedding feature to obtain an aligned memory item and an aligned text embedding feature; determining the matching value corresponding to the image set and the text according to the aligned memory item and the aligned text embedding feature; optimizing the matching degree of the image set and the text according to the matching value corresponding to the image set and the text, so that it can cope with complex scenes and realize accurate alignment of the image and the corresponding text matching, further solving the problem of separation of the image and the corresponding text. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1A schematic diagram of a scene of the image and text matching processing method provided in an embodiment of the present application;

[0022] Figure 2 Schematic diagram of the process of image and text matching processing method provided in the embodiment of the present application Figure 1 ;

[0023] Figure 3 Schematic diagram of the process of image and text matching processing method provided in the embodiment of the present application Figure 2 ;

[0024] Figure 4 A schematic diagram of the structure of an image and text matching processing device provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0026] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0028] An image may be a medical image, a traffic image or other image. Taking medical images as an example, it is a study of the interaction between a certain medium (such as X-rays, electromagnetic fields, ultrasound, etc.) and the human body to express the structure, density and other characteristics of the internal tissues and organs of the human body in the form of images. At the same time, the image and the corresponding text complement each other. The image is used to intuitively represent the image content, while the text is used to describe the corresponding image content. In the related art, the image and the corresponding text are integrated to complete the matching of the image and the corresponding text. However, in the related art, by integrating the image and the corresponding text, it is difficult to cope with complex scenes, which easily causes the separation problem of image and corresponding text matching.

[0029] In order to solve the above technical problems, the embodiments of the present application propose the following technical concepts: the inventor considers the acquired image set, performs multiple perspective feature extraction processes based on the images of one or more stages in the image set to obtain the extracted image set, and inputs the extracted image set into the gated model for multi-stage feature aggregation processing to obtain the aggregated memory items, extracts the text embedding features of the corresponding text, and uses the aligned memory items and the aligned text embedding features to determine the corresponding matching values ​​of the image set and the text to optimize the corresponding matching degree, so that it can cope with complex scenes and achieve accurate alignment of the image and the corresponding text description.

[0030] Figure 1 It is a schematic diagram of an application scenario of the image and text matching processing method provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the scene includes: a display terminal 101 and a computer device 102.

[0032] The display terminal 101 may be a display or other display devices.

[0033] The computer device 102 may be an independent device or a cluster consisting of multiple devices.

[0034] The computer device 102 obtains an image data set, performs stage filling processing on the images according to a preset threshold value to obtain a filled image set; performs multiple view feature extraction processing on the images of each stage in the filled image set to obtain an extracted image set, and inputs the extracted image set into a pre-generated gated model for multi-stage feature aggregation processing to obtain an aggregated memory item; extracts text embedding features corresponding to the text; aligns the aggregated memory items and the embedded features to obtain aligned memory items and aligned text embedding features; determines the matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features; optimizes the matching degree corresponding to the image set and the text according to the matching value corresponding to the image set and the text, and outputs it to the display terminal 101.

[0035] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0036] Figure 2 Schematic diagram of the process of image and text matching processing method provided in the embodiment of the present application Figure 1 , the execution subject of this embodiment can be Figure 1The computer device in the embodiment shown is not particularly limited in this embodiment. Figure 2 As shown, the method includes:

[0037] S201: Acquire an image data set, wherein the image data set includes one or more image samples, the image samples include an image set and text, and the image set includes images of one or more stages.

[0038] In this embodiment, the image may be a medical image, a traffic image or other images.

[0039] In this embodiment, the text is a corresponding medical record report, a traffic penalty statement or other text.

[0040] In this embodiment, the image is a 3D image.

[0041] In this embodiment, the formula for obtaining the image data set, includes:

[0042]

[0043] In the formula, is an image dataset; is the image set in the i-th sample; is the description content of the text in the i-th sample; N is the number of image samples.

[0044] Accordingly, the formula for the image set, includes:

[0045]

[0046] In the formula, represents the medical image set of the i-th sample; For the Medical imaging at each stage; ; is a real number; C is the number of channels; D is the depth of the image; H is the height of the image; W is the width of the image.

[0047] Exemplarily, C corresponds to a channel of a grayscale image of 1.

[0048] S202: Performing stage filling processing on the image according to a preset threshold value to obtain a filled image set.

[0049] Specifically, it is determined whether the number of stages of images in the image set is less than a preset threshold; if the number of stages of images in the image set is less than the preset threshold, the images are stage-filled according to the preset threshold to obtain a filled image set.

[0050] The preset threshold is a preset value k.

[0051] In this embodiment, the image is stage-filled according to a preset threshold value to obtain a filled image set, and the calculation formula includes:

[0052]

[0053] In the formula, is the padded image set corresponding to the i-th sample; The number of images used for the preset threshold; is a zero tensor, ; is a real number; C is the number of channels; D is the depth of the image; H is the height of the image; W is the width of the image.

[0054] S203: performing multiple viewing angle feature extraction processes on the images of each stage in the filled image set to obtain an extracted image set.

[0055] Specifically, multiple perspective feature extraction processes are performed on the images of each stage in the filled image set according to the visual coding network to obtain the extracted image set.

[0056] Exemplarily, the visual encoding network is ViT (Visual Transformer) or 3D-CNN (Three-Dimensional Convolutional Neural Network).

[0057] In this embodiment, multiple viewing angle feature extraction processes are performed on the image sets of each stage in the filled image set to obtain the extracted image set, and the calculation formula includes:

[0058]

[0059] In the formula, is the extracted image set; It is a collection of images for each stage; ; is a real number; H is the dimension of hidden features.

[0060] S204: Input the extracted image set into the pre-generated gating model for multi-stage feature aggregation processing to obtain aggregated memory items.

[0061] In this embodiment, the initial memory corresponding to the aggregated memory item is defined as: M0=F1, where if the image set of the first stage is not available, it is initialized to zero.

[0062] S205: Extract text embedding features corresponding to the text.

[0063] Specifically, the text embedding features corresponding to the text are extracted through the text encoder.

[0064] In this embodiment, the calculation formula for extracting the text embedding feature corresponding to the text includes:

[0065]

[0066] In the formula, Embed features for text; is a real number; L is the length of the text sequence; H is the embedding dimension.

[0067] S206: Align the aggregated memory items and embedded features to obtain aligned memory items and aligned text embedded features.

[0068] In this embodiment, the aggregated memory items and text embedding features are aligned to obtain the aligned memory items and the aligned text embedding features, and the calculation formula includes:

[0069]

[0070] In the formula, is the memory item after alignment, is determined based on the aggregated memory items; the aggregated memory items are determined by the gating model Obtained through multi-stage feature aggregation processing; Embed features for aligned text;

[0071] S207: Determine the matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features.

[0072] In this embodiment, the calculation formula for determining the matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features includes:

[0073]

[0074] In the formula, is the matching value between the image set and the text; is the number of image samples; is the similarity function; is the memory item after alignment; is the i-th aligned text embedding feature; is the jth aligned text embedding feature.

[0075] S208: Optimizing the matching degree between the image set and the text according to the matching value between the image set and the text.

[0076] Specifically, according to the matching values ​​of the image set and the text, a contrastive learning loss method or a cross entropy loss method is used to optimize the matching degree of the image set and the text.

[0077] In summary, the image and text matching processing method provided in the present embodiment is by obtaining an image data set, wherein the image data set includes one or more image samples, the image samples include an image set and text, and the image set includes images of one or more stages; performing stage filling processing on the image according to a preset threshold to obtain a filled image set; performing multiple perspective feature extraction processing on the images of each stage in the filled image set to obtain an extracted image set; inputting the extracted image set into a pre-generated gated model for multi-stage feature aggregation processing to obtain an aggregated memory item; extracting text embedding features corresponding to the text; aligning the aggregated memory items and the embedding features to obtain aligned memory items and aligned text embedding features; determining the matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features; optimizing the matching degree of the image set and the text according to the matching value corresponding to the image set and the text, so that it can cope with complex scenes and realize accurate alignment of the image and the corresponding text matching, further solving the problem of separation of the image and the corresponding text.

[0078] In addition, the image and text matching processing method provided in this embodiment aligns the aggregated memory items and embedded features to obtain aligned memory items and aligned text embedded features, thereby realizing semantic mapping and precise association between the image and text to ensure maximization of matching similarity.

[0079] In addition, the image and text matching processing method provided in this embodiment performs stage filling processing on the image according to a preset threshold to obtain a filled image set, by filling in the samples with insufficient number of stages and adopting a dynamic selection strategy based on maximizing diversity and semantic relevance, to ensure that the core stage information is fully retained while reducing the interference of redundant stages.

[0080] In addition, the image and text matching processing method provided in this embodiment, through the combination of dynamic gating mechanism and contrastive learning, enables the gating model to learn compact and discriminative feature representation in the multimodal embedding space, thereby improving the accuracy and robustness of image and text alignment.

[0081] In addition, the image and text matching processing method provided in this embodiment can simultaneously capture the spatial correlation between different anatomical planes (such as axial, sagittal, and coronal) of the image and the temporal dynamics between image stages by using a visual encoder to extract multi-image features. The gating mechanism further enhances the dynamic fusion capability of key spatiotemporal semantic information and provides comprehensive image understanding capabilities for the gating model.

[0082] In addition, the image and text matching processing method provided in this embodiment can adapt to the diagnosis and auxiliary decision-making needs in different clinical scenarios.

[0083] Figure 3 Schematic diagram of the process of image and text matching processing method provided in the embodiment of the present application Figure 2 In the embodiment of the present application, Figure 2 Based on the embodiment provided, the specific implementation method of the process of generating the gated model in step S205 is described in detail. Figure 3 As shown, the method includes:

[0084] S301: extracting images of multiple stages from any image sample in an image data set, wherein the image of each stage includes corresponding multiple viewing angle features.

[0085] S302: extracting and processing each viewing angle feature to obtain a corresponding first feature extraction result, a second feature extraction result and a valid viewing angle mask.

[0086] In this embodiment, each viewing angle feature is extracted and processed to obtain the corresponding first feature extraction result, second feature extraction result and effective viewing angle mask, and the calculation formula includes:

[0087]

[0088] In the formula, is the first feature extraction result; is the visual feature tensor of the t-th perspective; is a real number; is the batch size; is the dimension of the hidden features.

[0089]

[0090] In the formula, is the second feature extraction result; is the previous memory tensor;

[0091]

[0092] In the formula, is the effective viewing angle mask.

[0093] In this embodiment, the second feature extraction result is old memory.

[0094] Among them, the first feature extraction result can be extracted from different scanning angles or modalities (such as CT, MRI).

[0095] Among them, CT stands for computed tomography; MRI stands for magnetic resonance imaging.

[0096] The second feature extraction result is image information of a previous perspective extracted from the memory unit storage of the gated model.

[0097] Among them, the valid viewing angle mask is used to mark certain invalid or missing viewing angles, wherein the invalid or missing viewing angles are the unimaged areas of the patient's specific area.

[0098] S303: Determine a final memory update item according to the first feature extraction result and the second feature extraction result.

[0099] Specifically, step S303 includes:

[0100] S3031: Determine a reset gate vector and an update gate vector according to the first feature extraction result and the second feature extraction result.

[0101] In this embodiment, the calculation formulas for resetting the gate vector and updating the gate vector are determined according to the first feature extraction result and the second feature extraction result, including:

[0102]

[0103] In the formula, To update the gate vector; is the reset gate vector; is the first feature extraction result; is the second feature extraction result; is the activation function; is a learnable convolution kernel; * is a two-dimensional convolution operation.

[0104] Among them, the update gate vector and the reset gate vector dynamically filter information through weight adjustment, thereby ensuring that the gating model can efficiently utilize important information when processing multi-view features.

[0105] Among them, the update gate vector and the reset gate vector are calculated by the Sigmoid activation function σ. The role of the Sigmoid function is to map the input value to the [0,1] interval, thereby providing an appropriate proportional range for the weight.

[0106] in, is a learnable convolution kernel with the shape , where K is the size of the convolution kernel.

[0107] For example, K=3 represents a 3\times3 convolution kernel.

[0108] S3032: Determine a corresponding candidate memory value according to the reset gate vector, the first feature extraction result and the second feature extraction result.

[0109] In this embodiment, the calculation formula for determining the corresponding candidate memory value according to the reset gate vector, the first feature extraction result and the second feature extraction result includes:

[0110]

[0111] In the formula, is the candidate memory value; is the hyperbolic tangent activation function; is the preset parameter; is the first feature extraction result; is the reset gate vector; is element-wise multiplication; is the second feature extraction result; To reset the memory.

[0112] Among them, the candidate memory is generated by combining the first feature extraction result and the reset memory. The purpose of this step is to effectively combine the current input information with the historical memory information to construct a candidate memory state. The reset operation here is controlled by the reset gate vector and implemented by element-level multiplication of the second feature extraction result. This operation can dynamically adjust the contribution of historical information.

[0113] In addition, in order to enhance the nonlinear expression ability of candidate memories, the hyperbolic tangent activation function is used , which maps the input value to Within a certain range, the model is given stronger feature modeling capabilities to capture complex patterns and better adapt to the changes and correlations of multi-view features.

[0114] S3033: Determine a corresponding memory update item according to the update gate vector, the second feature extraction result and the candidate memory value.

[0115] In this embodiment, the calculation formula of the corresponding memory update item is determined according to the update gate vector, the second feature extraction result and the candidate memory value, including:

[0116]

[0117] In the formula, is the memory update item; To update the gate vector; is element-wise multiplication; is the second feature extraction result; is the candidate memory value.

[0118] The memory update term is obtained by weighted combination of the second feature extraction result and the candidate memory value. This weighting process is dynamically controlled by the update gate vector, where the value of the update gate vector determines the degree of memory update at the current moment. When the update gate vector is close to 1, it is more inclined to introduce the information of the candidate memory value; when the update gate vector is close to 0, it is more inclined to retain the old memory corresponding to the second feature extraction result. This weighting mechanism allows the gated model to dynamically adjust the information flow from different perspectives, thereby effectively balancing the contribution of historical information and new input information during the memory update process.

[0119] S3034: Normalize the memory update items to obtain preliminarily normalized memory update items.

[0120] Specifically, step S3034 includes:

[0121] S30341: Obtain the dimensions and feature values ​​of the hidden features corresponding to the memory update items.

[0122] S30342: Determine the mean corresponding to the memory update item according to the dimension of the hidden feature and each feature value.

[0123] In this embodiment, the calculation formula for determining the mean value corresponding to the memory update item according to the dimension of the hidden feature and each feature value includes:

[0124]

[0125] In the formula, is the mean; is the dimension of hidden features; For the The value of a feature.

[0126] For example, .

[0127] S30343: Determine the variance corresponding to the memory update item according to the dimension, each eigenvalue and the mean of the hidden feature.

[0128] In this embodiment, the calculation formula of the variance corresponding to the memory update item is determined according to the dimension, each eigenvalue and the mean of the hidden feature, including:

[0129]

[0130] In the formula, is the variance; is the dimension of hidden features; For the The value of the feature is the mean.

[0131] S30344: Perform normalization processing based on the memory update item, mean and variance to obtain a preliminary normalized memory update item.

[0132] In this embodiment, normalization is performed according to the memory update term, the mean, and the variance to obtain a calculation formula for a preliminary normalized memory update term, including:

[0133]

[0134] In the formula, is the initial normalized memory update item; is the memory update item; For the The value of the feature is the mean; is the variance; is a constant used for numerical stability to avoid zero denominator.

[0135] The variance is used to measure the range of variation of memory in the hidden dimension.

[0136] S3035: Perform linear transformation processing on the initially normalized memory update items to obtain final normalized memory update items.

[0137] In this embodiment, a linear transformation is performed on the preliminary normalized memory update term to obtain a final normalized memory update term, and the calculation formula includes:

[0138]

[0139] In the formula, is the final normalized memory update item; is the initial normalized memory update item; and are learnable parameters.

[0140] Among them, the shapes of γ and β are Their role is to restore the feature distribution of the model before normalization, while retaining the numerical stability and optimization efficiency brought by normalization, so as to ensure that the updated memory has both good numerical stability and the ability to flexibly adjust the feature distribution.

[0141] S3036: Perform residual connection processing on the final normalized memory update item according to the first feature extraction result to obtain the final memory update item.

[0142] In this embodiment, the final normalized memory update item is subjected to residual connection processing according to the first feature extraction result to obtain the final memory update item, and the calculation formula thereof includes:

[0143]

[0144] In the formula, is the final memory update item; is the final normalized memory update item; are the parameters obtained by pre-training; This is the first feature extraction result.

[0145] In this embodiment, the residual connection is used to Join directly to , thus forming a skip connection. This operation can effectively prevent information from being gradually lost in multiple nonlinear transformations, ensuring that the information of the original input features can be directly passed to the subsequent layers. At the same time, the residual connection can also alleviate the problem of gradient disappearance in deep networks, so as to improve the training efficiency of the model and the stability of feature expression.

[0146] S304: Processing the effective viewing angle mask according to the dimension of the hidden feature to obtain a processed effective viewing angle mask.

[0147] Specifically, step S304 includes:

[0148] S3041: Expanding the effective viewing angle mask according to the dimension of the hidden feature to obtain an expanded effective viewing angle mask;

[0149] In this embodiment, the effective viewing angle mask is expanded according to the dimension of the hidden feature to obtain the expanded effective viewing angle mask, and the calculation formula includes:

[0150]

[0151] In the formula, is the effective viewing angle mask after expansion; is the effective viewing angle mask; the shape is ; is the batch size.

[0152] in, Is a binary vector, where 1 indicates that the view is available and 0 indicates that the view is invalid. Through expansion and broadcasting, the mask is applied to all feature dimensions, ensuring that invalid views do not contribute to the results in subsequent calculations.

[0153] S3042: Perform broadcast processing on the expanded effective viewing angle mask to obtain a processed effective viewing angle mask.

[0154] In this embodiment, the expanded effective viewing angle mask is broadcasted to obtain a processed effective viewing angle mask, including:

[0155]

[0156] In the formula, is the effective viewing angle mask after processing; The expanded effective viewing angle mask; For a length of The row vector of is composed entirely of 1s; the shape is ; is the batch size; is the dimension of the hidden features.

[0157] in, The role of is to exclude the information of invalid perspectives, thereby avoiding its interference with the model.

[0158] S305: Determine a mask reset gate vector and a mask update gate vector according to the first feature extraction result, the final memory update item, and the processed effective viewing angle mask.

[0159] In this embodiment, the calculation formula of determining the mask reset gate vector and the mask update gate vector according to the first feature extraction result, the final memory update item and the processed effective viewing angle mask includes:

[0160]

[0161] In the formula, is the update gate vector of the mask; is the reset gate vector of the mask; is the first feature extraction result; is the second feature extraction result; is the activation function; is a learnable convolution kernel; is the effective viewing angle mask after processing.

[0162] S306: Determine a memory update item of the corresponding mask according to the reset gate vector of the mask, the update gate vector of the mask, the final memory update item, the first feature extraction result, and the processed effective viewing angle mask.

[0163] Specifically, step S306 includes:

[0164] S3061: Determine candidate memory values ​​of the mask according to the reset gate vector of the mask, the final memory update item, the first feature extraction result, and the processed effective viewing angle mask.

[0165] In this embodiment, the candidate memory value of the mask is determined according to the reset gate vector of the mask, the final memory update item, the first feature extraction result, and the processed effective viewing angle mask, and the calculation formula includes:

[0166]

[0167] In the formula, is the candidate memory value of the mask; is the hyperbolic tangent activation function; is the preset parameter; is the first feature extraction result; is the reset gate vector of the mask; is element-wise multiplication; is the final memory update item.

[0168] The formula implements dynamic memory update by weighted combination of old memory and candidate memory. The update gate vector ensures that the model can adjust the degree of memory update according to the importance of the current feature. This ensures that features from invalid perspectives do not participate in memory updates.

[0169] S3062: Determine the memory update item corresponding to the mask according to the update gate vector of the mask, the final memory update item, and the candidate memory value of the mask.

[0170] In this embodiment, the calculation formula for determining the memory update item corresponding to the mask according to the update gate vector of the mask, the final memory update item, and the candidate memory value of the mask includes:

[0171]

[0172] In the formula, is the memory update item of the mask; a proportionate portion reserved for old memories; The proportionate portion introduced for candidate memory; is the update gate vector of the mask; is the final memory update item; is element-wise multiplication; Candidate memory values ​​for the mask.

[0173] S307: Determine the masked memory update item as a gated model.

[0174] In summary, the image and text matching processing method provided in this embodiment extracts images of multiple stages from any image sample of an image data set, wherein the images of each stage include corresponding multiple viewing angle features; extracts and processes each viewing angle feature to obtain a corresponding first feature extraction result, a second feature extraction result, and an effective viewing angle mask; determines a final memory update item based on the first feature extraction result and the second feature extraction result; processes the effective viewing angle mask based on the dimension of the hidden feature to obtain a processed effective viewing angle mask; determines a mask reset gate vector and a mask update gate vector based on the first feature extraction result, the final memory update item, and the processed effective viewing angle mask; determines a corresponding mask memory update item based on the mask reset gate vector, the mask update gate vector, the final memory update item, the first feature extraction result, and the processed effective viewing angle mask; determines the mask memory update item as a gated model, which can effectively capture dynamic changes between stages while suppressing interference from noise and redundant features, and is particularly suitable for integrating multi-stage images reflecting temporal characteristics such as contrast agent distribution and lesion dynamics.

[0175] In addition, the image and text matching processing method provided in this embodiment dynamically adjusts and updates the gate vector and resets the gate vector through a gating mechanism, and accurately screens important information and suppresses noise and invalid features while gradually integrating features from different perspectives and stages.

[0176] In addition, the image and text matching processing method provided in this embodiment uses the validity mask to control the interference of invalid viewpoints, thereby realizing efficient fusion and dynamic update of multi-view image features. In addition, the validity mask can also ensure that the features of invalid viewpoints or stages will not affect memory updates through element-by-element operations, thereby avoiding the interference of data heterogeneity and missing information on model performance.

[0177] Figure 4 A schematic diagram of the structure of the image and text matching processing device provided in the embodiment of the present application, such as Figure 4 As shown, the image and text matching processing device provided in this embodiment includes: an acquisition module 401, a filling module 402, a first extraction module 403, an aggregation module 404, a second extraction module 405, an alignment module 406, a first determination module 407 and an optimization module 408.

[0178] An acquisition module 401 is used to acquire an image data set, wherein the image data set includes one or more image samples, the image samples include an image set and text, and the image set includes images of one or more stages;

[0179] A filling module 402 performs a stage filling process on the image according to a preset threshold value to obtain a filled image set;

[0180] A first extraction module 403 is used to perform multiple viewing angle feature extraction processes on the images of each stage in the filled image set to obtain an extracted image set;

[0181] Aggregation module 404, used for inputting the extracted image set into a pre-generated gating model to perform multi-stage feature aggregation processing to obtain aggregated memory items;

[0182] A second extraction module 405, used to extract text embedding features corresponding to the text;

[0183] An alignment module 406, configured to align the aggregated memory items and the embedded features to obtain aligned memory items and aligned text embedded features;

[0184] A first determination module 407, configured to determine a matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features;

[0185] The optimization module 408 is used to optimize the matching degree between the image set and the text according to the matching value between the image set and the text.

[0186] In a possible implementation, the formula for obtaining the image data set includes:

[0187]

[0188] In the formula, is an image dataset; is the image set in the i-th sample; is the description content of the text in the i-th sample; N is the number of image samples;

[0189] Accordingly, the formula of the image set, includes:

[0190]

[0191] In the formula, represents the medical image set of the i-th sample; For the Medical imaging at each stage; ; is a real number; C is the number of channels; D is the depth of the image; H is the height of the image; W is the width of the image;

[0192] Accordingly, the image is subjected to stage filling processing according to a preset threshold value to obtain a filled image set, and the calculation formula includes:

[0193]

[0194] In the formula, is the padded image set corresponding to the i-th sample; The number of images used for the preset threshold; is a zero tensor, ; is a real number; C is the number of channels; D is the depth of the image; H is the height of the image; W is the width of the image.

[0195] In a possible implementation, the calculation formula for performing multiple viewing angle feature extraction processes on the image sets of each stage in the filled image set to obtain the extracted image set includes:

[0196]

[0197] In the formula, is the extracted image set; It is a collection of images for each stage; ; is a real number; H is the dimension of hidden features;

[0198] Accordingly, the calculation formula for extracting the text embedding feature corresponding to the text includes:

[0199]

[0200] In the formula, Embed features for text; is a real number; L is the length of the text sequence; H is the embedding dimension;

[0201] Accordingly, the calculation formula for aligning the aggregated memory items and the text embedding features to obtain aligned memory items and aligned text embedding features includes:

[0202]

[0203] In the formula, is the memory item after alignment, is determined according to the aggregated memory items; the aggregated memory items are determined by the gating model Obtained through multi-stage feature aggregation processing; Embed features for aligned text;

[0204] Accordingly, the calculation formula for determining the matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features includes:

[0205]

[0206] In the formula, is the matching value corresponding to the image set and the text; is the number of image samples; is the similarity function; is the memory item after alignment; is the i-th aligned text embedding feature; is the jth aligned text embedding feature.

[0207] In a possible implementation manner, the device further includes:

[0208] A third extraction module 409 is used to extract images of multiple stages from any image sample in the image data set, wherein the image of each stage includes corresponding multiple viewing angle features;

[0209] The fourth extraction module 410 is used to extract and process each viewing angle feature to obtain a corresponding first feature extraction result, a second feature extraction result and an effective viewing angle mask;

[0210] A second determination module 411, configured to determine a final memory update item according to the first feature extraction result and the second feature extraction result;

[0211] A processing module 412, configured to process the effective viewing angle mask according to the dimension of the hidden feature to obtain a processed effective viewing angle mask;

[0212] A third determination module 413, configured to determine a mask reset gate vector and a mask update gate vector according to the first feature extraction result, the final memory update item, and the processed effective viewing angle mask;

[0213] a fourth determination module 414, configured to determine a memory update item of a corresponding mask according to the reset gate vector of the mask, the update gate vector of the mask, the final memory update item, the first feature extraction result, and the processed effective viewing angle mask;

[0214] The fifth determination module 415 is used to determine the memory update item of the mask as a gated model.

[0215] In a possible implementation, the extraction process of each viewing angle feature to obtain the corresponding first feature extraction result, second feature extraction result and effective viewing angle mask is performed, and the calculation formula includes:

[0216]

[0217] In the formula, is the first feature extraction result; is the visual feature tensor of the t-th perspective; is a real number; is the batch size; is the dimension of hidden features;

[0218]

[0219] In the formula, is the second feature extraction result; is the previous memory tensor;

[0220]

[0221] In the formula, is the effective viewing angle mask.

[0222] In a possible implementation, the calculation formula for determining the mask reset gate vector and the mask update gate vector according to the first feature extraction result, the final memory update item, and the processed effective viewing angle mask includes:

[0223]

[0224] In the formula, is the update gate vector of the mask; is the reset gate vector of the mask; is the first feature extraction result; is the second feature extraction result; is the activation function; is a learnable convolution kernel; is the effective viewing angle mask after processing.

[0225] In a possible implementation, the second determining module 411 specifically includes:

[0226] A first determining unit 4111 is used to determine a reset gate vector and an update gate vector according to the first feature extraction result and the second feature extraction result;

[0227] A second determining unit 4112, configured to determine a corresponding candidate memory value according to a reset gate vector, the first feature extraction result, and the second feature extraction result;

[0228] A third determining unit 4113, configured to determine a corresponding memory update item according to the update gate vector, the second feature extraction result and the candidate memory value;

[0229] A normalization unit 4114, configured to perform normalization processing on the memory update item to obtain a preliminarily normalized memory update item;

[0230] A transformation unit 4115, configured to perform a linear transformation process on the preliminary normalized memory update item to obtain a final normalized memory update item;

[0231] The connection unit 4116 is used to perform residual connection processing on the final normalized memory update item according to the first feature extraction result to obtain a final memory update item.

[0232] In a possible implementation, the calculation formula for determining the reset gate vector and the update gate vector according to the first feature extraction result and the second feature extraction result includes:

[0233]

[0234] In the formula, To update the gate vector; is the reset gate vector; is the first feature extraction result; is the second feature extraction result; is the activation function; is a learnable convolution kernel;

[0235] Accordingly, the calculation formula for determining the corresponding candidate memory value according to the reset gate vector, the first feature extraction result and the second feature extraction result includes:

[0236]

[0237] In the formula, is the candidate memory value; is the hyperbolic tangent activation function; is the preset parameter; is the first feature extraction result; is the reset gate vector; is element-wise multiplication; is the second feature extraction result;

[0238] Accordingly, the calculation formula for determining the corresponding memory update item according to the update gate vector, the second feature extraction result and the candidate memory value includes:

[0239]

[0240] In the formula, is the memory update item; To update the gate vector; is element-wise multiplication; is the second feature extraction result; is the candidate memory value;

[0241] Accordingly, the calculation formula for performing linear transformation processing on the preliminary normalized memory update term to obtain the final normalized memory update term includes:

[0242]

[0243] In the formula, is the final normalized memory update item; is the initial normalized memory update item; and is a learnable parameter;

[0244] Accordingly, the final normalized memory update term is subjected to residual connection processing according to the first feature extraction result to obtain a final memory update term, the calculation formula of which includes:

[0245]

[0246] In the formula, is the final memory update item; is the final normalized memory update item; are the parameters obtained by pre-training; This is the first feature extraction result.

[0247] In a possible implementation, the normalization unit 4114 specifically includes:

[0248] An acquisition unit 41141 is used to acquire the dimension and each feature value of the hidden feature corresponding to the memory update item;

[0249] A first determining unit 41142, configured to determine a mean value corresponding to the memory update item according to the dimension of the hidden feature and each feature value;

[0250] A second determining unit 41143 is used to determine the variance corresponding to the memory update item according to the dimension of the hidden feature, the feature values ​​and the mean;

[0251] The normalization unit 41144 is used to perform normalization processing according to the memory update item, the mean and the variance to obtain a preliminarily normalized memory update item.

[0252] In a possible implementation, the calculation formula for determining the mean value corresponding to the memory update item according to the dimension of the hidden feature and the feature values ​​includes:

[0253]

[0254] In the formula, is the mean; is the dimension of hidden features; For the The value of a feature;

[0255] Accordingly, the calculation formula for determining the variance corresponding to the memory update item according to the dimension of the hidden feature, the eigenvalues ​​and the mean value includes:

[0256]

[0257] In the formula, is the variance; is the dimension of hidden features; For the The value of the feature is the mean;

[0258] Accordingly, normalization is performed according to the memory update term, the mean value and the variance to obtain a calculation formula of a preliminary normalized memory update term, including:

[0259]

[0260] In the formula, is the initial normalized memory update item; is the memory update item; For the The value of the feature is the mean; is the variance; is a constant used for numerical stability to avoid zero denominator.

[0261] In a possible implementation, the processing module 412 specifically includes:

[0262] An expansion unit 4121, configured to expand the effective viewing angle mask according to the dimension of the hidden feature to obtain an expanded effective viewing angle mask;

[0263] The processing unit 4122 is used to perform broadcast processing on the extended effective viewing angle mask to obtain a processed effective viewing angle mask.

[0264] In a possible implementation, the effective viewing angle mask is expanded according to the dimension of the hidden feature to obtain a calculation formula for the expanded effective viewing angle mask, including:

[0265]

[0266] In the formula, is the effective viewing angle mask after expansion; is the effective viewing angle mask; the shape is ; is the batch size;

[0267] Accordingly, the broadcasting process is performed on the expanded effective viewing angle mask to obtain a processed effective viewing angle mask, including:

[0268]

[0269] In the formula, is the effective viewing angle mask after processing; The expanded effective viewing angle mask; For a length of The row vector of is composed entirely of 1s; the shape is ; is the batch size; is the dimension of the hidden features.

[0270] The image and text matching processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0271] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0272] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0273] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0274] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0275] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0276] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0277] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0278] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0279] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0280] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0281] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0282] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0283] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0284] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0285] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0286] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application, are not limited to the precise structures described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for matching an image and a text, characterized in that: include: Acquire an image data set, wherein the image data set includes one or more image samples, the image samples include an image set and text, and the image set includes images of one or more stages; Performing stage filling processing on the image according to a preset threshold value to obtain a filled image set; Performing multiple viewing angle feature extraction processes on the images of each stage in the filled image set to obtain an extracted image set; Inputting the extracted image set into a pre-generated gating model to perform multi-stage feature aggregation processing to obtain aggregated memory items; Extracting text embedding features corresponding to the text; Aligning the aggregated memory items and the embedded features to obtain aligned memory items and aligned text embedded features; Determine a matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features; According to the matching values ​​of the image set and the text, the matching degree of the image set and the text is optimized.

2. The method according to claim 1, characterized in that The formula for obtaining the image data set includes: In the formula, is an image dataset; is the image set in the i-th sample; is the description content of the text in the i-th sample; N is the number of image samples; Accordingly, the formula of the image set, includes: In the formula, represents the medical image set of the i-th sample; For the Medical imaging at each stage; ; is a real number; C is the number of channels; D is the depth of the image; H is the height of the image; W is the width of the image; Accordingly, the image is subjected to stage filling processing according to a preset threshold value to obtain a filled image set, and the calculation formula includes: In the formula, is the padded image set corresponding to the i-th sample; The number of images used for the preset threshold; is a zero tensor, ; is a real number; C is the number of channels; D is the depth of the image; H is the height of the image; W is the width of the image.

3. The method according to claim 1, characterized in that The calculation formula for performing multiple viewing angle feature extraction processes on the image sets of each stage in the filled image set to obtain the extracted image set includes: In the formula, is the extracted image set; It is a collection of images for each stage; ; is a real number; H is the dimension of hidden features; Accordingly, the calculation formula for extracting the text embedding feature corresponding to the text includes: In the formula, Embed features for text; is a real number; L is the length of the text sequence; H is the embedding dimension; Accordingly, the calculation formula for aligning the aggregated memory items and the text embedding features to obtain aligned memory items and aligned text embedding features includes: In the formula, is the memory item after alignment, is determined according to the aggregated memory items; the aggregated memory items are determined by the gating model Obtained through multi-stage feature aggregation processing; Embed features for aligned text; Accordingly, the calculation formula for determining the matching value corresponding to the image set and the text according to the aligned memory items and the aligned text embedding features includes: In the formula, is the matching value corresponding to the image set and the text; is the number of image samples; is the similarity function; is the memory item after alignment; is the i-th aligned text embedding feature; is the jth aligned text embedding feature.

4. The method according to claim 1, characterized in that The generation process of the gating model includes: Extracting images of multiple stages from any image sample in the image data set, wherein the image of each stage includes corresponding multiple viewing angle features; Extracting and processing each viewing angle feature to obtain a corresponding first feature extraction result, a second feature extraction result, and an effective viewing angle mask; Determining a final memory update item according to the first feature extraction result and the second feature extraction result; Processing the effective viewing angle mask according to the dimension of the hidden feature to obtain a processed effective viewing angle mask; Determining a mask reset gate vector and a mask update gate vector according to the first feature extraction result, the final memory update item, and the processed effective viewing angle mask; Determining a memory update item of a corresponding mask according to a reset gate vector of the mask, an update gate vector of the mask, the final memory update item, the first feature extraction result, and the processed effective viewing angle mask; A memory update term of the mask is determined as a gated model.

5. The method according to claim 4, characterized in that The calculation formula for extracting and processing each viewing angle feature to obtain the corresponding first feature extraction result, second feature extraction result and effective viewing angle mask includes: In the formula, is the first feature extraction result; is the visual feature tensor of the t-th perspective; is a real number; is the batch size; is the dimension of hidden features; In the formula, is the second feature extraction result; is the previous memory tensor; In the formula, is the effective viewing angle mask.

6. The method according to claim 4, characterized in that The calculation formula for determining the mask reset gate vector and the mask update gate vector according to the first feature extraction result, the final memory update item and the processed effective viewing angle mask includes: In the formula, is the update gate vector of the mask; is the reset gate vector of the mask; is the first feature extraction result; is the second feature extraction result; is the activation function; is a learnable convolution kernel; is the effective viewing angle mask after processing.

7. The method according to claim 4, characterized in that The determining a final memory update item according to the first feature extraction result and the second feature extraction result includes: Determine a reset gate vector and an update gate vector according to the first feature extraction result and the second feature extraction result; Determine a corresponding candidate memory value according to the reset gate vector, the first feature extraction result, and the second feature extraction result; Determine a corresponding memory update item according to the update gate vector, the second feature extraction result and the candidate memory value; Normalizing the memory update item to obtain a preliminarily normalized memory update item; Performing a linear transformation process on the preliminary normalized memory update item to obtain a final normalized memory update item; The final normalized memory update item is subjected to residual connection processing according to the first feature extraction result to obtain a final memory update item.

8. The method according to claim 7, characterized in that The calculation formula for determining the reset gate vector and the update gate vector according to the first feature extraction result and the second feature extraction result includes: In the formula, To update the gate vector; is the reset gate vector; is the first feature extraction result; is the second feature extraction result; is the activation function; is a learnable convolution kernel; Accordingly, the calculation formula for determining the corresponding candidate memory value according to the reset gate vector, the first feature extraction result and the second feature extraction result includes: In the formula, is the candidate memory value; is the hyperbolic tangent activation function; is the preset parameter; is the first feature extraction result; is the reset gate vector; is element-wise multiplication; is the second feature extraction result; Accordingly, the calculation formula for determining the corresponding memory update item according to the update gate vector, the second feature extraction result and the candidate memory value includes: In the formula, is the memory update item; To update the gate vector; is element-wise multiplication; is the second feature extraction result; is the candidate memory value; Accordingly, the calculation formula for performing linear transformation processing on the preliminary normalized memory update term to obtain the final normalized memory update term includes: In the formula, is the final normalized memory update item; is the initial normalized memory update item; and is a learnable parameter; Accordingly, the final normalized memory update term is subjected to residual connection processing according to the first feature extraction result to obtain a final memory update term, the calculation formula of which includes: In the formula, is the final memory update item; is the final normalized memory update item; are the parameters obtained by pre-training; This is the first feature extraction result.

9. The method according to claim 7, characterized in that: The normalizing the memory update item to obtain a preliminarily normalized memory update item includes: Obtaining the dimension and each feature value of the hidden feature corresponding to the memory update item; Determining a mean value corresponding to the memory update item according to the dimension of the hidden feature and each feature value; Determining the variance corresponding to the memory update item according to the dimension of the hidden feature, the feature values ​​and the mean; Normalization processing is performed according to the memory update item, the mean value and the variance to obtain a preliminarily normalized memory update item.

10. The method according to claim 9, characterized in that The calculation formula for determining the mean value corresponding to the memory update item according to the dimension of the hidden feature and the characteristic values ​​includes: In the formula, is the mean; is the dimension of hidden features; For the The value of a feature; Accordingly, the calculation formula for determining the variance corresponding to the memory update item according to the dimension of the hidden feature, the eigenvalues ​​and the mean value includes: In the formula, is the variance; is the dimension of hidden features; For the The value of the feature is the mean; Accordingly, normalization is performed according to the memory update term, the mean value and the variance to obtain a calculation formula of a preliminary normalized memory update term, including: In the formula, is the initial normalized memory update item; is the memory update item; For the The value of the feature is the mean; is the variance; is a constant used for numerical stability to avoid zero denominator.

11. The method according to claim 4, characterized in that The processing of the effective viewing angle mask according to the dimension of the hidden feature to obtain a processed effective viewing angle mask includes: Expanding the effective viewing angle mask according to the dimension of the hidden feature to obtain an expanded effective viewing angle mask; The expanded effective viewing angle mask is broadcasted to obtain a processed effective viewing angle mask.

12. The method according to claim 11, characterized in that The calculation formula for expanding the effective viewing angle mask according to the dimension of the hidden feature to obtain the expanded effective viewing angle mask includes: In the formula, is the effective viewing angle mask after expansion; is the effective viewing angle mask; the shape is ; is the batch size; Accordingly, the broadcasting process is performed on the expanded effective viewing angle mask to obtain a processed effective viewing angle mask, including: In the formula, is the effective viewing angle mask after processing; The expanded effective viewing angle mask; For a length of The row vector of is composed entirely of 1s; the shape is ; is the batch size; is the dimension of the hidden features.

13. An electronic device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the image and text matching processing method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the image and text matching processing method as described in any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image and text matching processing method described in any one of claims 1 to 12 is implemented.

Images