Intelligent matching method, device and equipment based on text and image

Abstract

The application relates to an artificial intelligence technology and discloses an intelligent matching method and device based on text and image and equipment, which can be applied to application scenarios such as medical diagnosis. The method comprises the following steps: extracting text features of a user's search text, analyzing description features of the search text, fusing the description features and the text features to generate target text features; querying a text image to be matched from a pre-constructed text image library, extracting image features of the text image, performing semantic recognition on the image features to obtain image semantic features; recognizing a text category of the target text features and an image category of the image semantic features; detecting a text global feature of the target text features based on the text category and detecting an image global feature of the image semantic features based on the image category; and calculating a matching probability of the text global feature and the image global feature to obtain a matching result of the search text and the text image. The application can improve the matching efficiency between the text and the image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, and device for intelligent matching between text and images. Background Technology

[0002] With the rapid development of computer vision and neural networks, people often search for the images they want from a vast amount of online resources by inputting a short text description or keywords. This process of retrieving matching images based on text content is essentially an emerging information retrieval technology with wide applications in life, such as finding auxiliary diagnostic information for patients' medical images in medical auxiliary diagnosis.

[0003] Currently, the usual text-image retrieval and matching is limited to a single modality, that is, retrieving matching text images through text content. To complete the text-image retrieval and matching, a large number of images to be matched need to be manually labeled in advance, which consumes a lot of manpower and time. Furthermore, it cannot intelligently recognize the semantics of the image to match the retrieved text, which greatly reduces the efficiency of text-image matching in medical auxiliary diagnosis. Summary of the Invention

[0004] This invention provides a method, apparatus, and device for intelligent matching between text and images, the main purpose of which is to improve the matching efficiency between text and images in medical auxiliary diagnosis.

[0005] To achieve the above objectives, the present invention provides an intelligent matching method based on text and images, comprising:

[0006] Obtain the user's search text, extract the text features of the search text, analyze the descriptive features of the search text, and fuse the descriptive features and the text features to generate target text features;

[0007] The text image to be matched is queried from a pre-built text image library, the image features of the text image are extracted, and the image features are semantically recognized to obtain the image semantic features;

[0008] The text category of the target text features is identified using the text classification network in the trained text-image matching model, and the image category of the image semantic features is identified using the image classification network in the trained text-image matching model.

[0009] Based on the text category, the text global features of the target text are detected using the text regression network in the trained text-image matching model; and based on the image category, the image global features of the image semantic features are detected using the image regression network in the trained text-image matching model.

[0010] The matching probability of the global features of the text and the global features of the image is calculated using the fully connected layer in the trained text-image matching model to obtain the matching result of the retrieved text and the text image.

[0011] Optionally, the extraction of image features from the text image can be achieved using the following formula, including:

[0012] x [l+2] =Relu(w [l+2] (Reluw [l+1] x [l] +b [l+1] )+b [l+2] +x [l] )

[0013] Where, x [l+2] Image features representing text images, w [l+2] w [l+1] b represents the slope coefficients of the (l+2)th and (l+1)th layers in the text image, respectively. [l+2] b [l+1] x represents the intercept coefficients of the (l+2)th and (l+1)th layers in the text image, respectively. [l] Let represent the image features of the original text image in layer l, and ReLU represent the activation function.

[0014] Optionally, the step of performing semantic recognition on the image features to obtain image semantic features includes:

[0015] The image features are reconstructed using the residual convolutional layer in the trained semantic segmentation model through multiple paths to obtain the reconstructed features.

[0016] Using the fusion layer in the trained semantic segmentation model, each feature in the reconstructed features is upsampled to obtain upsampled features. The upsampled features are then normalized to obtain fused features.

[0017] The activation features of the fused features are calculated using the chained residual pooling layer in the trained semantic segmentation model.

[0018] The activation features are subjected to convolution and pooling operations to obtain pooled convolution features;

[0019] The image semantic features are obtained by summing the pooled convolution features.

[0020] Optionally, the step of using a trained text-image matching model's text classification network to identify the text category of the target text features includes:

[0021] The text feature category of each text feature in the target text features is calculated using the text topic algorithm in the text classification network.

[0022] The text category frequency of the text feature category is calculated using the text statistical algorithm in the text classification network.

[0023] The text category corresponding to the target text feature is determined based on the text category frequency.

[0024] Optionally, the text topic algorithm includes:

[0025]

[0026] in, Indicates text feature categories, This represents the word vectors corresponding to text features. Let α and β represent the text topic vectors corresponding to the text features, and let α and β represent the distribution parameters of the text features. Let represent a vector consisting of α, in the form (α, α, ..., α). Let represent a vector composed of β, in the form (β,β,…β), representing the vector corresponding to the text features of the k-th topic. Let T represent the vector corresponding to the text feature of the m-th topic, T represent the number of topics corresponding to the target text feature, Q represent the number of corpus documents in the text feature, ∏ represent the multiplication symbol, and Δ represent the increment symbol.

[0027] Optionally, the step of using the image classification network in the trained text-image matching model to identify the image category based on the semantic features of the image includes:

[0028] The semantic analysis algorithm in the image classification network is used to analyze the semantic relationships among the semantic features in the image semantic features, and a syntax tree of the image semantic features is constructed based on the semantic relationships.

[0029] Based on the syntax tree, determine the image feature category of each image feature in the image semantic features, and calculate the image category frequency of the image feature category using the image statistical algorithm in the classification network;

[0030] The image category of the semantic features of the image is determined based on the image category frequency.

[0031] Optionally, the step of calculating the matching probability of the text global features and the image global features using the fully connected layer in the trained text-image matching model includes:

[0032] Using the Siamese algorithm in the fully connected layer, the comparison error function value of the text global features and the image global features is calculated;

[0033] Based on the error function value, the matching probability of text global features and image global features is calculated using the matching algorithm in the fully connected layer;

[0034] The twin algorithm includes:

[0035]

[0036] Where L(Y,X1,X2) represents the comparison error function value, and D w Equivalent to D w (X1,X2) represents the Euclidean distance between text global feature X1 and image global feature X2, Y represents the similarity between text global feature X1 and image global feature X2, m represents the preset threshold of the Euclidean distance between X1 and X2, and max represents the function for finding the maximum value.

[0037] To address the aforementioned problems, the present invention also provides an intelligent matching device based on text and images, the device comprising:

[0038] The text feature extraction module is used to acquire the user's search text, extract the text features of the search text, analyze the descriptive features of the search text, and fuse the descriptive features and the text features to generate target text features;

[0039] The image semantic recognition module is used to query the text image to be matched from the pre-built text image library, extract the image features of the text image, and perform semantic recognition on the image features to obtain the image semantic features;

[0040] The category recognition module is used to identify the text category of the target text features using the text classification network in the trained text-image matching model, and to identify the image category of the image semantic features using the image classification network in the trained text-image matching model.

[0041] The global feature detection module is used to detect the global text features of the target text based on the text category using the text regression network in the trained text-image matching model, and to detect the global image features of the image semantic features based on the image category using the image regression network in the trained text-image matching model.

[0042] The matching result calculation module is used to calculate the matching probability of the global features of the text and the global features of the image using the fully connected layer in the trained text-image matching model, so as to obtain the matching result of the retrieved text and the text image.

[0043] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0044] At least one processor; and,

[0045] A memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores a computer program that can be executed by the at least one processor to implement the above-described intelligent matching method based on text and images.

[0047] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned intelligent matching method based on text and images.

[0048] As can be seen, this embodiment of the invention first obtains the user's search text for allocation with text images. By extracting the text features of the search text, key semantic information in the search text can be obtained, reducing the computational complexity of subsequent information and thus improving the processing efficiency of subsequent information. Furthermore, analyzing the descriptive features of the search text can determine the global semantic representation corresponding to the search text, enriching the semantic information of the search text and ensuring the accuracy of subsequent text processing. The descriptive features and text features are then fused to generate target text features that contain both the basic features and the overall semantic features of the search text, providing a guarantee for the global features of the subsequently decoded text. Secondly, this embodiment of the invention, by querying the text images to be matched from a pre-built text image library, can clarify the source of the matching object of the search text, ensuring the matching premise between subsequent text and images. By extracting the image features of the text images, key semantic information in the text images can be obtained, reducing the computational complexity of subsequent information and thus improving the processing efficiency of subsequent information. Semantic recognition of the image features can obtain the semantic information of the image features without requiring... The text image is manually annotated to reduce the complexity of subsequent matching operations with the retrieved text, thereby improving the execution efficiency of subsequent matching processing. Furthermore, in this embodiment of the invention, the text classification network in the trained text-image matching model identifies the text category of the target text features to provide text data support for generating the retrieved text of the final matching object. The image classification network in the trained text-image matching model identifies the image category of the image semantic features to provide image data support for generating the retrieved image of the final matching object. Based on the text category, the text regression network in the trained text-image matching model detects the global text features of the target text features to determine the data format of the retrieved text of the final matching object. Based on the image category, the image regression network in the trained text-image matching model detects the global image features of the image semantic features to determine the data format of the text image of the final matching object. Finally, the fully connected layers in the trained text-image matching model calculate the matching probability of the global text features and the global image features to obtain the final matching result between the retrieved text and the text image. Therefore, the intelligent matching method, device, electronic device and storage medium based on text and image proposed in the embodiments of the present invention can improve the matching efficiency between text and image, especially in the field of medical auxiliary diagnosis, it can greatly improve the diagnostic efficiency and accuracy of self-diagnosis. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating an intelligent matching method between text and images provided in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of a module of an intelligent matching device based on text and images provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the internal structure of an electronic device that implements an intelligent matching method between text and images according to an embodiment of the present invention;

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] This invention provides an intelligent matching method based on text and images. The executing entity of this intelligent matching method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this invention: a server, a terminal, etc. In other words, the intelligent matching method based on text and images can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0055] Reference Figure 1 The diagram shown is a flowchart illustrating an intelligent matching method based on text and images according to an embodiment of the present invention. In this embodiment, the intelligent matching method based on text and images includes the following steps S1-S5:

[0056] S1. Obtain the user's search text, extract the text features of the search text, analyze the descriptive features of the search text, and fuse the descriptive features and the text features to generate target text features.

[0057] In this embodiment of the invention, the search text refers to the text retrieved based on keywords, topics, semantics, etc., and is generated based on different business scenarios. For example, in the scenario of medical auxiliary diagnosis, the search text can be descriptive information about disease symptoms, specifically descriptive information about symptoms such as herpes, joint swelling, etc.

[0058] Furthermore, by extracting the textual features of the searched text, this embodiment of the invention can obtain key semantic information from the searched text, reducing the computational complexity of subsequent information and thus improving the processing efficiency of subsequent information. Here, the textual features can be understood as textual feature information used to characterize the searched text.

[0059] Furthermore, in an optional embodiment of the present invention, the extraction of text features of the retrieved text can be achieved by a pre-trained language model, such as the BERT language model, the TF-IDF model, and a vocabulary model.

[0060] Furthermore, in this embodiment of the invention, by analyzing the descriptive features of the retrieved text, the global semantic representation corresponding to the retrieved text can be determined, thereby enriching the semantic information of the retrieved text and ensuring the accuracy of subsequent text processing.

[0061] Furthermore, in an optional embodiment of the present invention, the analysis of the descriptive features of the retrieved text can be achieved by inserting a [CLS] symbol before the text using the BERT model, and using the output vector corresponding to the symbol as the descriptive features of the retrieved text to characterize the semantics of the entire retrieved text.

[0062] Furthermore, in this embodiment of the invention, the feature fusion of the descriptive features and the text features generates target text features that include both the basic features of the retrieved text and its overall semantic features, thus ensuring the global features of the subsequently decoded text. Optionally, the feature fusion of the descriptive features and the text features can be implemented using a preset fusion algorithm, such as a fusion algorithm based on Bayesian decision theory or a fusion algorithm based on deep learning theory.

[0063] S2. Query the text image to be matched from the pre-built text image library, extract the image features of the text image, and perform semantic recognition on the image features to obtain the image semantic features.

[0064] This invention, through querying a pre-built text-image library to find matching text images, clarifies the source of the matching object for the retrieved text, ensuring the prerequisite for subsequent text-image matching. The pre-built text-image library refers to publicly available image databases, or custom image databases can be constructed by collecting and labeling relevant images for specific usage scenarios. The text image refers to an image used to match the text, such as a medical imaging image.

[0065] Furthermore, in an optional embodiment of the present invention, the query for the text image to be matched from the pre-built text image library can be achieved by using an SQL query statement to query the pre-built text image library.

[0066] Furthermore, by extracting image features from the text image, this embodiment of the invention can obtain key semantic information from the text image, reducing the computational complexity of subsequent information processing and thus improving the efficiency of subsequent information processing. Here, the image features can be understood as image feature information used to characterize the text image.

[0067] Furthermore, in an optional embodiment of the present invention, the extraction of image features from the text image can be achieved using the following formula:

[0068] x [l+2] =Relu(w [l+2] (Relu(w [l+1] x [l] +b [l+1] )+b [l+2] +x [l] )

[0069] Where, x [l+2] Image features representing text images, w [l+2] w [l+1] b represents the slope coefficients of the (l+2)th and (l+1)th layers in the text image, respectively. [l+2] b [l+1] x represents the intercept coefficients of the (l+2)th and (l+1)th layers in the text image, respectively. [l] Let f(x) represent the image features of the original text image in layer l, and ReLU represent the activation function, which is expressed as f(x) = max(0, w). T x+b).

[0070] Furthermore, by performing semantic recognition on the image features, the semantic information of the image features can be obtained without manually annotating the text image, thereby reducing the complexity of subsequent matching operations with the retrieved text and improving the execution efficiency of subsequent matching processing.

[0071] Further, in an optional embodiment of the present invention, the step of performing semantic recognition on the image features to obtain image semantic features includes: using a residual convolutional layer in a trained semantic segmentation model to perform multi-path reconstruction on the image features to obtain reconstructed features; using a fusion layer in a trained semantic segmentation model to upsample each feature in the reconstructed features to obtain upsampled features; performing feature normalization processing on the upsampled features to obtain fused features; using an activation algorithm in a chained residual pooling layer in a trained semantic segmentation model to calculate the activation features of the fused features; performing convolution processing and pooling operations on the activation features to obtain pooled convolutional features; and summing the pooled convolutional features to obtain the image semantic features.

[0072] The multi-path reconstruction of the image features uses residual connections to effectively fuse coarse high-level semantic features and fine-grained low-level features, thereby generating image features with higher semantic features.

[0073] S3. Use the text classification network in the trained text-image matching model to identify the text category of the target text features, and use the image classification network in the trained text-image matching model to identify the image category of the image semantic features.

[0074] In this embodiment of the invention, the text category of the target text features is identified by the text classification network in the trained text-image matching model, so as to provide text data support for the subsequent generation of the retrieval text of the final matching object.

[0075] Furthermore, in an optional embodiment of the present invention, the step of identifying the text category of the target text feature using a text classification network in a trained text-image matching model includes: calculating the text feature category of each text feature in the target text feature using a text topic algorithm in the text classification network; calculating the text category frequency of the text feature category using a text statistical algorithm in the text classification network; and determining the text category corresponding to the target text feature based on the text category frequency.

[0076] Furthermore, in an optional embodiment of the present invention, the text topic algorithm includes:

[0077]

[0078] in, Indicates text feature categories, The word vectors representing the text features. Let α and β represent the text topic vectors corresponding to the text features, and let α and β represent the distribution parameters of the text features. Let represent a vector consisting of α, in the form (α, α, ..., α). Let represent a vector consisting of β, in the form (β,β,…β). The vector representing the text features corresponding to the k-th topic. Let T represent the vector corresponding to the text feature of the m-th topic, T represent the number of topics corresponding to the target text feature, Q represent the number of corpus documents in the text feature, ∏ represent the multiplication symbol, and Δ represent the increment symbol.

[0079] Furthermore, in an optional embodiment of the present invention, the text category frequency of the text feature category is the proportion of the text feature category of each text feature in the total categories corresponding to the target text feature.

[0080] It should be noted that the image topic algorithm and the image category frequency of the image feature categories are implemented in the same way as the text topic algorithm and the text category frequency, and will not be elaborated further here.

[0081] Furthermore, in this embodiment of the invention, the image category of the image semantic features is identified by the image classification network in the trained text-image matching model, so as to provide image data support for the subsequent generation of the retrieval image of the final matching object.

[0082] Further, in an optional embodiment of the present invention, the step of identifying the image category of the image semantic features using the image classification network in the trained text-image matching model includes: analyzing the semantic relationships among the semantic features in the image semantic features using a semantic analysis algorithm in the image classification network; constructing a syntax tree of the image semantic features based on the semantic relationships; determining the image feature category of each image feature in the image semantic features based on the syntax tree; calculating the image category frequency of the image feature category using an image statistical algorithm in the classification network; and determining the image category of the image semantic features based on the image category frequency.

[0083] Furthermore, in an optional embodiment of the present invention, the image category frequency of the image feature category is the proportion of the image feature category of each image feature in the total categories corresponding to the image semantic features.

[0084] S4. Based on the text category, use the text regression network in the trained text-image matching model to detect the global text features of the target text features, and based on the image category, use the image regression network in the trained text-image matching model to detect the global image features of the image semantic features.

[0085] In this embodiment of the invention, the data format of the final matching object retrieved text is determined by using the text regression network in the trained text-image matching model to detect the global text features of the target text based on the text category.

[0086] Furthermore, in an optional embodiment of the present invention, the step of detecting the global text features of the target text features based on the text category using the text regression network in the trained text-image matching model includes: extracting text category features from the target text features based on the text category using the text regression network to obtain category features; and adding association features between the category features according to the semantic association rules of the text category to generate the global text features of the target text features.

[0087] Furthermore, in this embodiment of the invention, the data format of the final matching object text image is determined by detecting the global image features of the semantic features of the image based on the image category using the image regression network in the trained text-image matching model.

[0088] Furthermore, in an optional embodiment of the present invention, the step of detecting the global image features of the image semantic features based on the image category using the image regression network in the trained text-image matching model includes: based on the image category, using the image regression network to denoise the image semantic features to obtain denoised features; and performing region generation and edge connectivity operations on the denoised features to obtain global image features.

[0089] S5. Calculate the matching probability of the global features of the text and the global features of the image using the fully connected layer in the trained text-image matching model, so as to obtain the matching result of the retrieved text and the text image.

[0090] In this embodiment of the invention, the matching probability between the global features of the text and the global features of the image is calculated by utilizing the fully connected layer in the trained text-image matching model, thereby obtaining the final matching result between the retrieved text and the text image.

[0091] Furthermore, in an optional embodiment of the present invention, the step of calculating the matching probability of the text global features and the image global features using the fully connected layer in the trained text-image matching model includes: calculating the comparison error function value of the text global features and the image global features using the Siamese algorithm in the fully connected layer; and calculating the matching probability of the text global features and the image global features using the matching algorithm in the fully connected layer based on the error function value.

[0092] Furthermore, in an optional embodiment of the present invention, the twin algorithm includes:

[0093]

[0094] Where L(Y,X1,X2) represents the comparison error function value, and D w Equivalent to D w (X1,X2) represents the Euclidean distance (norm 2) between text global feature X1 and image global feature X2, Y represents the similarity between text global feature X1 and image global feature X2, m represents the preset threshold of the Euclidean distance between X1 and X2, and max represents the function for finding the maximum value.

[0095] Furthermore, in an optional embodiment of the present invention, the matching algorithm includes: BM algorithm, KMP algorithm, naive matching algorithm, and Sunday algorithm, etc.

[0096] Furthermore, in this embodiment of the invention, the matching result between the searched text and the text image is obtained based on the matching probability. That is, when the matching probability is greater than a preset threshold, the matching result between the searched text and the text image is considered successful, and when the matching probability is not greater than the preset threshold, the matching result between the searched text and the text image is considered unsuccessful. Optionally, the preset threshold can be set to 0.5, or it can be set according to the actual business scenario.

[0097] As can be seen, this embodiment of the invention first obtains the user's search text for allocation with text images. By extracting the text features of the search text, key semantic information in the search text can be obtained, reducing the computational complexity of subsequent information and thus improving the processing efficiency of subsequent information. Furthermore, analyzing the descriptive features of the search text can determine the global semantic representation corresponding to the search text, enriching the semantic information of the search text and ensuring the accuracy of subsequent text processing. The descriptive features and text features are then fused to generate target text features that contain both the basic features and the overall semantic features of the search text, providing a guarantee for the global features of the subsequently decoded text. Secondly, this embodiment of the invention, by querying the text images to be matched from a pre-built text image library, can clarify the source of the matching object of the search text, ensuring the matching premise between subsequent text and images. By extracting the image features of the text images, key semantic information in the text images can be obtained, reducing the computational complexity of subsequent information and thus improving the processing efficiency of subsequent information. Semantic recognition of the image features can obtain the semantic information of the image features without requiring... The text image is manually annotated to reduce the complexity of subsequent matching operations with the retrieved text, thereby improving the execution efficiency of subsequent matching processing. Furthermore, in this embodiment of the invention, the text classification network in the trained text-image matching model identifies the text category of the target text features to provide text data support for generating the retrieved text of the final matching object. The image classification network in the trained text-image matching model identifies the image category of the image semantic features to provide image data support for generating the retrieved image of the final matching object. Based on the text category, the text regression network in the trained text-image matching model detects the global text features of the target text features to determine the data format of the retrieved text of the final matching object. Based on the image category, the image regression network in the trained text-image matching model detects the global image features of the image semantic features to determine the data format of the text image of the final matching object. Finally, the fully connected layers in the trained text-image matching model calculate the matching probability of the global text features and the global image features to obtain the final matching result between the retrieved text and the text image. Therefore, the intelligent matching method, device, electronic device and storage medium based on text and image proposed in the embodiments of the present invention can improve the matching efficiency between text and image, especially in the field of medical auxiliary diagnosis, it can greatly improve the diagnostic efficiency and accuracy of self-diagnosis.

[0098] like Figure 2 The diagram shown is a functional block diagram of the intelligent matching device between text and images based on the present invention.

[0099] The intelligent matching device 100 based on text and images described in this invention can be installed in an electronic device. Depending on the functions implemented, the intelligent matching device may include a text feature extraction module 101, an image semantic recognition module 102, a category recognition module 103, a global feature detection module 104, and a matching result calculation module 105. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0100] In this embodiment, the functions of each module / unit are as follows:

[0101] The text feature extraction module 101 is used to acquire the user's search text, extract the text features of the search text, analyze the descriptive features of the search text, and fuse the descriptive features and the text features to generate target text features.

[0102] The image semantic recognition module 102 is used to query the text image to be matched from the pre-built text image library, extract the image features of the text image, and perform semantic recognition on the image features to obtain image semantic features;

[0103] The category recognition module 103 is used to identify the text category of the target text features using the text classification network in the trained text-image matching model, and to identify the image category of the image semantic features using the image classification network in the trained text-image matching model.

[0104] The global feature detection module 104 is used to detect the text global features of the target text features based on the text category using the text regression network in the trained text-image matching model, and to detect the image global features of the image semantic features based on the image category using the image regression network in the trained text-image matching model.

[0105] The matching result calculation module 105 is used to calculate the matching probability of the global features of the text and the global features of the image using the fully connected layer in the trained text-image matching model, so as to obtain the matching result of the retrieved text and the text image.

[0106] In detail, the modules in the intelligent matching device 100 based on text and images described in this embodiment of the invention employ the same methods as described above. Figure 1 The method uses the same techniques as the intelligent matching method between text and images described above, and can produce the same technical effects, so it will not be repeated here.

[0107] like Figure 3The diagram shown is a structural schematic of an electronic device 1 that implements an intelligent matching method between text and images according to the present invention.

[0108] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as an intelligent matching program based on text and images.

[0109] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing intelligent matching programs based on text and images) and calls data stored in the memory 11 to perform various functions and process data of the electronic device 1.

[0110] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for intelligent matching programs based on text and images, but also to temporarily store data that has been output or will be output.

[0111] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0112] The communication interface 13 is used for communication between the aforementioned electronic device 1 and other devices, including a network interface and an employee interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish a communication connection between the electronic device 1 and other electronic devices 1. The employee interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual employee interface.

[0113] Figure 3 Only electronic device 1 with components is shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0114] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0115] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in terms of the scope of the patent invention.

[0116] The intelligent matching program based on text and images stored in the memory 11 of the electronic device 1 is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:

[0117] Obtain the user's search text, extract the text features of the search text, analyze the descriptive features of the search text, and fuse the descriptive features and the text features to generate target text features;

[0118] The text image to be matched is queried from a pre-built text image library, the image features of the text image are extracted, and the image features are semantically recognized to obtain the image semantic features;

[0119] The text category of the target text features is identified using the text classification network in the trained text-image matching model, and the image category of the image semantic features is identified using the image classification network in the trained text-image matching model.

[0120] Based on the text category, the text global features of the target text are detected using the text regression network in the trained text-image matching model; and based on the image category, the image global features of the image semantic features are detected using the image regression network in the trained text-image matching model.

[0121] The matching probability of the global features of the text and the global features of the image is calculated using the fully connected layer in the trained text-image matching model to obtain the matching result of the retrieved text and the text image.

[0122] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0123] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0124] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device 1, can perform the following:

[0125] Obtain the user's search text, extract the text features of the search text, analyze the descriptive features of the search text, and fuse the descriptive features and the text features to generate target text features;

[0126] The text image to be matched is queried from a pre-built text image library, the image features of the text image are extracted, and the image features are semantically recognized to obtain the image semantic features;

[0127] The text category of the target text features is identified using the text classification network in the trained text-image matching model, and the image category of the image semantic features is identified using the image classification network in the trained text-image matching model.

[0128] Based on the text category, the text global features of the target text are detected using the text regression network in the trained text-image matching model; and based on the image category, the image global features of the image semantic features are detected using the image regression network in the trained text-image matching model.

[0129] The matching probability of the global features of the text and the global features of the image is calculated using the fully connected layer in the trained text-image matching model to obtain the matching result of the retrieved text and the text image.

[0130] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0131] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0133] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0134] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0135] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0136] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for intelligent matching between text and image, characterized in that, The method comprises: acquiring a search text of a user, extracting text features of the search text, analyzing description features of the search text, fusing the description features and the text features to generate target text features; querying a text image to be matched from a pre-constructed text image library, extracting image features of the text image, and performing semantic recognition on the image features to obtain image semantic features; using a text classification network in a trained text-image matching model to recognize a text category of the target text features, and using an image classification network in the trained text-image matching model to recognize an image category of the image semantic features; based on the text category, using a text regression network in the trained text-image matching model to detect text global features of the target text features, and based on the image category, using an image regression network in the trained text-image matching model to detect image global features of the image semantic features; using a full connection layer in the trained text-image matching model to calculate a matching probability of the text global features and the image global features to obtain a matching result of the search text and the text image.

2. The method for intelligent matching between text and image based on claim 1, characterized in that, The extraction of the image features of the text image comprises: extracting the image features of the text image using the following formula: x [l+2] = Relu(w [l+2] (Reluw [l+1] x [1] + b [l+1] ) + b [l+2] + x [l] ) wherein x [l+2] represents the image feature of the text image, w [l+2] , w [l+1] respectively represent the slope coefficient of the l+2th layer and the l+1th layer in the text image, b [l+2] , b [l+1] respectively represent the intercept coefficient of the l+2th layer and the l+1th layer in the text image, x [l] represents the image feature of the original text image of the lth layer, and Relu represents an activation function.

3. The method for intelligent matching between text and image based on claim 1, characterized in that, The semantic recognition of the image features to obtain image semantic features comprises: using a residual convolution layer in a trained semantic segmentation model to perform multi-path reconstruction on the image features to obtain reconstructed features; using a fusion layer in the trained semantic segmentation model to perform up-sampling on each feature in the reconstructed features to obtain up-sampled features, and performing feature normalization processing on the up-sampled features to obtain fused features; using a chained residual pooling layer in the trained semantic segmentation model to calculate activation features of the fused features; performing convolution processing and pooling operation on the activation features to obtain pooled convolution features; summing the pooled convolution features to obtain the image semantic features.

4. The method for intelligent matching between text and image based on claim 1, wherein, The recognition of the text category of the target text features using the text classification network in the trained text-image matching model comprises: using a text topic algorithm of the text classification network to calculate a text feature category of each text feature in the target text features; using a text frequency algorithm of the text classification network to calculate a text category frequency of the text feature category; determining a text category corresponding to the target text features according to the text category frequency.

5. The method of claim 4, wherein the intelligent matching between the text and the image is based on, The text topic algorithm comprises: wherein, denotes a text feature category, denotes a word vector corresponding to the text feature, denotes a text topic vector corresponding to the text feature, and α and β denote distribution parameters of the text feature, denotes a vector composed of α in the form of (α, α, … α), denotes a vector composed of β in the form of (β, β, … β), denotes a vector corresponding to the text feature of the k-th topic, denotes a vector corresponding to the text feature of the m-th topic, T denotes a number of topics corresponding to a target text feature, Q denotes a number of corpus documents in the text feature, ∏ denotes a continuous multiplication symbol, and Δ denotes an increment symbol.

6. The method for intelligent matching between text and image based on claim 1, wherein, The recognition of the image category of the image semantic features using the image classification network in the trained text-image matching model comprises: using a semantic analysis algorithm in the image classification network to analyze semantic relationships of semantic features in the image semantic features, and constructing a syntax tree of the image semantic features according to the semantic relationships; determining an image feature category of each image feature in the image semantic features according to the syntax tree, and using an image frequency algorithm in the classification network to calculate an image category frequency of the image feature category. According to the image category frequency, an image category of the image semantic feature is determined.

7. The method for intelligent matching between text and image based on claim 1, wherein, The matching probability of the text global feature and the image global feature is calculated by using a full connection layer in the trained text-image matching model, including: The contrast error function value of the text global feature and the image global feature is calculated by using a twin algorithm in the full connection layer; According to the error function value, the matching probability of the text global feature and the image global feature is calculated by using a matching algorithm in the full connection layer; The twin algorithm includes: wherein L(Y, X1, X2) represents a contrast error function value, D w is equivalent to D w (X1, X2) represents the Euclidean distance of the text global feature X1 and the image global feature X2, Y represents the similarity of the text global feature X1 and the image global feature X2, m represents a preset threshold value of the Euclidean distance between X1 and X2, and max represents a maximum value function.

8. An apparatus for intelligent matching between text and image based on, characterized by, The device includes: The text feature extraction module is configured to obtain a search text of a user, extract a text feature of the search text, analyze a description feature of the search text, perform feature fusion on the description feature and the text feature, and generate a target text feature. The image semantic recognition module is configured to query a text image to be matched from a pre-constructed text image library, extract an image feature of the text image, and perform semantic recognition on the image feature to obtain an image semantic feature. The category recognition module is configured to recognize a text category of the target text feature by using a text classification network in a trained text-image matching model, and recognize an image category of the image semantic feature by using an image classification network in the trained text-image matching model. The global feature detection module is configured to detect a text global feature of the target text feature based on the text category by using a text regression network in the trained text-image matching model, and detect an image global feature of the image semantic feature based on the image category by using an image regression network in the trained text-image matching model. The matching result calculation module is configured to calculate a matching probability of the text global feature and the image global feature by using a full connection layer in the trained text-image matching model, so as to obtain a matching result of the search text and the text image.

9. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the intelligent matching method between text and image according to any one of claims 1 to 7.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the intelligent matching method between text and image according to any one of claims 1 to 7.

Images