Method and device for matching image with text

Abstract

The invention provides a method and device for matching an image with a text, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, image processing, natural language processing, deep learning and large language models. According to one specific embodiment, the method comprises the steps that an image feature vector is extracted from an image, and the image feature vector comprises visual information and semantic information of the image; performing similarity matching on the image feature vector and a text semantic vector in a text semantic vector library to obtain a similarity matching result; and based on a similarity matching result, selecting a text matched with the image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, image processing, natural language processing, deep learning and large language model technology. Background Art

[0002] In the digital age, with the popularity of social media, when users share their life moments through photos, they often want to add text to express the mood and emotion of the photos. This demand is particularly prominent among photography enthusiasts, tourists and cultural enthusiasts. However, for most users, it is difficult to match the appropriate poem for the photo by themselves.

[0003] At present, traditional photo-to-text matching methods mainly rely on simple tag or keyword matching. For example, photos are classified to determine a set of tags that match the photos. The keywords corresponding to the set of tags are used to perform text search to obtain text that matches the photos. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, apparatus, device, storage medium, and program product for matching an image to a text.

[0005] In a first aspect, an embodiment of the present disclosure proposes a method for matching an image with text, comprising: extracting an image feature vector from an image, wherein the image feature vector contains visual information and semantic information of the image; performing similarity matching between the image feature vector and a text semantic vector in a text semantic vector library to obtain a similarity matching result; and selecting text that matches the image based on the similarity matching result.

[0006] In the second aspect, an embodiment of the present disclosure proposes a device for matching an image with text, comprising: an extraction module, configured to extract an image feature vector from an image, wherein the image feature vector contains visual information and semantic information of the image; a matching module, configured to perform similarity matching between the image feature vector and a text semantic vector in a text semantic vector library to obtain a similarity matching result; and a selection module, configured to select text that matches the image based on the similarity matching result.

[0007] In a third aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in the first aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0010] The key or important features of the embodiments of the present disclosure are not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0012] Figure 1 is a flowchart of an embodiment of a method for matching text with an image according to the present disclosure;

[0013] Figure 2 is a flowchart of another embodiment of the method for matching text with an image according to the present disclosure;

[0014] Figure 3 This is a schematic diagram of the interface for image matching text;

[0015] Figure 4 is a flow chart of a scenario for implementing the method for matching text with an image according to an embodiment of the present disclosure;

[0016] Figure 5 is a structural schematic diagram of an embodiment of an apparatus for matching text with an image according to the present disclosure;

[0017] Figure 6 The present invention is a block diagram of an electronic device for implementing the method for matching text with an image according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] Figure 1A process 100 of an embodiment of a method for matching text with an image according to the present disclosure is shown. The method for matching text with an image comprises the following steps:

[0021] Step 101: extracting an image feature vector from an image.

[0022] In this embodiment, the execution subject of the method for matching image to text may extract an image feature vector from the image.

[0023] The execution subject of the method for matching image to text is usually a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0024] A social application may be installed on the user's client. The user may publish the images taken or stored on the client on the social application. Usually, an image upload button is provided on the social application. The user clicks the image upload button and may select at least one image to upload.

[0025] After receiving the image uploaded by the client, the image feature vector can be extracted from the image. In addition, the visual and semantic aspects of the image are jointly understood to extract the image feature vector. The image feature vector contains not only the visual information of the image, but also the semantic information of the image, thereby extracting more complex semantic relationships. Among them, the visual information can include external information such as elements and styles in the image. The semantic information can include internal information such as the semantic and emotional connotations in the image.

[0026] In some embodiments, the execution subject can identify visual information from the image, and perform semantic understanding on the visual information to obtain semantic information; the semantic information can be converted into a vector to obtain an image feature vector. For example, the image is input into a target detection model to obtain the visual information of the image. The visual information is input into a natural language processing model to obtain the semantic information of the image.

[0027] Step 102: perform similarity matching between the image feature vector and the text semantic vector in the text semantic vector library to obtain a similarity matching result.

[0028] In this embodiment, the execution entity may perform similarity matching between the image feature vector and the text semantic vector in the text semantic vector library to obtain a similarity matching result.

[0029] The text can be a beautiful sentence, such as a beautiful paragraph in a book. The text semantic vector library can be a vector library that contains semantic and emotional dimensions by semantically vectorizing the text. Semantic vectorization of the text enables the system to capture the deep connotation of the text and achieve accurate matching with the image semantics. For example, the text is input into a natural language processing model (such as a pre-trained language model such as BERT (Bidirectional Encoder Representation from Transformers)) to obtain a text semantic vector. Based on a multimodal vector matching algorithm (such as the cosine similarity algorithm), the image feature vector is matched with the text semantic vector in the text semantic vector library one by one to obtain the similarity matching degree between the image and the text, so as to select the text that best fits the image.

[0030] Step 103: based on the similarity matching result, select the text that matches the image.

[0031] In this embodiment, the execution entity may select text that matches the image based on the similarity matching result.

[0032] For example, the text with the highest similarity to the image is selected as the text matching the image. Selecting text based on similarity can ensure the relevance and artistry of the selected text.

[0033] In some embodiments, the text may be poetry. Unlike other texts, poetry is a literary art that expounds the soul, and in accordance with strict rhythmic requirements, uses concise language, dense composition, abundant emotions, and rich images to highly concentrate on expressing social life and the human spiritual world.

[0034] Since the original text of the poem is highly condensed and concentrated, in order to dig out more deep connotations of the poem, the original text of the poem, the translation of the poem, the annotation of the poem, etc. can be semantically vectorized. For example, the original text of the poem, the translation of the poem, the annotation of the poem, etc. are input into the natural language processing model to obtain the semantic vector of the poem. In this way, the obtained semantic vector of the poem can not only contain the shallow description information of the poem, but also contain the deep artistic conception, emotion, atmosphere and other information of the poem. For example, for the poem "The lonely sail is far away and the blue sky is gone, only the Yangtze River is seen flowing in the sky", its shallow description information can be the description of the sailboat and the Yangtze River; its deep connotation information can be to use the sailboat and the Yangtze River as the background to render the lonely and desolate atmosphere, construct a miserable farewell picture, and then express the reluctance to leave friends.

[0035] As a result, the information of the poem used to match the image is richer, so that the poem that is highly consistent with the image can be matched.

[0036] Moreover, poetry is more literary than other texts. When poetry is matched with images, the literary flavor is stronger, making the images matched with poetry more valuable for viewing.

[0037] The disclosed embodiments provide a method for matching images with text, which can automatically match images with text and improve user experience. In addition, by cross-modally extracting image feature vectors containing visual information and semantic information to perform text semantic vector retrieval, the subject and artistic conception of the image can be automatically analyzed to match the appropriate text. When applied to social media, it can simplify the process of users sharing poetic photos on the platform. When applied to photography and tourism, it can add cultural value to photographic works and travel photos. When applied to cultural education, it can promote text learning and dissemination in education.

[0038] Continue to refer Figure 2 , which shows a process 200 of another embodiment of the method for matching text with an image according to the present disclosure. The method for matching text with an image includes the following steps:

[0039] Step 201 , based on the prompt of the prompt word, use a multimodal large language model to perform image analysis on the image to obtain visual information, and use the multimodal large language model to perform text analysis on the visual information to obtain semantic information.

[0040] In this embodiment, the executor of the method for matching image to text can perform image analysis on the image based on the prompt of the prompt word using a multimodal large language model to obtain visual information, and perform text analysis on the visual information using a multimodal large language model to obtain semantic information.

[0041] The execution subject of the method for matching image to text is usually a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0042] A social application may be installed on the user's client. The user may publish the images taken or stored on the client on the social application. Usually, an image upload button is provided on the social application. The user clicks the image upload button and may select at least one image to upload.

[0043] After receiving the image uploaded by the client, the visual and semantic aspects of the image can be jointly understood. The prompt words and the image are input into the multimodal large language model. Under the prompt of the prompt words, the multimodal large language model can perform image analysis on the image to obtain visual information, and further perform text analysis on the visual information to obtain semantic information. Among them, the prompt words can be pre-set, and the user can fine-tune the prompt words according to their own needs. The prompt words can be used to prompt the multimodal large language model to extract information such as elements, style, scene, emotion, mood, atmosphere, etc. in the image. Visual information can include external information such as elements and style in the image. Semantic information can include internal information such as semantic and emotional connotations in the image.

[0044] Step 202: Convert the semantic information into a vector to obtain an image feature vector.

[0045] In this embodiment, the execution subject can convert the semantic information into a vector to obtain an image feature vector, wherein the image feature vector contains not only the visual information of the image but also the semantic information of the image, thereby extracting a more complex semantic relationship.

[0046] Step 203, performing similarity matching between the image feature vector and the poem semantic vector in the poem semantic vector library to obtain a similarity matching result.

[0047] In this embodiment, the above-mentioned execution subject can perform similarity matching between the image feature vector and the poem semantic vector in the poem semantic vector library to obtain a similarity matching result.

[0048] The poetry semantic vector library can be a vector library that contains semantic and emotional dimensions by semantically vectorizing poetry. Semantic vectorization of poetry can enable the system to capture the deep connotation of poetry and achieve accurate matching with image semantics. For example, poetry is input into a natural language processing model (such as a pre-trained language model such as BERT) to obtain a poetry semantic vector.

[0049] Based on a multimodal vector matching algorithm (such as the cosine similarity algorithm), the image feature vector is similarly matched with the poem semantic vector in the poem semantic vector library one by one to obtain the similarity matching degree between the image and the poem, so as to select the poem that best fits the image.

[0050] Step 204, based on the similarity matching results, select the poem that matches the image.

[0051] In this embodiment, the execution entity may select a poem that matches the image based on the similarity matching result.

[0052] For example, the poem with the highest similarity to the image is selected as the poem to match the image. Selecting poems based on similarity can ensure the relevance and artistry of the selected poems.

[0053] Step 205, sending the poem matching the image to the client that uploaded the image.

[0054] In this embodiment, the execution entity may send the poem that matches the image to the client that uploaded the image, so that the user can independently select whether the poem matches the image.

[0055] Step 206, in response to receiving the poem recognition information sent by the client, the poem matching the image is added to the image and sent to the client.

[0056] In this embodiment, if the user believes that the poem matches the image, the client can send a poem approval message. After receiving the poem approval message, the execution subject can add the poem that matches the image to the image and send it to the client. In addition, the translation of the poem can also be obtained at the same time and added to the image. The interface of matching the poem with the image can be as follows: Figure 3 As shown, the top is an image 301, the middle is a poem 302, and the bottom is a translation 303.

[0057] Step 207, in response to receiving the poem negation information sent by the client, adjusting the prompt word.

[0058] In this embodiment, if the user believes that the poem and the image do not match, a poem negation message may be sent. After receiving the poem negation message, the execution subject may adjust the prompt word.

[0059] The prompt word can be adjusted randomly or specifically according to the user's choice. For example, at least one adjustment option of the prompt word is sent to the client; based on the adjustment option selected by the client, the prompt word is adjusted. The adjustment options may include but are not limited to: inappropriate style, inappropriate scene, inappropriate emotion, etc.

[0060] Step 208, based on the prompt of the adjusted prompt word, use the multimodal large language model to perform image analysis on the image to obtain new visual information, and use the multimodal large language model to perform text analysis on the new visual information to obtain new semantic information.

[0061] In this embodiment, the above-mentioned execution subject can use the multimodal large language model to perform image analysis on the image based on the prompt of the adjusted prompt word to obtain new visual information, and use the multimodal large language model to perform text analysis on the new visual information to obtain new semantic information, and return to execute step 202.

[0062] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiment, the method of image matching text in this embodiment highlights the multimodal extraction step. Therefore, the scheme described in this embodiment can automatically match images with poems and enhance the user experience. In addition, by extracting image feature vectors containing visual information and semantic information through a multimodal large language model for poetry semantic vector retrieval, the subject and artistic conception of the image can be automatically analyzed to match the appropriate poetry. At the same time, the prompt words are dynamically adjusted according to user feedback, and the image feature vectors are re-extracted for poetry semantic vector retrieval, which improves the accuracy and flexibility of the matching. Applied to social media, it can simplify the process of users sharing poetic photos on the platform. Applied to photography and tourism, it can add cultural value to photographic works and travel photos. Applied to cultural education, it can promote the learning and dissemination of ancient poetry in education.

[0063] Further references Figure 4 , which shows a process 400 of a scenario for implementing the method for matching text with an image according to an embodiment of the present disclosure, including the following steps:

[0064] Step 401: The user opens the application.

[0065] Step 402: The user selects an image through the application and uploads it to the server.

[0066] Step 403: The server performs image preprocessing.

[0067] Step 404: Based on the prompt word, call the multimodal large language model to extract image features and generate an image feature vector.

[0068] Step 405, performing similarity matching between the image feature vector and the poem semantic vector in the poem semantic vector library, and selecting the poem that matches the image.

[0069] Step 406, sending the poem matching the image to the user.

[0070] Step 407, determine whether the user is satisfied with the poem matched with the image. If satisfied, execute step 408; if not satisfied, execute step 410.

[0071] Step 408, adding the poem matching the image to the image and displaying it to the user.

[0072] Step 409, the user chooses to save or share.

[0073] Step 410, determining whether the user provides feedback. If provided, executing step 411; if not, executing step 412.

[0074] Step 411, adjusting the prompt word based on user feedback. Return to step 404.

[0075] Step 412: randomly adjust the prompt words. Return to step 404.

[0076] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for matching image with text. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0077] like Figure 5 As shown, the apparatus 500 for matching text with an image in this embodiment may include: an extraction module 501, a matching module 502, and a selection module 503. The extraction module 501 is configured to extract an image feature vector from an image, wherein the image feature vector contains visual information and semantic information of the image; the matching module 502 is configured to perform similarity matching between the image feature vector and a text semantic vector in a text semantic vector library to obtain a similarity matching result; and the selection module 503 is configured to select text matching the image based on the similarity matching result.

[0078] In the present embodiment, in the apparatus 500 for matching text with image, the specific processing of the extraction module 501, the matching module 502 and the selection module 503 and the technical effects thereof can be referred to in the respective Figure 1 The relevant descriptions of steps 101 - 103 in the corresponding embodiment are not repeated here.

[0079] In some optional implementations of this embodiment, the text is poetry.

[0080] In some optional implementations of this embodiment, the text semantic vector library is a poetry semantic vector library, and the poetry semantic vectors in the poetry semantic vector library are obtained by semantically vectorizing at least one of the original poetry, poetry translation, and poetry annotation.

[0081] In some optional implementations of this embodiment, the extraction module 501 includes: an analysis submodule, configured to identify visual information from the image, and to perform semantic understanding on the visual information to obtain semantic information; and a conversion submodule, configured to convert the semantic information into a vector to obtain an image feature vector.

[0082] In some optional implementations of this embodiment, the analysis submodule is further configured to: based on the prompt of the prompt word, use a multimodal large language model to perform image analysis on the image to obtain visual information, and use the multimodal large language model to perform text analysis on the visual information to obtain semantic information.

[0083] In some optional implementations of the present embodiment, the poem semantic vectors in the poem semantic vector library are obtained by semantically vectorizing at least one of the poem original text, poem translation, and poem annotation using a natural language processing model.

[0084] In some optional implementations of this embodiment, the device 500 for matching an image with poetry also includes: a first sending module, configured to send the poetry matching the image to the client that uploaded the image; an adding module, configured to add the poetry matching the image to the image and send it to the client in response to receiving poetry approval information sent by the client.

[0085] In some optional implementations of the present embodiment, the device 500 for matching an image with poetry further includes: an adjustment module, configured to adjust the prompt word in response to receiving poetry negation information sent by the client; an analysis module, configured to perform image analysis on the image using a multimodal large language model based on the prompt of the adjusted prompt word to obtain new visual information, and to perform text analysis on the new visual information using the multimodal large language model to obtain new semantic information.

[0086] In some optional implementations of this embodiment, the adjustment module is further configured to: send at least one adjustment option of the prompt word to the client; and adjust the prompt word based on the adjustment option selected by the client.

[0087] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0088] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0089] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0090] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0091] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the method of image matching text. For example, in some embodiments, the method of image matching text may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the method of image matching text described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method of image matching text by any other appropriate means (e.g., by means of firmware).

[0093] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0095] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0098] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0099] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0100] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for matching text with an image, comprising: Extracting an image feature vector from the image, wherein the image feature vector contains visual information and semantic information of the image; Performing similarity matching between the image feature vector and the text semantic vector in the text semantic vector library to obtain a similarity matching result; Based on the similarity matching result, text matching the image is selected.

2. The method according to claim 1, wherein: The text is poetry.

3. The method according to claim 2, wherein: The text semantic vector library is a poetry semantic vector library, and the poetry semantic vectors in the poetry semantic vector library are obtained by semantically vectorizing at least one of the original text of the poetry, the translation of the poetry, and the annotation of the poetry.

4. The method according to claim 3, wherein: The step of extracting an image feature vector from an image comprises: Identifying the visual information from the image, and performing semantic understanding on the visual information to obtain the semantic information; The semantic information is converted into a vector to obtain the image feature vector.

5. The method according to claim 4, wherein: The identifying the visual information from the image and performing semantic understanding on the visual information to obtain the semantic information includes: Based on the prompt of the prompt word, the image is analyzed by using a multimodal large language model to obtain the visual information, and the text analysis is performed on the visual information by using the multimodal large language model to obtain the semantic information.

6. The method according to claim 3, wherein: The poem semantic vectors in the poem semantic vector library are obtained by semantically vectorizing at least one of the poem original text, poem translation, and poem annotation using a natural language processing model.

7. The method according to claim 5, wherein: The method further comprises: Sending the poem matching the image to the client that uploaded the image; In response to receiving the poem recognition information sent by the client, the poem matching the image is added to the image and sent to the client.

8. The method according to claim 7, wherein: The method further comprises: In response to receiving the poem negation information sent by the client, adjusting the prompt word; Based on the prompt of the adjusted prompt word, the image is analyzed by using the multimodal large language model to obtain new visual information, and the new visual information is analyzed by using the multimodal large language model to obtain new semantic information.

9. The method according to claim 8, wherein: The step of adjusting the prompt word includes: Sending at least one adjustment option of the prompt word to the client; The prompt word is adjusted based on the adjustment option selected by the client.

10. A device for matching an image with a text, comprising: An extraction module is configured to extract an image feature vector from an image, wherein the image feature vector contains visual information and semantic information of the image; A matching module is configured to perform similarity matching between the image feature vector and the text semantic vector in the text semantic vector library to obtain a similarity matching result; The selection module is configured to select text matching the image based on the similarity matching result.

11. The device according to claim 10, wherein: The text is poetry.

12. The device according to claim 11, wherein The text semantic vector library is a poetry semantic vector library, and the poetry semantic vectors in the poetry semantic vector library are obtained by semantically vectorizing at least one of the original text of the poetry, the translation of the poetry, and the annotation of the poetry.

13. The device according to claim 12, wherein: The extraction module comprises: an analysis submodule, configured to identify the visual information from the image, and perform semantic understanding on the visual information to obtain the semantic information; The conversion submodule is configured to convert the semantic information into a vector to obtain the image feature vector.

14. The device according to claim 13, wherein: The analysis submodule is further configured to: Based on the prompt of the prompt word, the image is analyzed by using a multimodal large language model to obtain the visual information, and the text analysis is performed on the visual information by using the multimodal large language model to obtain the semantic information.

15. The device according to claim 12, wherein: The poem semantic vectors in the poem semantic vector library are obtained by semantically vectorizing at least one of the poem original text, poem translation, and poem annotation using a natural language processing model.

16. The device according to claim 14, wherein: The device also includes: A first sending module is configured to send the poem matching the image to the client that uploaded the image; The adding module is configured to add the poem matching the image to the image and send it to the client in response to receiving the poem recognition information sent by the client.

17. The device according to claim 16, wherein: The device also includes: an adjustment module, configured to adjust the prompt word in response to receiving the poem negation information sent by the client; The analysis module is configured to perform image analysis on the image based on the prompt of the adjusted prompt word using the multimodal large language model to obtain new visual information, and perform text analysis on the new visual information using the multimodal large language model to obtain new semantic information.

18. The device according to claim 17, wherein: The adjustment module is further configured to: Sending at least one adjustment option of the prompt word to the client; The prompt word is adjusted based on the adjustment option selected by the client.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images