Image recognition, model training method and device, equipment and storage medium

Abstract

The disclosure provides an image recognition method and device, a model training method and device, equipment and a storage medium, relates to the technical field of computers, and in particular to the technical field of artificial intelligence, image processing, computer vision and deep learning. The specific implementation scheme is: determining at least one keyword text according to at least one similar image; the at least one similar image is an image with an image similarity to a target image greater than a first preset similarity; determining at least one similarity according to the target image and the at least one keyword text; the at least one similarity includes a similarity of the target image to each keyword text; determining a target text from the at least one keyword text as entity information of the target image according to the at least one similarity; the target text has a similarity to the target image greater than a second preset similarity.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical fields of artificial intelligence, image processing, computer vision and deep learning, and specifically relates to an image recognition method and device, a model training method and device, equipment and a storage medium. BACKGROUND

[0002] In a visual search scenario, after obtaining a query image, the image recognition method can be used to extract and output the text corresponding to the query image, so that the user can quickly obtain effective information related to the query image through the output text, and the time for the user to find the answer is reduced. SUMMARY

[0003] The present disclosure provides an image recognition method and device, a model training method and device, equipment and a storage medium.

[0004] According to a first aspect of the present disclosure, an image recognition method is provided, which includes: determining at least one keyword text according to at least one similar image; the at least one similar image is an image with an image similarity to a target image greater than a first preset similarity; determining at least one similarity according to the target image and the at least one keyword text; the at least one similarity includes the similarity of the target image to each keyword text; determining a target text from the at least one keyword text as entity information of the target image according to the at least one similarity; the similarity of the target text to the target image is greater than a second preset similarity.

[0005] According to a second aspect of the present disclosure, a model training method is provided, which includes: obtaining a first set of image-text pairs; each image-text pair in the first set of image-text pairs includes an image and a text associated with the image; training a first to-be-trained model according to the first set of image-text pairs, a first training target and a second training target to obtain a candidate model; the first training target is used to represent that the association degree between the image and the text in each image-text pair is greater than a first association degree; the second training target is used to represent that the association degree between the image in the first image-text pair and the text in the second image-text pair is less than a second association degree; the first image-text pair and the second image-text pair are any two image-text pairs in the first set of image-text pairs; for the image and the text in each image-text pair, obtaining at least one keyword text associated with the image from the text, and obtaining a text attribute of each keyword text; training the candidate model according to the image in each image-text pair, the at least one keyword text corresponding to each image-text pair and the text attribute of each keyword text to obtain a text understanding model.

[0006] According to a third aspect of the present disclosure, a model training method is provided, including: obtaining a second set of image-text pairs; each image-text pair in the second set of image-text pairs including an image and a text associated with the image; training a second to-be-trained model according to the second set of image-text pairs, a third training target and a fourth training target, to obtain an image-text correlation model; the third training target being used to represent that a correlation degree between the image and the text in each image-text pair is greater than a third correlation degree; the fourth training target being used to represent that a correlation degree between the image in a third image-text pair and the text in a fourth image-text pair is less than a fourth correlation degree; the third image-text pair and the fourth image-text pair being any two image-text pairs in the second set of image-text pairs.

[0007] According to a fourth aspect of the present disclosure, an image recognition apparatus is provided, including: an obtaining unit and a processing unit; the obtaining unit being configured to determine at least one keyword text according to at least one similar image; the at least one similar image being an image having a similarity to a target image greater than a first preset similarity; the processing unit being configured to determine at least one similarity according to the target image and the at least one keyword text; the at least one similarity including a similarity between the target image and each keyword text; and the processing unit being further configured to determine a target text from the at least one keyword text as entity information of the target image according to the at least one similarity; the target text having a similarity to the target image greater than a second preset similarity.

[0008] According to a fifth aspect of the present disclosure, a model training apparatus is provided, including: an obtaining unit and a processing unit; the obtaining unit being configured to obtain a first set of image-text pairs; each image-text pair in the first set of image-text pairs including an image and a text associated with the image; the processing unit being configured to train a first to-be-trained model according to the first set of image-text pairs, a first training target and a second training target, to obtain a candidate model; the first training target being used to represent that a correlation degree between the image and the text in each image-text pair is greater than a first correlation degree; the second training target being used to represent that a correlation degree between the image in a first image-text pair and the text in a second image-text pair is less than a second correlation degree; the first image-text pair and the second image-text pair being any two image-text pairs in the first set of image-text pairs; the obtaining unit being further configured to, for the image and the text in each image-text pair, obtain at least one keyword text associated with the image from the text, and obtain a text attribute of each keyword text; and the processing unit being further configured to train the candidate model according to the image in each image-text pair, the at least one keyword text corresponding to each image-text pair and the text attribute of each keyword text, to obtain a text understanding model.

[0009] According to a sixth aspect of the present disclosure, a model training apparatus is provided, comprising: an acquisition unit and a processing unit; the acquisition unit is configured to acquire a second set of image-text pairs; each image-text pair in the second set of image-text pairs comprises an image and a text associated with the image; the processing unit is configured to train a second to-be-trained model according to the second set of image-text pairs, a third training target and a fourth training target, to obtain an image-text correlation model; the third training target is configured to represent that the correlation between the image and the text in each image-text pair is greater than a third correlation degree; the fourth training target is configured to represent that the correlation between the image in a third image-text pair and the text in a fourth image-text pair is less than a fourth correlation degree; the third image-text pair and the fourth image-text pair are any two image-text pairs in the second set of image-text pairs.

[0010] According to a seventh aspect of the present disclosure, an electronic device is provided, comprising:

[0011] at least one processor; and

[0012] a memory in communication with the at least one processor; wherein

[0013] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods in the first aspect, or perform the method in the second aspect, or perform the method in the third aspect.

[0014] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, comprising:

[0015] The computer instructions are configured to enable a computer to perform any one of the methods in the first aspect, or perform the method in the second aspect, or perform the method in the third aspect.

[0016] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising:

[0017] The computer program is configured to enable a processor to perform any one of the methods in the first aspect, or perform the method in the second aspect, or perform the method in the third aspect.

[0018] The technical solution of the present disclosure solves the problem of low accuracy of image recognition results in the image recognition process.

[0019] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor are they used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0021] Figure 1 is a structural schematic diagram of an image recognition system provided by an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of an image recognition method provided by an embodiment of the present disclosure;

[0023] Figure 3 is a flowchart of a model training method provided by an embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of another model training method provided by an embodiment of the present disclosure;

[0025] Figure 5 is a flowchart of another image recognition method provided by an embodiment of the present disclosure;

[0026] Figure 6 is a flowchart of another image recognition method provided by an embodiment of the present disclosure;

[0027] Figure 7 is a structural schematic diagram of an image recognition device provided by an embodiment of the present disclosure;

[0028] Figure 8 is a structural schematic diagram of a model training device provided by an embodiment of the present disclosure;

[0029] Figure 9 is a structural schematic diagram of another model training device provided by an embodiment of the present disclosure;

[0030] Figure 10 is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help in understanding, which should be considered in their context only. Thus, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, the description below omits the description of well-known functions and structures.

[0032] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0033] In a visual search scenario, after obtaining a query image, the image recognition method can be used to extract and output the text corresponding to the query image, so that the user can quickly obtain effective information related to the query image through the output text, and reduce the time for the user to find the answer.

[0034] Common image recognition methods generally include fine-grained image classification technology and image description technology. However, the fine-grained image classification technology can generally only output the image category of the query image, and the text output by the image description technology is generally a summary description text. Therefore, neither of the two technologies can meet the user's needs.

[0035] To solve the above problems, the image recognition method provided by the embodiments of the present disclosure is applied to the scene of image recognition of an electronic device. In this method, the electronic device can determine at least one keyword text according to at least one similar image (an image with an image similarity to the target image greater than a first preset similarity), and determine at least one similarity according to the target image and the at least one keyword text. The at least one similarity includes the similarity between the target image and each keyword text. In this way, the electronic device can determine a target text with a similarity to the target image greater than a second preset similarity from the at least one keyword text as the entity information of the target image according to the at least one similarity.

[0036] As can be seen from the above, the image recognition method provided by the present disclosure can obtain the keyword text of the similar image, and determine the keyword text most similar to the target image from the keyword text of the similar image as the target text, thereby improving the accuracy of image recognition. Secondly, since the keyword text is usually a short keyword, compared with the image category and description text in the prior art, the target text determined by the electronic device can provide the user with the keyword text of the target image in fine granularity, so that the user can quickly obtain the effective information of the target image and reduce the time for the user to find the answer of the target image.

[0037] The image recognition method and model training method provided by the embodiments of the present disclosure can be applied to an image recognition system. Figure 1 A structural schematic diagram of an image recognition system provided by an embodiment of the present disclosure is shown. As shown in the figure, the image recognition system includes an electronic device 101 and a data storage device 102. Figure 1

[0038] Among them, the electronic device 101 and the data storage device 102 are in communication connection.

[0039] In actual application, the electronic device 101 can be connected to any number of data storage devices 102. For ease of understanding, Figure 1 Taking an electronic device 101 connected to one data storage device 102 as an example for description. ​

[0040] In the embodiments of the present application, the data storage device 102 is configured to provide the electronic device 101 with data (e.g., description text of at least one similar image, a set of image-text pairs, etc.) for image recognition or model training, so that the electronic device 101 performs image recognition or model training according to the data sent by the plurality of data storage devices 102.

[0041] For example, as shown in FIG. 2, when the electronic device 101 is used for image recognition, after obtaining the target image 201, the electronic device 101 can execute the image recognition method provided by the embodiments of the present disclosure, and output the target text 202 of the target image “rose flower”, so as to quickly obtain the effective information of the target image for the user and reduce the time for the user to find the answer of the target image. Figure 2

[0042] Optionally, the electronic device 101 and the data storage device 102 can be a terminal, a server, or other types of electronic devices, and the embodiments of the present application do not limit the same.

[0043] Optionally, the terminal can be a device that provides voice and / or data connectivity to users, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. The terminal can communicate with one or more core networks through a radio access network (RAN). In this way, the terminal can store the obtained image recognition result (i.e., the target text) in the server through the core network, so as to prevent data loss.

[0044] The terminal can be a mobile terminal, such as a mobile phone (or called “cellular” phone) and a computer with a mobile terminal, and can also be a portable, pocket, handheld, built-in computer, or vehicle-mounted mobile device that exchanges voice and / or data with a radio access network, such as a mobile phone, a tablet computer, a notebook computer, a netbook, a personal digital assistant (PDA).

[0045] Optionally, the server can be one server in a server cluster (composed of multiple servers), a chip in the server, or a system on chip in the server, and can also be implemented through a virtual machine (VM) deployed on a physical machine, and the embodiments of the present application do not limit the same.

[0046] ​Optionally, when the types of the entity devices of the electronic device 101 and the data storage device 102 are the same (for example, the electronic device 101 and the data storage device 102 are both servers or terminals), the electronic device 101 and the data storage device 102 can be two devices independently arranged with each other, or can be integrated in the same device.

[0047] It is easy to understand that when the electronic device 101 and the data storage device 102 are integrated in the same device, the communication mode between the electronic device 101 and the data storage device 102 is the communication between the internal modules of the device. In this case, the communication flow between the two is the same as the communication flow between the electronic device 101 and the data storage device 102 when they are independent of each other.

[0048] For ease of understanding, the present application takes the electronic device 101 and the data storage device 102 as an example for description.

[0049] The technical solutions provided by the embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0050] The image recognition method provided by the embodiments of the present disclosure includes the process of training a text understanding model by the electronic device according to a first set of image-text pairs, an image in each image-text pair in the first set of image-text pairs, at least one keyword text corresponding to each image-text pair, and the text attributes of the keyword text (referred to as "text understanding model training process"), the process of training a text understanding model by the electronic device according to a second set of image-text pairs (referred to as "image-text correlation model training process"), and the process of determining the target text of the target image by the electronic device (referred to as "image recognition process").

[0051] First, the "text understanding model training process" will be described.

[0052] As shown in Figure 3 , the "text understanding model training process" specifically includes:

[0053] S301, the electronic device obtains a first set of image-text pairs.

[0054] Among them, the first set of image-text pairs includes multiple image-text pairs, and each image-text pair includes an image and a text associated with an image.

[0055] Optionally, in combination with Figure 1 , the electronic device can obtain the first set of image-text pairs from the data storage device.

[0056] Specifically, the data storage device stores a large number of disclosed images, and these images have associated texts, such as keyword texts, abstract texts, and description texts of the images.

[0057] In the training of the text understanding model, a large amount of training data is needed to train the to-be-trained model. In this case, the electronic device can obtain a large amount of image and text of each image from the data storage device to form training data, such as a first set of image-text pairs.

[0058] S302, the electronic device trains the to-be-trained model according to the first set of image-text pairs, the first training target and the second training target to obtain a candidate model.

[0059] Optionally, the to-be-trained model can be a model implemented based on a convolutional neural network (CNN) algorithm, or a model implemented based on a recursive neural network (RNN) algorithm, or other models, which are not limited in the embodiments of the present disclosure.

[0060] Specifically, the training process of the text understanding model includes two stages. The first stage is to train the to-be-trained model to obtain a candidate model for determining the similarity between the input image and the input text. The second stage is to train the candidate model to obtain the text understanding model. Therefore, the electronic device can first train the to-be-trained model according to the first set of image-text pairs, the first training target and the second training target to obtain the candidate model.

[0061] Since the candidate model is used to determine the similarity between the input image and the input text, the electronic device can set the first training target and the second training target.

[0062] The first training target is used to indicate that the association degree between the image and the text in each image-text pair is greater than a first association degree. The second training target is used to indicate that the association degree between the image in the first image-text pair and the text in the second image-text pair is less than a second association degree. The first image-text pair and the second image-text pair are any two image-text pairs in the first set of image-text pairs.

[0063] Optionally, when the electronic device determines the association degree between the image and the text, the electronic device can first obtain the image feature of the image and the text feature of the text, and then determine the similarity between the image feature and the text feature as the association degree between the image and the text.

[0064] Optionally, the specific values of the first association degree and the second association degree can be set according to artificial experience, which are not limited in the embodiments of the present disclosure.

[0065] In actual application, the second correlation degree is usually less than the first correlation degree, that is, the correlation degree between the image and the text in the same image-text pair is as high as possible, and the correlation degree between the image and the text in different image-text pairs is as low as possible. In this way, the candidate model trained by the first training target and the second training target has higher robustness.

[0066] For example, the first image-text pair set obtained by the electronic device includes a first image-text pair and a second image-text pair. The first image-text pair includes an image 1 and a text 1 associated with the image 1. The second image-text pair includes an image 2 and a text 2 associated with the image 2.

[0067] Then, the electronic device can take the correlation degree between the image 1 and the text 1 greater than the first correlation degree and the correlation degree between the image 2 and the text 2 greater than the first correlation degree as the first training target, and take the correlation degree between the image 2 and the text 1 less than the second correlation degree and the correlation degree between the image 1 and the text 2 less than the second correlation degree as the second training target.

[0068] Then, the electronic device trains the to-be-trained model according to the above two image-text pairs, the above first training target and the above second training target, to obtain a candidate model.

[0069] S303, for the image and the text in each image-text pair, the electronic device obtains at least one keyword text associated with the image from the text, and obtains the text attribute of each keyword text.

[0070] Specifically, since the text understanding model is used to determine the candidate text associated with the input image in the input text and the text attribute of the candidate text, the electronic device needs to further obtain at least one keyword text associated with the image from the text of the image-text pair. Then, the electronic device needs to further obtain the text attribute of each keyword text.

[0071] Optionally, in combination with Figure 1 The electronic device can obtain at least one keyword text associated with the image and the text attribute of each keyword text from the data storage device, or can mark at least one keyword text associated with the image from the text by manual marking, and mark the text attribute of each keyword text by manual marking, which is not limited in the embodiments of the present disclosure.

[0072] S304, the electronic device trains the candidate model according to the image in each image-text pair, at least one keyword text corresponding to each image-text pair and the text attribute of each keyword text, to obtain a text understanding model.

[0073] For example, the image in the first image-text pair obtained by the electronic device is an image of a rose, and the text associated with the image is: a red rose is photographed in a park. Then, the electronic device can obtain the keyword texts: park, red, and rose from the text associated with the image, and the text attributes of each keyword text: the text attribute of park is address, the text attribute of red is red, and the text attribute of rose is plant. Then, the electronic device can train the candidate model according to the image of the rose obtained above, the keyword texts: park, red, and rose of the image, and the text attributes of each keyword text: address, red, and plant to obtain the text understanding model.

[0074] In this way, the text understanding model obtained by training can accurately determine at least one candidate text associated with a target image and the text attribute of each candidate text in a subsequent image recognition process.

[0075] The "image-text correlation model training process" will be described below.

[0076] As shown in Figure 4 , the "image-text correlation model training process" specifically includes:

[0077] S401, the electronic device obtains a second image-text pair set.

[0078] Each image-text pair in the second image-text pair set includes an image and a text associated with the image.

[0079] Optionally, the second image-text pair set can be the same as or different from the first image-text pair set, and the embodiments of the present disclosure do not limit this.

[0080] The specific implementation process of the electronic device obtaining the second image-text pair set can refer to the specific implementation process of the electronic device obtaining the first image-text pair set in S301, which will not be described here.

[0081] S402, the electronic device trains the second to-be-trained model according to the second image-text pair set, a third training target, and a fourth training target to obtain an image-text correlation model.

[0082] The third training target is used to represent that the association degree between the image and the text in each image-text pair is greater than a third association degree; the fourth training target is used to represent that the association degree between the image in the third image-text pair and the text in the fourth image-text pair is less than a fourth association degree; the third image-text pair and the fourth image-text pair are any two image-text pairs in the second image-text pair set.

[0083] Optionally, the third training target can be the same as or different from the first training target. The fourth training target can be the same as or different from the second training target. The model structure of the second to-be-trained model can be the same as or different from the model structure of the first to-be-trained model. The third correlation degree can be the same as or different from the first correlation degree. The fourth correlation degree can be the same as or different from the second correlation degree, which is not limited in the embodiments of the present disclosure.

[0084] The electronic device trains the second to-be-trained model according to the second set of image-text pairs, the third training target and the fourth training target, to obtain a specific implementation process of the image-text correlation model. For details, reference can be made to the specific implementation process of the candidate model obtained by training the first to-be-trained model according to the first set of image-text pairs, the first training target and the second training target in S302, which will not be described herein again.

[0085] In this way, the image-text correlation model can accurately determine the similarity between the target image and each keyword text in a subsequent image recognition process.

[0086] Next, the image recognition process will be described.

[0087] As shown in Figure 5 , the image recognition process specifically includes:

[0088] S501, the electronic device determines at least one keyword text according to at least one similar image.

[0089] The at least one similar image is an image with an image similarity to the target image greater than a first preset similarity.

[0090] Specifically, since the target image is generally an image of a query result, the target image usually has no relevant description text or keyword text. In this case, the electronic device can obtain at least one similar image with an image similarity to the target image greater than a first preset similarity, i.e., at least one similar image similar to the target image, and at least one keyword text of each image.

[0091] Optionally, the electronic device can receive a search image (query, also known as a query image) uploaded by a user terminal, and determine the search image as the target image.

[0092] Optionally, the electronic device can use an image search system to search the target image, and obtain image results (i.e., at least one similar image) identical or similar to the target image and text descriptions or abstracts (i.e., description texts of each similar image) corresponding to each search result.

[0093] It should be noted that the image search system is a general image search system, each image search system has the ability to search at least one similar image and a description text of each similar image.

[0094] Optionally, in combination with Figure 1 The electronic device can obtain at least one similar image and at least one keyword text of each image from the data storage device.

[0095] Specifically, the data storage device stores a large number of disclosed images, and these images are associated with texts such as keyword texts, abstract texts, and description texts of the images. The electronic device can obtain at least one similar image from the data storage device based on an image similarity algorithm.

[0096] Then, when the data storage device stores the keyword text of the similar image, the electronic device can directly obtain the keyword text of the similar image from the data storage device.

[0097] When the data storage device stores the similar image with a description text but does not store the keyword text, the electronic device can obtain the keyword text based on Figure 3 In the provided "text understanding model training process", the obtained text understanding model obtains at least one keyword text, and the at least one keyword text can also be obtained based on manual annotation, and the embodiments of the present disclosure are not limited in this regard.

[0098] S502, the electronic device determines at least one similarity according to the target image and the at least one keyword text.

[0099] The at least one similarity includes a similarity between the target image and each keyword text.

[0100] Specifically, after obtaining the at least one keyword text, since there may be a keyword text with a low similarity to the target image in the at least one keyword text, the electronic device can determine at least one similarity according to the target image and the at least one keyword text.

[0101] Optionally, the electronic device can determine an image feature of the target image and a text feature of each keyword text, and determine a similarity between the image feature and each text feature as a similarity between the target image and each keyword text.

[0102] The image feature and the text feature can be features in the same unit, such as vectors or matrices. In this way, the similarity between the target image and each keyword text can be accurately determined by the features in the same unit.

[0103] Optionally, the electronic device can determine the similarity between the target image and each keyword text through a cosine similarity algorithm, and can also determine the similarity between the target image and each keyword text based on Figure 4 In the provided "image-text correlation model training process", the image-text correlation model obtained through training determines the similarity between the target image and each keyword text, and the embodiments of the present disclosure do not limit this.

[0104] S503, the electronic device determines a target text from the at least one keyword text as the entity information of the target image according to the at least one similarity.

[0105] The similarity between the target text and the target image is greater than a second preset similarity.

[0106] Specifically, after obtaining the at least one similarity, since the at least one similarity includes the similarity between the target image and each keyword text, the electronic device can determine at least one keyword text that is most similar to the target image from the at least one similarity as the target text, as the entity information of the target image.

[0107] In this way, by determining at least one keyword text that is most similar to the target image from the keyword texts of the similar images as the target text, the accuracy of image recognition is improved. Secondly, since the keyword text is usually a short keyword, compared with the image category and the description text in the prior art, the target text determined by the electronic device can provide the user with a keyword text of the target image in a fine-grained manner, so that the user can quickly obtain effective information of the target image and reduce the time for the user to find the answer to the target image.

[0108] Figure 6 A flowchart of another image recognition method provided by an embodiment of the present disclosure is shown. As shown in the figure, Figure 6 The image recognition method specifically includes:

[0109] S601, the electronic device obtains at least one description text corresponding to at least one similar image.

[0110] Figure 6 In the embodiment, the at least one description text includes description text 1, description text 2,..., and description text n.

[0111] Optionally, in combination with Figure 1 A large number of disclosed images are stored in the data storage device, and each of the images has an associated description text. After determining the at least one similar image similar to the target image, the electronic device can obtain at least one description text corresponding to the at least one similar image from the data storage device.

[0112] S602, the electronic device inputs the target image and at least one descriptive text into the text understanding model to obtain at least one candidate text associated with the target image and the text attributes of each candidate text.

[0113] Figure 6 The text uses at least one candidate text and the text attributes of each candidate text as an example: candidate text 1-1 corresponding to description text 1 and its text attributes, candidate text 1-2 corresponding to description text 1 and its text attributes, ... candidate text 1-k corresponding to description text 1 and its text attributes, candidate text 2-1 corresponding to description text 2 and its text attributes, candidate text 2-2 corresponding to description text 2 and its text attributes, ... candidate text 2-k corresponding to description text 2 and its text attributes, ... candidate text n-1 corresponding to description text n and its text attributes, candidate text n-2 corresponding to description text n and its text attributes, ... candidate text nk corresponding to description text n and its text attributes.

[0114] After acquiring at least one descriptive text, since descriptive text is usually a general text with a lot of content, in order to determine more granular keyword text for the target image, the electronic device needs to determine the keyword text in each descriptive text.

[0115] Depend on Figure 3 As can be seen from the "Text Understanding Model Training Process" in the text, the text understanding model is used to extract candidate text and the attributes of candidate text from the input text. Therefore, electronic devices can input the target image and at least one descriptive text into the text understanding model to accurately obtain at least one candidate text associated with the target image and the text attributes of each candidate text.

[0116] The text attribute is used to represent the content category of the candidate text.

[0117] S603. The electronic device merges at least one candidate text according to the text attributes to obtain at least one keyword text.

[0118] Figure 6 The example given is a keyword text that includes at least one keyword text: keyword text 1 corresponding to description text 1, keyword text 2 corresponding to description text 2, ..., keyword text n corresponding to description text n.

[0119] Having obtained at least one candidate text associated with the target image and the text attributes of each candidate text, since multiple candidate texts with the same text attribute may be included, the electronic device can merge at least one candidate text according to the text attribute to obtain at least one keyword text.

[0120] For example, the three candidate texts obtained by the electronic device include: red, red, and red. The text attributes of the three candidate texts are all colors. In this case, the electronic device can merge the three candidate texts into one keyword text: red.

[0121] In S604, the electronic device inputs the target image and the at least one keyword text into the image-text correlation model to obtain at least one similarity.

[0122] Figure 6 For example, the at least one similarity includes: similarity 1 between the target image and keyword text 1, similarity 2 between the target image and keyword text 2, and similarity n between the target image and keyword text n.

[0123] After obtaining the at least one keyword text, although all the at least one keyword text are keyword texts of similar images similar to the target image, some keyword texts in the at least one keyword text may have a low similarity to the target image. In this case, the electronic device needs to screen the keyword text most similar to the target image from the at least one keyword text.

[0124] The image-text correlation model trained by the “image-text correlation training process” in the above embodiment can be used to determine the similarity between the input image and the input text. Therefore, the electronic device can input the target image and the at least one keyword text into the image-text correlation model to accurately obtain the at least one similarity. Figure 4

[0125] Optionally, in the process of determining the at least one similarity by the electronic device through the image-text correlation model, the electronic device can first determine the image features of the target image and the text features of the at least one keyword text, and then determine the similarity between the image features and the text features of the at least one keyword text as the at least one similarity.

[0126] In some embodiments, when the number of the at least one keyword text is a plurality, the number of the at least one similarity is also a plurality. The plurality of similarities further includes the similarities between different keyword texts in the plurality of keyword texts. In this case, the method of the electronic device for determining the target text from the at least one keyword text according to the at least one similarity specifically includes:

[0127] In S605, the electronic device constructs a graph structure according to the target image, the plurality of keyword texts, and the plurality of similarities.

[0128] Figure 6 ​The graph structure includes: a node corresponding to the target image, a node corresponding to the keyword text 1, a node corresponding to the keyword text 2, a node corresponding to the keyword text 3, a node corresponding to the keyword text 4, and edges between two nodes in the above five nodes are taken as examples for illustration.

[0129] Specifically, after obtaining the at least one similarity, in order to more accurately determine the target text most similar to the target image, the electronic device can construct a graph structure according to the target image, the plurality of keyword texts and the plurality of similarities.

[0130] The graph structure includes a plurality of nodes. Any two nodes are connected by an edge. The plurality of nodes includes a node for representing the target image and a node for representing the plurality of keyword texts. The weight of the edge between any two nodes is used to represent the similarity between any two nodes.

[0131] S606, the electronic device determines the ranking result of the similarity of each keyword text to the target image according to the preset algorithm and the graph structure, and determines the n keyword texts corresponding to the first n similarities in the ranking result as the target text as the entity information of the target image.

[0132] Specifically, since the graph structure not only includes edges between the node of the target image and the node of each keyword text, but also includes edges between the nodes of different keyword texts, and the weight of the edge between any two nodes is used to represent the similarity between any two nodes, the electronic device can adjust the weight of each edge in the graph structure according to the preset algorithm, so as to obtain a more accurate ranking result of the similarity of each keyword text to the target image according to the adjusted graph structure, thereby improving the accuracy of image recognition.

[0133] For example, when the weight of the edge between the node of the target image and the node of the first keyword text is 90%, but the weight of the edge between the node of the first keyword text and the node of the other keyword text is 10%, it indicates that the similarity of the first keyword text to the target image cannot reach 90%. In this case, the electronic device can adjust the weight of the edge between the node of the target image and the node of the first keyword text according to the preset algorithm.

[0134] Correspondingly, when the weight of the edge between the node of the target image and the node of the first keyword text is 10%, but the weight of the edge between the node of the first keyword text and the node of the other keyword text is 90%, it indicates that the similarity of the first keyword text to the target image may be more than 10%. In this case, the electronic device can adjust the weight of the edge between the node of the target image and the node of the first keyword text according to the preset algorithm.

[0135] Optionally, the preset algorithm can be a conditional random field algorithm, a graph neural network algorithm, etc., and the embodiments of the present disclosure do not make any limitation in this regard.

[0136] Since the weights of each edge in the adjusted graph structure represent the similarity of each keyword text to the target image and the similarity between different keyword texts, the electronic device can determine the ranking result of the similarity of each keyword text to the target image according to the size of the weight of each edge in the adjusted graph structure, and determine the n keyword texts corresponding to the top n similarities in the ranking result as the target text.

[0137] wherein n is a positive integer.

[0138] The above mainly introduces the scheme provided by the embodiments of the present disclosure from the perspective of the computer device. It can be understood that the computer device contains the hardware structure and / or software module corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that the image recognition method steps of each example described in combination with the embodiments disclosed in the present disclosure can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0139] The embodiments of the present disclosure can divide the image recognition method according to the above method examples into functional modules or functional units, for example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be realized in the form of hardware or software functional module or functional unit. Among them, the division of modules or units in the embodiments of the present disclosure is illustrative, and is only a logical function division. When actually implemented, there can be another division method.

[0140] As shown in Figure 7 Fig. 1 is a structural schematic diagram of an image recognition device provided by the embodiments of the present disclosure. The image recognition device comprises an acquisition unit 701 and a processing unit 702.

[0141] The acquisition unit 701 is configured to determine at least one keyword text according to at least one similar image; the at least one similar image is an image with an image similarity to the target image greater than a first preset similarity.

[0142] The processing unit 702 is configured to determine at least one similarity according to the target image and the at least one keyword text; the at least one similarity includes a similarity between the target image and each keyword text.

[0143] The processing unit 702 is further configured to determine a target text as entity information of the target image from the at least one keyword text according to the at least one similarity; the target text has a similarity with the target image greater than a second preset similarity.

[0144] Optionally, the obtaining unit 701 is specifically configured to:

[0145] Obtain at least one description text corresponding to the at least one similar image one by one;

[0146] Input the target image and the at least one description text into a text understanding model to obtain at least one candidate text associated with the target image and a text attribute of each candidate text;

[0147] Merge the at least one candidate text according to the text attribute to obtain the at least one keyword text.

[0148] Optionally, the processing unit 702 is specifically configured to:

[0149] Input the target image and the at least one keyword text into a picture-text correlation model to obtain the at least one similarity.

[0150] Optionally, when the number of the at least one keyword text is a plurality, the number of the at least one similarity is also a plurality; the plurality of similarities further include similarities between different keyword texts in the plurality of keyword texts; the processing unit 702 is specifically configured to:

[0151] Construct a graph structure according to the target image, the plurality of keyword texts and the plurality of similarities; the graph structure includes a plurality of nodes; any two nodes are connected through an edge; the plurality of nodes include a node for representing the target image and a node for representing the plurality of keyword texts; a weight of the edge between the any two nodes is used to represent a similarity between the any two nodes;

[0152] Determine a ranking result of the similarity between each keyword text and the target image according to a preset algorithm and the graph structure, and determine n keyword texts corresponding to a first n similarities in the ranking result as the target text; n is a positive integer.

[0153] As Figure 8The diagram shown is a structural schematic of a model training device provided in an embodiment of this disclosure. The model training device includes an acquisition unit 801 and a processing unit 802.

[0154] The acquisition unit 801 is used to acquire a first set of image-text pairs; each image-text pair in the first set of image-text pairs includes an image and a text associated with the image.

[0155] Processing unit 802 is used to train a first model to be trained based on a first set of image-text pairs, a first training objective, and a second training objective to obtain a candidate model; the first training objective is used to indicate that the correlation between the image and the text in each image-text pair is greater than a first correlation; the second training objective is used to indicate that the correlation between the image in the first image-text pair and the text in the second image-text pair is less than a second correlation; the first image-text pair and the second image-text pair are any two image-text pairs in the first set of image-text pairs.

[0156] The acquisition unit 801 is also used to acquire at least one keyword text associated with the image from the text for each image and text pair, and to acquire the text attributes of each keyword text;

[0157] The processing unit 802 is also used to train the candidate model based on the image in each image-text pair, at least one keyword text corresponding to each image-text pair, and the text attributes of each keyword text, so as to obtain a text understanding model.

[0158] like Figure 9 The diagram shown is a structural schematic of another model training device provided in an embodiment of this disclosure. The model training device includes an acquisition unit 901 and a processing unit 902.

[0159] Acquisition unit 901 is used to acquire a second set of image-text pairs; each image-text pair in the second set of image-text pairs includes an image and a text associated with the image.

[0160] Processing unit 902 is used to train the second model to be trained according to the second image-text pair set, the third training objective, and the fourth training objective to obtain the image-text relevance model; the third training objective is used to indicate that the correlation between the image and the text in each image-text pair is greater than the third correlation; the fourth training objective is used to indicate that the correlation between the image in the third image-text pair and the text in the fourth image-text pair is less than the fourth correlation; the third image-text pair and the fourth image-text pair are any two image-text pairs in the second image-text pair set.

[0161] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0162] Figure 10A schematic block diagram of an example device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0163] like Figure 10 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0164] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0165] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above, such as the image recognition method. For example, in some embodiments, the image recognition method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded onto the RAM 1103 and executed by the computing unit 1101, one or more steps of the image recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 can be configured to perform the image recognition method by any other appropriate means, such as by means of firmware.

[0166] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0167] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0168] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0170] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0172] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0173] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image recognition method, comprising: obtaining a description text corresponding to a similar image; the similar image is an image with an image similarity to a target image greater than a first preset similarity; inputting the target image and the description text into a text understanding model to obtain a candidate text associated with the target image and a text attribute of the candidate text; the text attribute is used to represent a content category of the candidate text; merging the candidate texts according to the text attributes to obtain a keyword text; determining a plurality of similarities according to the target image and the keyword text; the plurality of similarities include a similarity between the target image and the keyword text and a similarity between different keyword texts; constructing a graph structure according to the target image, the keyword text and the plurality of similarities; the graph structure includes nodes connected by edges; the nodes are used to represent the target image and the keywords; and a weight of the edge is used to represent a similarity between two nodes corresponding to the edge; adjusting the weight of each edge in the graph structure according to the weight of the edge corresponding to each node, and obtaining a ranking result of the similarity between the keyword text and the target image according to the adjusted graph structure, and determining a target text as entity information of the target image according to the ranking result; the similarity between the target text and the target image is greater than a second preset similarity.

2. The image recognition method of claim 1, wherein, determining a plurality of similarities according to the target image and the keyword text, comprising: inputting the target image and the keyword text into a graph-text correlation model to obtain the plurality of similarities.

3. The image recognition method of claim 1, further comprising: obtaining a first set of graph-text pairs; each graph-text pair in the first set of graph-text pairs includes an image and a text associated with the image; training a first to-be-trained model according to the first set of graph-text pairs, a first training target and a second training target to obtain a candidate model; the first training target is used to represent that an association degree between the image and the text in each graph-text pair is greater than a first association degree; the second training target is used to represent that an association degree between the image in the first graph-text pair and the text in the second graph-text pair is less than a second association degree; the first graph-text pair and the second graph-text pair are any two graph-text pairs in the first set of graph-text pairs; for the image and the text in each graph-text pair, obtaining at least one keyword text associated with the image from the text, and obtaining a text attribute of each keyword text; training the candidate model according to the image in each graph-text pair, the at least one keyword text corresponding to each graph-text pair and the text attribute of each keyword text to obtain the text understanding model.

4. The image recognition method of claim 2, further comprising: obtaining a second set of graph-text pairs; each graph-text pair in the second set of graph-text pairs includes an image and a text associated with the image; According to the second set of image-text pairs, the third training target and the fourth training target, the second to-be-trained model is trained to obtain the image-text correlation model. The third training target is used to represent that the correlation degree between the image and the text in each image-text pair is greater than a third correlation degree. The fourth training target is used to represent that the correlation degree between the image in the third image-text pair and the text in the fourth image-text pair is less than a fourth correlation degree; the third image-text pair and the fourth image-text pair are any two image-text pairs in the second set of image-text pairs.

5. An image recognition apparatus comprising: An acquisition unit and a processing unit; The acquisition unit is configured to acquire a description text corresponding to a similar image. The similar image is an image having an image similarity with the target image greater than a first preset similarity. The processing unit is configured to input the target image and the description text into a text understanding model to obtain a candidate text associated with the target image and a text attribute of the candidate text; the text attribute is used to represent a content category of the candidate text. The processing unit is further configured to merge the candidate texts according to the text attributes to obtain a keyword text. The processing unit is further configured to determine a plurality of similarities according to the target image and the keyword text; the plurality of similarities include a similarity between the target image and the keyword text and similarities between different keyword texts. The processing unit is further configured to construct a graph structure according to the target image, the keyword text and the plurality of similarities; the graph structure includes nodes connected by edges; the nodes are used to represent the target image and the keyword; and a weight of the edge is used to represent a similarity between two nodes corresponding to the edge. The processing unit is further configured to adjust the weights of the edges in the graph structure according to the weights of the edges corresponding to each node, obtain a ranking result of the similarities between the keyword text and the target image according to the adjusted graph structure, and determine a target text as entity information of the target image according to the ranking result; the similarity between the target text and the target image is greater than a second preset similarity. 6.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-4.

7. A non-transitory computer readable storage medium having computer instructions stored therein, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-4. 8.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-4.

Images