A method for locating image regions, a method for model training and related devices

Abstract

The present application discloses a method for locating an image region, including: obtaining a set of image candidate regions in an image to be located; generating a set of regional semantic information based on the set of image candidate regions; obtaining an enhanced semantic information set corresponding to the set of regional semantic information through a graph convolutional network, wherein the graph convolutional network is used to construct the association between each regional semantic information; obtaining a set of text features corresponding to the text to be located; obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model; determining a target image candidate region from the set of image candidate regions based on the matching degree between the text feature set and each enhanced semantic information. The present application also discloses a method for model training and related devices. The present application uses a graph convolutional network to enhance the semantic representation between image candidate regions, thereby facilitating the improvement of the accuracy of image region positioning, and further improving the ability to understand images.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for locating an image region, a method for model training, and related devices. Background Art

[0002] With the continuous development of artificial intelligence, locating the area in the image corresponding to natural sentences has become an important task in machine learning. When faced with a large number of images, manually extracting the area associated with natural sentences often takes a lot of time and is prone to errors. Therefore, it is very necessary to use machines to locate image areas.

[0003] At present, the method to achieve image region positioning is to first extract multiple candidate regions in the image through object nomination, and then use a matching model to determine the matching relationship between each object nomination and natural language, and then select the local area that best matches the natural language as the target image area, thereby completing the corresponding natural sentence image positioning task.

[0004] However, although the above method can locate the image area that best matches the natural language from the image, it does not take into account the spatial relationship between the local areas and ignores the semantic information between the local areas, resulting in inaccurate positioning of the target image area, thereby reducing the ability to understand the image. Summary of the invention

[0005] The embodiments of the present application provide a method for locating an image region, a method for model training, and related devices. The use of a graph convolutional network can effectively enhance the semantic representation between image candidate regions, and takes into account the spatial relationship between each image candidate region, which is conducive to improving the accuracy of image region positioning and further improving the ability to understand the image.

[0006] In view of this, the first aspect of the present application provides a method for locating an image region, comprising:

[0007] Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1;

[0008] Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region;

[0009] Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0010] Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0011] Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0012] According to the matching degree between the text feature set and each enhanced semantic information, a target image candidate region is determined from the image candidate region set.

[0013] A second aspect of the present application provides a model training method, comprising:

[0014] Acquire a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained;

[0015] Determining a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained;

[0016] The target loss function is used to train the image region positioning network model to be trained to obtain the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

[0017] A third aspect of the present application provides an image region positioning device, comprising:

[0018] An acquisition module, used to acquire a set of image candidate regions in the image to be located, wherein the set of image candidate regions includes N image candidate regions, where N is an integer greater than or equal to 1;

[0019] A generating module, configured to generate a region semantic information set according to the image candidate region set acquired by the acquiring module, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to one image candidate region;

[0020] The acquisition module is further used to acquire, through a graph convolutional network, an enhanced semantic information set corresponding to the regional semantic information set generated by the generation module, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0021] The acquisition module is further used to acquire a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0022] The acquisition module is further used to acquire the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0023] A determination module is used to determine a target image candidate region from the image candidate region set according to a matching degree between the text feature set acquired by the acquisition module and each enhanced semantic information.

[0024] In a possible design, in a first implementation of the third aspect of the embodiments of the present application,

[0025] The generating module is specifically used to obtain the regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes regional information, and the regional information includes the position information of the image candidate region in the image to be located, and the size information of the image candidate region;

[0026] When the region semantic information corresponding to the N image candidate regions is acquired, the region semantic information set is generated according to the N region semantic information.

[0027] In a possible design, in a second implementation of the third aspect of the embodiments of the present application,

[0028] The acquisition module is specifically used to acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any one regional semantic information in the regional semantic information set, and the second regional semantic information is any one regional semantic information in the regional semantic information set;

[0029] Acquire the connection edge strength between the first region semantic information and the second region semantic information;

[0030] Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

[0031] Determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0032] The enhanced semantic information set corresponding to the target connection matrix is ​​determined through the graph convolutional network.

[0033] In a possible design, in a third implementation of the third aspect of the embodiments of the present application,

[0034] The acquisition module is specifically used to generate a connection matrix according to the normalized strength between each piece of regional semantic information in the regional semantic information set;

[0035] The target connection matrix is ​​generated according to the connection matrix and the unit matrix.

[0036] In a possible design, in a fourth implementation of the third aspect of the embodiments of the present application,

[0037] The acquisition module is specifically used to calculate the enhanced semantic information set in the following manner:

[0038]

[0039] Among them, the represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the jth enhanced semantic information corresponding to the (k-1)th layer of the graph convolutional network, represents the first network parameter of the graph convolutional network of the kth layer, represents the second network parameter of the graph convolutional network of the kth layer, the j∈neighboring(i) represents that the jth node belongs to the neighboring node of the i-th node, the E ij Represents an element in the target connection matrix.

[0040] In a possible design, in a fifth implementation of the third aspect of the embodiments of the present application,

[0041] The acquisition module is specifically used to acquire the text to be located;

[0042] Acquire a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word;

[0043] Encoding each word vector in the text vector sequence to obtain text features;

[0044] When the text features corresponding to the T word vectors are obtained, the text feature set is generated according to the T text features.

[0045] In a possible design, in a sixth implementation of the third aspect of the embodiments of the present application,

[0046] The acquisition module is specifically used to acquire the text features in the following manner:

[0047] h t =LSTM(w t ,h t-1 );

[0048] Among them, the h t represents the tth text feature in the text feature set, the LSTM() represents the encoding using the long short-term memory LSTM network, the w t represents the tth word vector in the text vector sequence, the h t-1 represents the (t-1)th text feature in the text feature set.

[0049] A fourth aspect of the present application provides a model training device, comprising:

[0050] an acquisition module, used for acquiring a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained;

[0051] A determination module, configured to determine a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained acquired by the acquisition module;

[0052] A training module is used to train the image region positioning network model to be trained using the target loss function determined by the determination module to obtain an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned based on a text feature set and enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

[0053] In a possible design, in a first implementation of the fourth aspect of the embodiments of the present application,

[0054] The determination module is specifically used to determine the target loss function in the following manner:

[0055]

[0056] Wherein, L represents the target loss function, represents the first candidate region of the image to be trained, represents the first text to be trained, the represents the second candidate region of the image to be trained, represents the second text to be trained, d() represents the data pair to be trained, max() represents the maximum value, λ1 represents the first parameter control weight, λ2 represents the second parameter control weight, u1 represents the first preset threshold, and u2 represents the second preset threshold.

[0057] A fifth aspect of the present application provides a terminal device, including: a memory, a transceiver, a processor, and a bus system;

[0058] Wherein, the memory is used to store programs;

[0059] The processor is used to execute the program in the memory, comprising the following steps:

[0060] Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1;

[0061] Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region;

[0062] Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0063] Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0064] Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0065] Determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information;

[0066] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0067] A sixth aspect of the present application provides a server, including: a memory, a transceiver, a processor, and a bus system;

[0068] Wherein, the memory is used to store programs;

[0069] The processor is used to execute the program in the memory, comprising the following steps:

[0070] Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1;

[0071] Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region;

[0072] Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0073] Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0074] Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0075] Determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information;

[0076] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0077] A seventh aspect of the present application provides a server, including: a memory, a transceiver, a processor, and a bus system;

[0078] Wherein, the memory is used to store programs;

[0079] The processor is used to execute the program in the memory, comprising the following steps:

[0080] Acquire a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained;

[0081] Determining a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained;

[0082] The target loss function is used to train the image region positioning network model to be trained to obtain the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned;

[0083] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0084] An eighth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the methods described in the above aspects.

[0085] A ninth aspect of the present application provides a method for locating an image region, comprising:

[0086] receiving an image positioning instruction;

[0087] In response to the image positioning instruction, obtaining an image candidate region set in the image to be positioned according to the image positioning instruction, wherein the image candidate region set includes N image candidate regions, where N is an integer greater than or equal to 1;

[0088] Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region;

[0089] Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0090] Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0091] Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0092] Determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information;

[0093] An image generation instruction is sent to the client, so that the client displays the target image candidate area according to the image generation instruction.

[0094] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0095] In an embodiment of the present application, a method for image region positioning is provided, firstly obtaining an image candidate region set in the image to be positioned, wherein the image candidate region set includes N image candidate regions, then generating a region semantic information set based on the image candidate region set, each region semantic information corresponds to an image candidate region, and then obtaining an enhanced semantic information set corresponding to the region semantic information set through a graph convolutional network, each enhanced semantic information corresponds to a region semantic information, and the graph convolutional network is used to construct the association relationship between each region semantic information, and in addition, obtaining a text feature set corresponding to the text to be positioned, and then obtaining the matching degree of the text feature set and each enhanced semantic information through the image region positioning network model, and finally determining the target image candidate region from the image candidate region set based on the matching degree of the text feature set and each enhanced semantic information. In the above manner, the use of a graph convolutional network can effectively enhance the semantic representation between image candidate regions, and takes into account the spatial relationship between each image candidate region, which is conducive to improving the accuracy of image region positioning, and then improving the ability to understand the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A schematic diagram of the architecture of an image region positioning system in an embodiment of the present application;

[0097] Figure 2 A schematic diagram of an overall framework for positioning an image region in an embodiment of the present application;

[0098] Figure 3 A schematic diagram of an embodiment of a method for locating an image region in an embodiment of the present application;

[0099] Figure 4 This is a schematic diagram of an embodiment of the method for model training in the embodiments of the present application;

[0100] Figure 5 This is a schematic diagram of an embodiment of an image area positioning device in an embodiment of the present application;

[0101] Figure 6 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application;

[0102] Figure 7 This is a schematic diagram of the structure of a terminal device in an embodiment of the present application;

[0103] Figure 8 This is a structural diagram of a server in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The embodiments of the present application provide a method for locating an image region, a method for model training, and related devices. The use of a graph convolutional network can effectively enhance the semantic representation between image candidate regions, and takes into account the spatial relationship between each image candidate region, which is conducive to improving the accuracy of image region positioning and further improving the ability to understand the image.

[0105] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0106] It should be understood that the image region positioning method provided in the present application can be applied to fields such as image processing and pattern recognition, and the target of interest can be located from the image, and then the specific category of the target can be determined, and the bounding box of the target can be given. The method of image region positioning has a wide range of applications in fields such as face recognition, medical imaging, intelligent video monitoring, robot navigation, content-based image retrieval, image-based rendering technology, image editing and augmented reality. For example, in a content-based image retrieval scenario, it is assumed that there is an image A, and multiple candidate regions are extracted from image A. It is assumed that the user enters a sentence as "a boy hold an apple", so the sentence is matched with each candidate region, and the target candidate region is selected from multiple candidate regions according to the matching results. The present application is mainly a method of using a graph convolutional network to complete the image positioning of natural sentences. A natural sentence can be a word, a phrase or a sentence, and the target candidate region of the image corresponding to the natural sentence is found, and the target candidate region can be defined as a rectangular frame.

[0107] In practical applications, the positioning of image areas can be divided into three levels. The first level is the image level, that is, determining whether there are relevant target objects in the image. In image classification or image annotation technology, for example, for the word "apple", the apple object can be circled from the image.

[0108] The second layer is the region level, which is to determine whether a certain area in the image contains a certain type of target. In image target category detection, for example, for the sentence "a boy hold an apple", an area can be selected from the image, which includes a boy and an apple.

[0109] The third layer is the region level, which determines which type of target object each pixel in the image belongs to. Pixel-level segmentation is also divided into category-level target segmentation and semantic segmentation. The main difference between category-level target segmentation and semantic segmentation is that semantic segmentation requires all targets in the image, including the background, to be segmented and their categories to be determined, while target segmentation only requires segmenting and classifying the target of interest.

[0110] For ease of understanding, this application proposes a method for locating an image region. The method is applied to Figure 1 The image area positioning system shown is shown in Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image region positioning system in the embodiment of the present application. As shown in the figure, the image region positioning method provided by the present application can be used for a server or a client. If applied to a server, after the server determines the positioning result, the positioning result can be sent to the client, and the corresponding target image candidate area can be displayed by the client. If applied to a client, after the client determines the positioning result, the corresponding target image candidate area can be directly displayed. Specifically, for an image, an image detection method is first used to obtain multiple image candidate areas (i.e., local areas of the image). For multiple image candidate areas, a graph is constructed through their spatial relationship. Then, for the image candidate areas, a convolutional neural network can be used to extract corresponding semantic features. Based on the obtained semantic features and the constructed graph, a graph convolutional network (GCN) is used to further learn the representation of the image candidate areas. Based on the image candidate area representation obtained by GCN, the semantic relevance of these image candidate areas to a given natural sentence is measured by semantic matching, and then the most relevant image candidate area is determined as the final result of natural sentence image positioning, that is, the target image candidate area is obtained.

[0111] It should be noted that the client is deployed on a terminal device, wherein the terminal device includes but is not limited to tablet computers, laptop computers, PDAs, mobile phones, voice interaction devices and personal computers (PCs), which are not limited here. Among them, the voice interaction device includes but is not limited to smart speakers and smart home appliances.

[0112] The image area positioning method proposed in this application can provide a natural sentence image positioning service. The service can be deployed on the server side or on the terminal device side. It is understandable that the application of the image area positioning method on the server side can achieve a deeper understanding of the image, and then perform more detailed fine-grained annotation of the image, thereby helping users to search and match quickly and accurately, and can also be applied to personalized recommendations of graphic information. The image area positioning method can also be deployed on a terminal device, such as a mobile phone or a robot, etc., and the corresponding image signal is obtained through the robot's camera, and the user interacts with the corresponding robot through natural language. For example, the user obtains the corresponding natural language text through voice or keyboard input, and then uses the image area positioning network model to locate the local area of ​​the image for the corresponding natural language text. In this way, the terminal device can better interact with the user.

[0113] For easier understanding, see Figure 2 , Figure 2 This is a schematic diagram of the overall framework of image region positioning in the embodiment of the present application. As shown in the figure, for a natural image, the corresponding image candidate region is obtained by the object proposal method. After extracting the corresponding image candidate region, we use the corresponding convolutional neural network to extract the semantic representation of the corresponding image candidate region and represent each candidate region as a feature vector. Then we get the semantic representation of the corresponding candidate region {I1,I2,I3.I4...,I n}, where n represents the total number of image candidate regions extracted from the image. GCN is then used to enhance the semantic representation of the extracted image candidate regions to obtain the enhanced semantic representation of the image candidate regions {V1, V2, V3. V4..., V n In the process of constructing a graph convolutional network, it is necessary to consider the semantic similarity between the candidate regions of the image, thereby constructing the corresponding graph and defining the corresponding connection edge information. Through the connection edge information, the semantic representation of the corresponding candidate region of the image is enhanced.

[0114] For the input natural sentences (such as "the person on the left wearing white clothes playing baseball"), we use a recurrent neural network to encode the natural sentences and obtain their corresponding semantic representations. For the semantic representations of natural sentences and the corresponding enhanced semantic representations of image candidate regions, a matching learning method is used to learn the semantic relationship between the natural sentences and the corresponding image candidate regions. Finally, the semantic similarity between the two is used to select the most semantically relevant image candidate regions as the target image candidate regions.

[0115] In combination with the above introduction, the method for locating the image area in this application will be introduced below. Figure 3 In one embodiment of the present application, a method for locating an image region includes:

[0116] 101. Obtain an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, where N is an integer greater than or equal to 1;

[0117] In this embodiment, the image area positioning device first obtains the image to be positioned, wherein the image to be positioned can be an image stored in the server background, can be an image uploaded by the client, or can be a local image of the client. It can be understood that the image area positioning device can be deployed on a server or on a terminal device, which is not limited here.

[0118] After the image region positioning device obtains the image to be positioned, an image candidate region set in the image to be positioned can be extracted through the object nomination method, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than 1 or equal to 1. When N is equal to 1, it means that there is only one image candidate region in the image to be positioned, and the image candidate region can be directly used as the target image candidate region.

[0119] The object nomination method can extract the image candidate region in the image. Specifically, the image candidate region is to find out the possible location of the target in the image in advance, and to ensure that a high recall rate (Intersection-over-Union, IoU) can be maintained when selecting fewer windows by using information such as texture, edge and color in the image. Among them, the object nomination method includes but is not limited to region-based convolutional neural networks (R-CNN), fast region-based convolutional neural networks (Fast R-CNN) and faster region-based convolutional neural networks (Faster R-CNN), etc., which are not limited here.

[0120] 102. Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to one image candidate region;

[0121] In this embodiment, after acquiring the image candidate region set, the image region positioning device generates a corresponding image candidate region semantic representation through a neural network, that is, obtains a region semantic information set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region.

[0122] It should be noted that the neural network can specifically be a convolutional neural network (CNN). In practical applications, it can also be other types of neural networks. This is only an illustration and should not be construed as a limitation on the present application.

[0123] 103. Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0124] In this embodiment, the image region positioning device obtains the enhanced semantic information set corresponding to the regional semantic information set through GCN, that is, the semantic representation of the image candidate region can be enhanced by using GCN. Among them, the enhanced semantic information set includes N enhanced semantic information, that is, each image candidate region corresponds to a regional semantic information, and each image candidate region corresponds to a regional semantic enhanced semantic information. GCN can construct an association relationship between nodes, and in this application, the association relationship between each regional semantic information can be constructed.

[0125] GCN is a convolutional network model. For GCN, its goal is to learn a mapping of signals or features on the graph G = (V, E). The graph construction process is performed after obtaining the image candidate regions. It is constructed based on the spatial information between the image candidate regions. It can use the information contained in the data and the relationship between the data to increase the semantic expression of the image candidate regions, thereby obtaining enhanced semantic information.

[0126] 104. Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0127] In this embodiment, the image area positioning device obtains the text to be positioned. It can be understood that step 104 can be executed before step 101, can be executed after step 103, or can be executed simultaneously with step 101. The execution order of step 104 is not limited here. The text to be positioned can be a text input by the user, or a voice input by the user, and a text obtained after the voice is recognized. The text to be positioned is expressed in the form of words, phrases, sentences or paragraphs, and the text to be positioned can be Chinese, English, Japanese, French, German or Russian, etc., which is not limited here.

[0128] After obtaining the text to be located, feature extraction and encoding processing are performed on each word in the text to be located, and finally a text feature set is obtained. For example, the text to be located "boy hold an apple" includes four words, namely "boy", "hold", "an" and "apple". After extracting the features of these four words, encoding processing is performed to obtain a text feature set. Among them, the text to be located includes T words, and the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1.

[0129] It can be understood that the text to be located "boy hold an apple" is a concrete expression, and therefore, an image candidate region containing both "boy" and "apple" can be obtained in the image to be located.

[0130] 105. Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0131] In this embodiment, the image region positioning device can input each enhanced semantic information and text feature set into the image region positioning network model, and the image region positioning network model outputs the corresponding matching degree. The image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned, that is, the higher the matching degree, the stronger the matching relationship.

[0132] It is understandable that the matching degree can be expressed as a matching score or a matching identifier, and can also be expressed as other types of matching relationships.

[0133] 106. Determine a target image candidate region from the image candidate region set according to the matching degree between the text feature set and each enhanced semantic information.

[0134] In this embodiment, the image region positioning device can select the image candidate region with the highest matching degree from the image candidate region set as the target image candidate region according to the matching degree between the text feature set and each enhanced semantic information. For ease of introduction, please refer to Table 1, which is a schematic diagram of the matching degree between the text feature set and the enhanced semantic information.

[0135] Table 1

[0136]

[0137]

[0138] It can be seen from Table 1 above that for “text feature set + enhanced semantic information D”, the corresponding matching degree is the largest, so the image region positioning device uses the image candidate region D as the target image candidate region.

[0139] In an embodiment of the present application, a method for image region positioning is provided, firstly obtaining an image candidate region set in the image to be positioned, wherein the image candidate region set includes N image candidate regions, then generating a region semantic information set based on the image candidate region set, each region semantic information corresponds to an image candidate region, and then obtaining an enhanced semantic information set corresponding to the region semantic information set through a graph convolutional network, each enhanced semantic information corresponds to a region semantic information, and the graph convolutional network is used to construct the association relationship between each region semantic information, in addition, obtaining a text feature set corresponding to the text to be positioned, then obtaining the matching degree of the text feature set and each enhanced semantic information through the image region positioning network model, and finally determining the target image candidate region from the image candidate region set based on the matching degree of the text feature set and each enhanced semantic information. In the above manner, the use of a graph convolutional network can effectively enhance the semantic representation between image candidate regions, and takes into account the spatial relationship between each image candidate region, which is conducive to improving the accuracy of image region positioning, and thus improving the ability to understand the image.

[0140] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in a first optional embodiment of the method for locating an image region provided in the embodiments of the present application, generating a region semantic information set according to an image candidate region set may include:

[0141] Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes regional information, and the regional information includes position information of the image candidate region in the image to be located, and size information of the image candidate region;

[0142] When the region semantic information corresponding to the N image candidate regions is obtained, a region semantic information set is generated according to the N region semantic information.

[0143] In this embodiment, after acquiring the image candidate region set, the image region positioning device can generate region semantic information of each image candidate region through CNN, wherein the region semantic information is the semantic representation of the image candidate region. Specifically, assuming that the image candidate region set is defined as {b1, b2, b3. b4..., b n}, where each image candidate region includes region information b i ={x1,y1,w1.h1}, image candidate region b i represents an image candidate region in the image candidate region set, x1 and y1 represent the position information of the image candidate region in the image to be located, specifically, x1 represents the horizontal coordinate position information of the vertex of the image candidate region in the image to be located, y1 represents the vertical coordinate position information of the vertex of the image candidate region in the image to be located, w1 and h1 represent the size information of the image candidate region, and the size information refers to the proportional size of the image candidate region relative to the image to be located, w1 represents the width information of the image candidate region in the image to be located, and h1 represents the height information of the image candidate region in the image to be located.

[0144] The image candidate region b i Input to CNN, and we get the following representation:

[0145] I i =CNN(b i );

[0146] Thus, the corresponding regional semantic information I is obtained i Based on the above method, we get the image candidate region set

[0147] {b1,b2,b3,b4,...,b n} corresponds to the regional semantic information set {I1,I2,I3,I4,...,I n}, n is an integer greater than or equal to 1 and less than or equal to N.

[0148] For ease of understanding, CNN usually includes the following layers:

[0149] Convolutional layer: Each convolutional layer in a convolutional neural network consists of several convolutional units, and the parameters of each convolutional unit are optimized through the back-propagation algorithm. The purpose of the convolution operation is to extract different features of the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features.

[0150] Rectified Linear Units layer (ReLU layer), the activation function of this layer of neurons uses ReLU (Rectified Linear Units).

[0151] The pooling layer usually obtains large-dimensional features after the convolution layer. The features are cut into several regions, and the maximum or average value is taken to obtain new features with smaller dimensions.

[0152] The fully connected layer combines all local features into global features to calculate the final score of each category.

[0153] Secondly, in an embodiment of the present application, a method for generating a set of regional semantic information is provided. First, the regional semantic information corresponding to the image candidate area is obtained through a convolutional neural network. The image candidate area includes regional information, and the regional information includes the position information of the image candidate area in the image to be located, and the size information of the image candidate area. When the regional semantic information corresponding to N image candidate areas is obtained, a set of regional semantic information is generated according to the N regional semantic information. Through the above method, CNN can be used to extract the regional semantic information of each image candidate area. CNN is a feedforward neural network whose artificial neurons can respond to surrounding units within a part of the coverage range, so that it has excellent performance in large image processing, thereby improving the accuracy of information extraction.

[0154] Optionally, in the above Figure 3 On the basis of the corresponding embodiment, in a second optional embodiment of the method for locating an image region provided in the embodiment of the present application, obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network may include:

[0155] Acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any regional semantic information in the regional semantic information set, and the second regional semantic information is any regional semantic information in the regional semantic information set;

[0156] Obtaining the connection edge strength between the first region semantic information and the second region semantic information;

[0157] Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

[0158] Determine the target connection matrix according to the normalized strength between the semantic information of each region in the regional semantic information set;

[0159] The enhanced semantic information set corresponding to the target connection matrix is ​​determined through a graph convolutional network.

[0160] In this embodiment, the image region positioning device uses a GCN to enhance the semantic representation of the image candidate region. First, we need to build a graph in which each node corresponds to the regional semantic information of the image candidate region. There are corresponding connecting edges between each node, and the strength of the connection information comes from a deep network for prediction:

[0161]

[0162] in, represents the semantic information of the first region, represents the semantic information of the second region, f edge () represents a deep network, which can be implemented by multi-layer perceptron, vector inner product or cosine similarity. ij represents the strength of the connection edge between the semantic information of the first region and the semantic information of the second region. ij Normalization is performed to obtain normalized strength. Then, the target connection matrix is ​​determined based on the normalized strength between the semantic information of each region in the regional semantic information set, and finally GCN is used to generate the enhanced semantic information set corresponding to the target connection matrix.

[0163] A graph is a data format that can be used to represent social networks, communication networks, and protein molecular networks. The nodes in the graph represent individuals in the network, and the edges represent the connection relationship between individuals. Many machine learning tasks require graph structured data, so the emergence of GCN provides a new idea for solving these problems. Convolution slices can be constructed in three steps: Step 1, select a node sequence of a fixed length from the graph; Step 2, for each node in the sequence, collect a neighborhood set of a fixed size; Step 3, normalize the subgraph consisting of the current node and its corresponding neighborhood as the input of the convolution structure. After all the convolution slices are constructed through the above three steps, the convolution structure is used to operate on each slice separately.

[0164] Secondly, in an embodiment of the present application, a method for obtaining an enhanced semantic information set through GCN is provided. First, the first region semantic information and the second region semantic information are obtained from the region semantic information set, and then the connection edge strength between the first region semantic information and the second region semantic information is obtained. Next, the connection edge strength between the first region semantic information and the second region semantic information is normalized to obtain the normalized strength, and then the target connection matrix is ​​determined according to the normalized strength between the regional semantic information in the regional semantic information set. Finally, the enhanced semantic information set corresponding to the target connection matrix is ​​determined through a graph convolutional network. In the above manner, GCN is used to construct the semantic relationship between candidate regions of the image, thereby fully considering the spatial information and semantic relationship, thereby improving the image-based positioning performance.

[0165] Optionally, in the above Figure 3 On the basis of the corresponding second embodiment, in a third optional embodiment of the method for locating an image region provided by the embodiment of the present application, determining the target connection matrix according to the normalized strength between the respective regional semantic information in the regional semantic information set may include:

[0166] Generate a connection matrix according to the normalized strength between the regional semantic information in the regional semantic information set;

[0167] Generate a target connection matrix based on the connection matrix and the unit matrix.

[0168] In this embodiment, the image region positioning device may first normalize the strength of the connection edge between the first region semantic information and the second region semantic information to obtain the normalized strength. Based on the above embodiment, the strength of the connection edge between the first region semantic information and the second region semantic information is specifically expressed as:

[0169]

[0170] in, represents the semantic information of the first region, represents the semantic information of the second region, b1, b2, w1 and w2 are all GCN model parameters, e ij Represents the strength of the connection edge between the semantic information of the first region and the semantic information of the second region.

[0171] The corresponding side information can be further normalized to obtain the normalized strength:

[0172]

[0173] in, Represents the normalized strength between the first region semantic information and the second region semantic information. By traversing each region semantic information, we can obtain the connection information of the corresponding region semantic information, and then construct a complete connection matrix:

[0174]

[0175] In order to further enhance its own information, the corresponding complete connection matrix is ​​further added with a unit matrix to obtain the target connection matrix:

[0176]

[0177] Again, in an embodiment of the present application, a method for determining a target connection matrix is ​​provided based on the normalized strength between each regional semantic information in a regional semantic information set, that is, firstly, a connection matrix is ​​generated based on the normalized strength between each regional semantic information in a regional semantic information set, and then a target connection matrix is ​​generated based on the connection matrix and a unit matrix. Through the above method, a normalized processing method is adopted, so that the absolute value of the numerical value of the physical system can be converted into a relative value relationship, thereby simplifying the calculation and reducing the value. In addition, in order to further enhance its own information, the corresponding connection matrix is ​​further added with a unit matrix to form a target connection matrix.

[0178] Optionally, in the above Figure 3 On the basis of the corresponding second or third embodiment, in a fourth optional embodiment of the method for image region positioning provided by the embodiment of the present application, determining the enhanced semantic information set corresponding to the target connection matrix through a graph convolutional network may include:

[0179] The enhanced semantic information set is calculated as follows:

[0180]

[0181] in, represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the j-th enhanced semantic information corresponding to the (k-1)-th layer of the graph convolutional network, represents the first network parameter of the k-th layer graph convolutional network, represents the second network parameter of the k-th layer graph convolutional network, j∈neighboring(i) means that the j-th node belongs to the neighboring node of the i-th node, E ij Represents the elements in the target connection matrix.

[0182] In this embodiment, the image region positioning device uses GCN to enhance the semantic representation of the image candidate region based on the target connection matrix:

[0183]

[0184] in, represents the first network parameter of the k-th layer GCN, The second network parameter of the GCN representing the kth layer. It can be understood that the network parameters of the GCN are not shared between graph convolutional layers, but between a fixed convolutional layer, its network parameters can be shared or not. The J node selects the neighbor node corresponding to the node i. Because the similarity between nodes can be measured by the semantic similarity of the nodes. Therefore, a fully connected graph structure is constructed, corresponding to each node being connected to other nodes. Finally, based on the constructed target connection matrix, the semantic representation of each corresponding node is updated.

[0185] In GCN, multi-layer graph convolution processing can be performed, and the above formula can be executed multiple times. The parameters in the formula can share the same set of network parameters or not.

[0186] Furthermore, in an embodiment of the present application, a specific method for determining the enhanced semantic information set corresponding to the target connection matrix through GCN is provided. Through the above method, a specific calculation method is provided for GCN-based calculation, thereby improving the feasibility and operability of the solution.

[0187] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in a fifth optional embodiment of the method for locating an image region provided in the embodiments of the present application, obtaining a text feature set corresponding to the text to be located may include:

[0188] Get the text to be located;

[0189] Obtain a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponding to a word;

[0190] Encode each word vector in the text vector sequence to obtain text features;

[0191] When the text features corresponding to the T word vectors are obtained, a text feature set is generated based on the T text features.

[0192] In this embodiment, the image area positioning device first obtains the text to be positioned. The text to be positioned can be a text input by the user, or a voice input by the user, which is converted into text, or a text extracted by the background. After obtaining the text to be positioned, each word in the text to be positioned is extracted, and then a word vector is constructed for each word. Assuming that the text to be positioned includes T words, T word vectors can be obtained, and these T word vectors constitute a text vector sequence. The image area positioning device uses a long short-term memory network (Long Short-Term Memory, LSTM) structure to encode the text vector sequence. Specifically, the LSTM structure is used to encode each word vector, thereby obtaining T text features, thereby generating a text feature set.

[0193] In natural language processing tasks, we first need to consider how words are represented in computers. There are usually two representation methods, namely discrete representation (one-hot representation) and distributed representation (distributionrepresentation). One-hot representation represents each word as a long vector. The dimension of this vector is the size of the vocabulary. Only one dimension in the vector has a value of 1, and the other dimensions are 0. This dimension represents the current word. The word vector dimension in this application can be 300 dimensions. Word embedding refers to the conversion of words into a distributed representation, also known as word vectors. There are many ways to generate word vectors. These methods are based on the idea that the meaning of any word can be represented by its surrounding words. The methods for generating word vectors can be divided into: statistical-based methods and language model-based methods.

[0194] Secondly, in an embodiment of the present application, a method for obtaining a text feature set is provided, that is, first obtaining the text to be located, and then obtaining a text vector sequence based on the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word, and then each word vector in the text vector sequence is encoded to obtain text features, and when the text features corresponding to the T word vectors are obtained, a text feature set is generated based on the T text features. In the above manner, the text to be located can be represented as a feature form, which is convenient for subsequent model prediction, thereby improving the feasibility and operability of the solution.

[0195] Optionally, in the above Figure 3 On the basis of the corresponding fifth embodiment, in a sixth optional embodiment of the method for locating an image region provided by an embodiment of the present application, encoding each word vector in the text vector sequence to obtain text features may include:

[0196] The text features are obtained as follows:

[0197] h t =LSTM(w t ,h t-1 );

[0198] Among them, h t represents the tth text feature in the text feature set, LSTM() represents the encoding using the long short-term memory LSTM network, and w t represents the tth word vector in the text vector sequence, h t-1 Represents the (t-1)th text feature in the text feature set.

[0199] In this embodiment, the image region positioning device can encode each word vector through the LSTM structure to obtain text features. T}, where T means there are T words in the text to be located, e t represents the tth word in the text to be located. First, we can represent each word by its word vector, and get the word vector representation of the text to be located, that is, we get the text vector sequence {w1,w2,...,w T}, where the dimension of each word vector can be 300. For the text to be located, a recurrent neural network with an LSTM structure is used to encode it, namely:

[0200] h t =LSTM(w t ,h t-1 );

[0201] The dimension of the hidden state in LSTM can be set to 512. After processing, the feature representation of the text to be located is obtained, that is, the text feature set h = {h1,h2,...,h T}, where the specific LSTM processing method is as follows:

[0202]

[0203] Ct=ft☉ct-1+it☉gt;

[0204] ht=ot☉tanh(ct);

[0205] Among them, w t represents the tth word vector in the text vector sequence, h t-1 represents the (t-1)th text feature in the text feature set, i t represents the input gate, f t represents the forget gate, o t represents the output gate, ht represents the hidden state, σ is the sigmoid function, tanh() represents the hyperbolic function, g t Represents memory information, c t represents LSTM parameters, ⊙ represents dot product, and T represents the transformation mapping matrix.

[0206] LSTM is a long-term storage input. A special unit called a memory cell is similar to an accumulator and a gated neuron. It will have a weight and connect to itself in the next time step, copying the true value of its own state and the accumulated external signal, but this self-connection is controlled by another unit learning and deciding when to clear the memory content.

[0207] Again, in the embodiment of the present application, a method for obtaining text features is provided, that is, a recurrent neural network with an LSTM structure is used to encode word vectors. In the above manner, the gradient vanishing problem caused by the gradual reduction of the gradient back propagation process can be solved by using a network with an LSTM structure. Specifically for language processing tasks, LSTM is suitable for processing problems that are highly related to time series, such as machine translation, dialogue generation, encoding, and decoding.

[0208] Combined with the above introduction, the following will introduce the model training method in this application, please refer to Figure 4 , one embodiment of the method for model training in the embodiments of the present application includes:

[0209] 201. Obtain a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained;

[0210] In this embodiment, the model training device first obtains a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, and the set of candidate image regions to be trained includes a first image candidate region to be trained and a second image candidate region to be trained. At this time, the first text to be trained and the first image candidate region to be trained that have a matching relationship are used as positive samples, and the second text to be trained and the second image candidate region to be trained that have a matching relationship are used as positive samples. The first text to be trained and the second image candidate region to be trained that do not have a matching relationship are used as negative samples, and the second text to be trained and the first image candidate region to be trained that do not have a matching relationship are used as negative samples.

[0211] It is understandable that the model training device is deployed on the server.

[0212] 202. Determine a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained;

[0213] In this embodiment, the model training device learns the matching relationship between natural sentences and candidate image regions based on positive samples and negative samples, thereby constructing a target loss function. The target loss function is mainly used to measure the similarity between the candidate image region and the natural sentence.

[0214] 203. The target loss function is used to train the image region localization network model to be trained to obtain the image region localization network model, wherein the image region localization network model is used to determine the matching relationship between the image candidate region and the text to be located according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be located.

[0215] In this embodiment, the model training device uses the constructed target loss function to train the image region positioning network model to be trained, thereby obtaining the image region positioning network model. The image region positioning network model can be used to predict the matching degree between the image candidate region and the text to be positioned. The higher the matching degree, the higher the degree of association.

[0216] In an embodiment of the present application, a method for model training is provided, firstly obtaining a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, and the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, then determining a target loss function based on the first text to be trained, the second text to be trained, the first candidate image region to be trained, and the second candidate image region to be trained, and finally using the target loss function to train a network model for locating image regions to be trained, to obtain an image region locating network model. In the above manner, an image region locating network model for determining the matching relationship between an image candidate region and a text can be trained, and the adopted target function can measure the similarity between the image candidate region and the text, so as to learn the matching relationship between the text and the image candidate region, thereby improving the feasibility and operability of the scheme.

[0217] Optionally, in the above Figure 4 On the basis of the corresponding embodiments, in a first optional embodiment of the model training method provided in the embodiments of the present application, determining the target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained may include:

[0218] The target loss function is determined as follows:

[0219]

[0220] Among them, L represents the target loss function, represents the first candidate region of the image to be trained, represents the first text to be trained, represents the second candidate region of the image to be trained, represents the second text to be trained, d() represents the data pair to be trained, max() represents the maximum value, λ1 represents the first parameter control weight, λ2 represents the second parameter control weight, u1 represents the first preset threshold, and u2 represents the second preset threshold.

[0221] In this embodiment, the target loss function constructed by the model training device is introduced. Based on positive samples and negative samples, the constructed target loss function is expressed as:

[0222]

[0223] in, represents positive samples, that is, image candidate regions and natural language pairs with semantic relationships. and Represents negative samples, that is, irrelevant image candidate regions and natural language pairs. is a positive sample, for Take a corresponding negative sample The learning of this matching function makes the matching relationship of positive samples higher than that of negative samples. is a positive sample, for Take a corresponding negative sample The learning of this matching function makes the matching relationship of positive samples higher than that of negative samples.

[0224] Secondly, in an embodiment of the present application, a method for determining a target loss function is provided based on a first text to be trained, a second text to be trained, a first image candidate region to be trained, and a second image candidate region to be trained. Through the above method, the defined target loss function describes the matching relationship between the image and the natural language from two different directions, wherein one direction is to de-associate the image candidate region with the natural language, and the other direction is to de-associate the natural language with the image candidate region. The main goal of the design of this target loss function is to make the similarity between the semantically associated image candidate region and the natural language pair higher than that between the non-semantically associated image candidate region and the natural language pair. Thereby improving the training accuracy of the model.

[0225] The image area positioning device in this application is described in detail below. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of an image region positioning device in an embodiment of the present application. The image region positioning device 30 includes:

[0226] An acquisition module 301 is used to acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1;

[0227] A generating module 302, configured to generate a region semantic information set according to the image candidate region set acquired by the acquiring module 301, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to one image candidate region;

[0228] The acquisition module 301 is further used to acquire the enhanced semantic information set corresponding to the regional semantic information set generated by the generation module 302 through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0229] The acquisition module 301 is further used to acquire a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0230] The acquisition module 301 is further used to acquire the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0231] The determination module 303 is used to determine a target image candidate region from the image candidate region set according to the matching degree between the text feature set obtained by the acquisition module 301 and each enhanced semantic information.

[0232] In this embodiment, the acquisition module 301 acquires an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1. The generation module 302 generates a region semantic information set according to the image candidate region set acquired by the acquisition module 301, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region. The acquisition module 301 acquires an enhanced semantic information set corresponding to the region semantic information set generated by the generation module 302 through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, and each enhanced semantic information corresponds to a region semantic information. The graph convolutional network is used to construct An association relationship between the semantic information of each region is established, the acquisition module 301 acquires a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1, the acquisition module 301 acquires the matching degree between the text feature set and each enhanced semantic information through the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be located, and the determination module 303 determines the target image candidate region from the image candidate region set according to the matching degree between the text feature set acquired by the acquisition module 301 and each enhanced semantic information.

[0233] In an embodiment of the present application, an image region positioning device is provided, which first obtains an image candidate region set in the image to be positioned, wherein the image candidate region set includes N image candidate regions, then generates a region semantic information set based on the image candidate region set, each region semantic information corresponds to an image candidate region, and then obtains an enhanced semantic information set corresponding to the region semantic information set through a graph convolutional network, each enhanced semantic information corresponds to a region semantic information, and the graph convolutional network is used to construct the association relationship between each region semantic information. In addition, a text feature set corresponding to the text to be positioned is obtained, and then the matching degree of the text feature set and each enhanced semantic information can be obtained through the image region positioning network model. Finally, according to the matching degree of the text feature set and each enhanced semantic information, the target image candidate region is determined from the image candidate region set. In the above manner, the use of a graph convolutional network can effectively enhance the semantic representation between image candidate regions, and takes into account the spatial relationship between each image candidate region, which is conducive to improving the accuracy of image region positioning, and then improving the ability to understand the image.

[0234] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the image area positioning device 30 provided in the embodiment of the present application,

[0235] The generating module 302 is specifically configured to obtain the regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes regional information, and the regional information includes position information of the image candidate region in the image to be located, and size information of the image candidate region;

[0236] When the region semantic information corresponding to the N image candidate regions is acquired, the region semantic information set is generated according to the N region semantic information.

[0237] Secondly, in an embodiment of the present application, a method for generating a set of regional semantic information is provided. First, the regional semantic information corresponding to the image candidate area is obtained through a convolutional neural network. The image candidate area includes regional information, and the regional information includes the position information of the image candidate area in the image to be located, and the size information of the image candidate area. When the regional semantic information corresponding to N image candidate areas is obtained, a set of regional semantic information is generated according to the N regional semantic information. Through the above method, CNN can be used to extract the regional semantic information of each image candidate area. CNN is a feedforward neural network whose artificial neurons can respond to surrounding units within a part of the coverage range, so that it has excellent performance in large image processing, thereby improving the accuracy of information extraction.

[0238] Optionally, in the above Figure 5On the basis of the corresponding embodiment, in another embodiment of the image area positioning device 30 provided in the embodiment of the present application,

[0239] The acquisition module 301 is specifically used to acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any one regional semantic information in the regional semantic information set, and the second regional semantic information is any one regional semantic information in the regional semantic information set;

[0240] Acquire the connection edge strength between the first region semantic information and the second region semantic information;

[0241] Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

[0242] Determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0243] The enhanced semantic information set corresponding to the target connection matrix is ​​determined through the graph convolutional network.

[0244] Secondly, in an embodiment of the present application, a method for obtaining an enhanced semantic information set through GCN is provided. First, the first region semantic information and the second region semantic information are obtained from the region semantic information set, and then the connection edge strength between the first region semantic information and the second region semantic information is obtained. Next, the connection edge strength between the first region semantic information and the second region semantic information is normalized to obtain the normalized strength, and then the target connection matrix is ​​determined according to the normalized strength between the regional semantic information in the regional semantic information set. Finally, the enhanced semantic information set corresponding to the target connection matrix is ​​determined through a graph convolutional network. In the above manner, GCN is used to construct the semantic relationship between candidate regions of the image, thereby fully considering the spatial information and semantic relationship, thereby improving the image-based positioning performance.

[0245] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the image area positioning device 30 provided in the embodiment of the present application,

[0246] The acquisition module 301 is specifically used to generate a connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0247] The target connection matrix is ​​generated according to the connection matrix and the unit matrix.

[0248] Again, in an embodiment of the present application, a method for determining a target connection matrix is ​​provided based on the normalized strength between each regional semantic information in a regional semantic information set, that is, firstly, a connection matrix is ​​generated based on the normalized strength between each regional semantic information in a regional semantic information set, and then a target connection matrix is ​​generated based on the connection matrix and a unit matrix. Through the above method, a normalized processing method is adopted, so that the absolute value of the numerical value of the physical system can be converted into a relative value relationship, thereby simplifying the calculation and reducing the value. In addition, in order to further enhance its own information, the corresponding connection matrix is ​​further added with a unit matrix to form a target connection matrix.

[0249] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the image area positioning device 30 provided in the embodiment of the present application,

[0250] The acquisition module is specifically used to calculate the enhanced semantic information set in the following manner:

[0251]

[0252] Among them, the represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the jth enhanced semantic information corresponding to the (k-1)th layer of the graph convolutional network, represents the first network parameter of the graph convolutional network of the kth layer, represents the second network parameter of the graph convolutional network of the kth layer, the j∈neighboring(i) represents that the jth node belongs to the neighboring node of the i-th node, the E ij Represents an element in the target connection matrix.

[0253] Furthermore, in an embodiment of the present application, a specific method for determining the enhanced semantic information set corresponding to the target connection matrix through GCN is provided. Through the above method, a specific calculation method is provided for GCN-based calculation, thereby improving the feasibility and operability of the solution.

[0254] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the image area positioning device 30 provided in the embodiment of the present application,

[0255] The acquisition module 301 is specifically used to acquire the text to be located;

[0256] Acquire a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word;

[0257] Encoding each word vector in the text vector sequence to obtain text features;

[0258] When the text features corresponding to the T word vectors are obtained, the text feature set is generated according to the T text features.

[0259] Secondly, in an embodiment of the present application, a method for obtaining a text feature set is provided, that is, first obtaining the text to be located, and then obtaining a text vector sequence based on the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word, and then each word vector in the text vector sequence is encoded to obtain text features, and when the text features corresponding to the T word vectors are obtained, a text feature set is generated based on the T text features. In the above manner, the text to be located can be represented as a feature form, which is convenient for subsequent model prediction, thereby improving the feasibility and operability of the solution.

[0260] Optionally, in the above Figure 5 On the basis of the corresponding embodiment, in another embodiment of the image area positioning device 30 provided in the embodiment of the present application,

[0261] The acquisition module 301 is specifically used to acquire the text features in the following manner:

[0262] h t =LSTM(w t ,h t-1 );

[0263] Among them, the h t represents the tth text feature in the text feature set, the LSTM() represents the encoding using the long short-term memory LSTM network, the w t represents the tth word vector in the text vector sequence, the h t-1 represents the (t-1)th text feature in the text feature set.

[0264] Again, in the embodiment of the present application, a method for obtaining text features is provided, that is, a recurrent neural network with an LSTM structure is used to encode word vectors. In the above manner, the gradient vanishing problem caused by the gradual reduction of the gradient back propagation process can be solved by using a network with an LSTM structure. Specifically for language processing tasks, LSTM is suitable for processing problems that are highly related to time series, such as machine translation, dialogue generation, encoding, and decoding.

[0265] The model training device in this application is described in detail below. Figure 6 , Figure 6 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application. The model training device 40 includes:

[0266] The acquisition module 401 is used to acquire a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained;

[0267] A determination module 402, configured to determine a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained acquired by the acquisition module 401;

[0268] The training module 403 is used to train the image region positioning network model to be trained using the target loss function determined by the determination module 402 to obtain the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

[0269] In this embodiment, the acquisition module 401 acquires a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained, and the determination module 402 determines whether the text to be trained is a text set or a set of candidate image regions to be trained. 02 According to the first text to be trained, the second text to be trained, the first image candidate area to be trained and the second image candidate area to be trained obtained by the acquisition module 401, a target loss function is determined, and the training module 403 uses the target loss function determined by the determination module 402 to train the image area positioning network model to be trained to obtain an image area positioning network model, wherein the image area positioning network model is used to determine the matching relationship between the image candidate area and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate area, and the text feature set has a corresponding relationship with the text to be positioned.

[0270] In an embodiment of the present application, a model training device is provided. First, the model training device obtains a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, and the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained. Then, according to the first text to be trained, the second text to be trained, the first candidate image region to be trained and the second candidate image region to be trained, a target loss function is determined. Finally, the target loss function is used to train a network model for locating image regions to be trained, and an image region locating network model is obtained. In the above manner, an image region locating network model for determining the matching relationship between an image candidate region and a text can be trained, and the adopted target function can measure the similarity between the image candidate region and the text, so as to learn the matching relationship between the text and the image candidate region, thereby improving the feasibility and operability of the scheme.

[0271] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the model training device 40 provided in the embodiment of the present application,

[0272] The determination module 402 is specifically used to determine the target loss function in the following manner:

[0273]

[0274] Wherein, L represents the target loss function, represents the first candidate region of the image to be trained, represents the first text to be trained, the represents the second candidate region of the image to be trained, represents the second text to be trained, d() represents the data pair to be trained, max() represents the maximum value, λ1 represents the first parameter control weight, λ2 represents the second parameter control weight, u1 represents the first preset threshold, and u2 represents the second preset threshold.

[0275] Secondly, in an embodiment of the present application, a method for determining a target loss function is provided based on a first text to be trained, a second text to be trained, a first image candidate region to be trained, and a second image candidate region to be trained. Through the above method, the defined target loss function describes the matching relationship between the image and the natural language from two different directions, wherein one direction is to de-associate the image candidate region with the natural language, and the other direction is to de-associate the natural language with the image candidate region. The main goal of the design of this target loss function is to make the similarity between the semantically associated image candidate region and the natural language pair higher than that between the non-semantically associated image candidate region and the natural language pair. Thereby improving the training accuracy of the model.

[0276] The embodiment of the present invention also provides another image region positioning device, such as Figure 7 For the sake of convenience, only the parts related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present invention. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example:

[0277] Figure 7 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present invention. Figure 7 The mobile phone includes: a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590. Those skilled in the art will understand that Figure 7The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0278] Combine the following Figure 7 A detailed introduction to the various components of the mobile phone:

[0279] The RF circuit 510 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 580 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 510 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 510 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0280] The memory 520 can be used to store software programs and modules. The processor 580 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 520. The memory 520 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0281] The input unit 530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 531 or near the touch panel 531 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 580, and can receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 531, the input unit 530 may also include other input devices 532. Specifically, other input devices 532 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0282] The display unit 540 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 540 may include a display panel 541. Optionally, the display panel 541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 531 may cover the display panel 541. When the touch panel 531 detects a touch operation on or near it, it is transmitted to the processor 580 to determine the type of touch event. Subsequently, the processor 580 provides a corresponding visual output on the display panel 541 according to the type of touch event. Although in Figure 7 In the embodiment, the touch panel 531 and the display panel 541 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the mobile phone.

[0283] The mobile phone may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 541 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0284] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the mobile phone. The audio circuit 560 can transmit the received audio data to the speaker 561 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another mobile phone through the RF circuit 510, or the audio data is output to the memory 520 for further processing.

[0285] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 570. It provides users with wireless broadband Internet access. Figure 7 A WiFi module 570 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0286] The processor 580 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 580 may include one or more processing units; optionally, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 580.

[0287] The mobile phone also includes a power source 590 (such as a battery) for supplying power to various components. Optionally, the power source can be logically connected to the processor 580 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0288] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0289] In the embodiment of the present invention, the processor 580 included in the terminal device also has the following functions:

[0290] Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1;

[0291] Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region;

[0292] Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0293] Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0294] Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0295] According to the matching degree between the text feature set and each enhanced semantic information, a target image candidate region is determined from the image candidate region set.

[0296] Optionally, the processor 580 is specifically configured to perform the following steps:

[0297] Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes region information, and the region information includes position information of the image candidate region in the image to be located, and size information of the image candidate region;

[0298] When the region semantic information corresponding to the N image candidate regions is acquired, the region semantic information set is generated according to the N region semantic information.

[0299] Optionally, the processor 580 is specifically configured to perform the following steps:

[0300] Acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any one regional semantic information in the regional semantic information set, and the second regional semantic information is any one regional semantic information in the regional semantic information set;

[0301] Acquire the connection edge strength between the first region semantic information and the second region semantic information;

[0302] Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

[0303] Determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0304] The enhanced semantic information set corresponding to the target connection matrix is ​​determined through the graph convolutional network.

[0305] Optionally, the processor 580 is specifically configured to perform the following steps:

[0306] Generate a connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0307] The target connection matrix is ​​generated according to the connection matrix and the unit matrix.

[0308] Optionally, the processor 580 is specifically configured to perform the following steps:

[0309] The enhanced semantic information set is calculated in the following manner:

[0310]

[0311] Among them, the represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the jth enhanced semantic information corresponding to the (k-1)th layer of the graph convolutional network, represents the first network parameter of the graph convolutional network of the kth layer, represents the second network parameter of the graph convolutional network of the kth layer, the j∈neighboring(i) represents that the jth node belongs to the neighboring node of the i-th node, the Eij Represents an element in the target connection matrix.

[0312] Optionally, the processor 580 is specifically configured to perform the following steps:

[0313] Obtain the text to be located;

[0314] Acquire a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word;

[0315] Encoding each word vector in the text vector sequence to obtain text features;

[0316] When the text features corresponding to the T word vectors are obtained, the text feature set is generated according to the T text features.

[0317] Optionally, the processor 580 is specifically configured to perform the following steps:

[0318] The text features are obtained in the following manner:

[0319] h t =LSTM(w t ,h t-1 );

[0320] Among them, the h t represents the tth text feature in the text feature set, the LSTM() represents the encoding using the long short-term memory LSTM network, the w t represents the tth word vector in the text vector sequence, the h t-1 represents the (t-1)th text feature in the text feature set.

[0321] Figure 8 6 is a schematic diagram of a server structure provided by an embodiment of the present invention. The server 600 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 622 (for example, one or more processors) and a memory 632, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 may be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 622 may be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the server 600.

[0322] The server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0323] The steps performed by the server in the above embodiment can be based on the Figure 8 The server structure shown.

[0324] In the embodiment of the present invention, the CPU 622 included in the server also has the following functions:

[0325] Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1;

[0326] Generate a region semantic information set according to the image candidate region set, wherein the region semantic information set includes N region semantic information, and each region semantic information corresponds to an image candidate region;

[0327] Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information;

[0328] Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1;

[0329] Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned;

[0330] According to the matching degree between the text feature set and each enhanced semantic information, a target image candidate region is determined from the image candidate region set.

[0331] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0332] Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes region information, and the region information includes position information of the image candidate region in the image to be located, and size information of the image candidate region;

[0333] When the region semantic information corresponding to the N image candidate regions is acquired, the region semantic information set is generated according to the N region semantic information.

[0334] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0335] Acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any one regional semantic information in the regional semantic information set, and the second regional semantic information is any one regional semantic information in the regional semantic information set;

[0336] Acquire the connection edge strength between the first region semantic information and the second region semantic information;

[0337] Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

[0338] Determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0339] The enhanced semantic information set corresponding to the target connection matrix is ​​determined through the graph convolutional network.

[0340] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0341] Generate a connection matrix according to the normalized strength between each region semantic information in the region semantic information set;

[0342] The target connection matrix is ​​generated according to the connection matrix and the unit matrix.

[0343] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0344] The enhanced semantic information set is calculated in the following manner:

[0345]

[0346] Among them, the represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the jth enhanced semantic information corresponding to the (k-1)th layer of the graph convolutional network, represents the first network parameter of the graph convolutional network of the kth layer, represents the second network parameter of the graph convolutional network of the kth layer, the j∈neighboring(i) represents that the jth node belongs to the neighboring node of the i-th node, the E ij Represents an element in the target connection matrix.

[0347] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0348] Obtain the text to be located;

[0349] Acquire a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word;

[0350] Encoding each word vector in the text vector sequence to obtain text features;

[0351] When the text features corresponding to the T word vectors are obtained, the text feature set is generated according to the T text features.

[0352] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0353] The text features are obtained in the following manner:

[0354] h t =LSTM(w t ,h t-1 );

[0355] Among them, the h t represents the tth text feature in the text feature set, the LSTM() represents the encoding using the long short-term memory LSTM network, the w t represents the tth word vector in the text vector sequence, the h t-1 represents the (t-1)th text feature in the text feature set.

[0356] In the embodiment of the present invention, the CPU 622 included in the server also has the following functions:

[0357] Acquire a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained;

[0358] Determining a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained;

[0359] The target loss function is used to train the image region positioning network model to be trained to obtain the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

[0360] Optionally, the CPU 622 is specifically configured to perform the following steps:

[0361] The objective loss function is determined in the following manner:

[0362]

[0363] Wherein, L represents the target loss function, represents the first candidate region of the image to be trained, represents the first text to be trained, the represents the second candidate region of the image to be trained, represents the second text to be trained, d() represents the data pair to be trained, max() represents the maximum value, λ1 represents the first parameter control weight, λ2 represents the second parameter control weight, u1 represents the first preset threshold, and u2 represents the second preset threshold.

[0364] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0365] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0366] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0367] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0368] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0369] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for locating an image region, characterized in that: include: Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1; Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes region information, and the region information includes position information of the image candidate region in the image to be located, and size information of the image candidate region; When the regional semantic information corresponding to the N image candidate regions is obtained, the regional semantic information set is generated according to the N regional semantic information, wherein the regional semantic information set includes N regional semantic information, each regional semantic information corresponds to an image candidate region, and the regional semantic information is a semantic representation of the image candidate region; Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information; Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1; Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned; According to the matching degree between the text feature set and each enhanced semantic information, a target image candidate region is determined from the image candidate region set.

2. The method according to claim 1, characterized in that The step of obtaining the enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network includes: Acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any one regional semantic information in the regional semantic information set, and the second regional semantic information is any one regional semantic information in the regional semantic information set; Acquire the connection edge strength between the first region semantic information and the second region semantic information; Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength; Determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set; The enhanced semantic information set corresponding to the target connection matrix is ​​determined through the graph convolutional network.

3. The method according to claim 2, characterized in that The step of determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set includes: Generate a connection matrix according to the normalized strength between each region semantic information in the region semantic information set; The target connection matrix is ​​generated according to the connection matrix and the unit matrix.

4. The method according to claim 2 or 3, characterized in that: The step of determining the enhanced semantic information set corresponding to the target connection matrix through the graph convolutional network includes: The enhanced semantic information set is calculated in the following manner: Among them, the represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the jth enhanced semantic information corresponding to the (k-1)th layer of the graph convolutional network, represents the first network parameter of the graph convolutional network of the kth layer, represents the second network parameter of the graph convolutional network of the kth layer, the j∈neighboring(i) represents that the jth node belongs to the neighboring node of the i-th node, the E ij Represents an element in the target connection matrix.

5. The method according to claim 1, characterized in that The step of obtaining a text feature set corresponding to the text to be located includes: Obtain the text to be located; Acquire a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word; Encoding each word vector in the text vector sequence to obtain text features; When the text features corresponding to the T word vectors are obtained, the text feature set is generated according to the T text features.

6. The method according to claim 5, characterized in that The encoding process is performed on each word vector in the text vector sequence to obtain text features, including: The text features are obtained in the following manner: h t =LSTM(w t ,h t-1 ); Among them, the h t represents the tth text feature in the text feature set, the LSTM() represents the encoding using the long short-term memory LSTM network, the w t represents the tth word vector in the text vector sequence, the h t-1 represents the (t-1)th text feature in the text feature set.

7. The method according to claim 1, characterized in that The training process of the model includes: Acquire a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained; Determining a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained; The target loss function is used to train the image region positioning network model to be trained to obtain the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

8. The method according to claim 7, characterized in that The determining of the target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained includes: The objective loss function is determined in the following manner: Wherein, L represents the target loss function, represents the first candidate region of the image to be trained, represents the first text to be trained, the represents the second candidate region of the image to be trained, represents the second text to be trained, d() represents the data pair to be trained, max() represents the maximum value, λ1 represents the first parameter control weight, λ2 represents the second parameter control weight, u1 represents the first preset threshold, and u2 represents the second preset threshold.

9. An image area positioning device, characterized in that: include: An acquisition module, used to acquire a set of image candidate regions in the image to be located, wherein the set of image candidate regions includes N image candidate regions, where N is an integer greater than or equal to 1; A generating module, configured to generate a region semantic information set according to the image candidate region set acquired by the acquiring module, wherein the region semantic information set includes N region semantic information, each region semantic information corresponds to an image candidate region, and the region semantic information is a semantic representation of the image candidate region; The acquisition module is further used to acquire, through a graph convolutional network, an enhanced semantic information set corresponding to the regional semantic information set generated by the generation module, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information; The acquisition module is further used to acquire a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1; The acquisition module is further used to acquire the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned; A determination module, configured to determine a target image candidate region from the image candidate region set according to a matching degree between the text feature set acquired by the acquisition module and each enhanced semantic information; The generating module is specifically used to obtain the regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes regional information, and the regional information includes the position information of the image candidate region in the image to be located, and the size information of the image candidate region; When the region semantic information corresponding to the N image candidate regions is acquired, the region semantic information set is generated according to the N region semantic information.

10. The device according to claim 9, characterized in that The acquisition module is specifically used to acquire first regional semantic information and second regional semantic information from the regional semantic information set, wherein the first regional semantic information is any one regional semantic information in the regional semantic information set, and the second regional semantic information is any one regional semantic information in the regional semantic information set; Acquire the connection edge strength between the first region semantic information and the second region semantic information; Normalizing the strength of the connection edge between the first region semantic information and the second region semantic information to obtain a normalized strength; Determining a target connection matrix according to the normalized strength between each region semantic information in the region semantic information set; The enhanced semantic information set corresponding to the target connection matrix is ​​determined through the graph convolutional network.

11. The device according to claim 10, characterized in that The acquisition module is specifically used to generate a connection matrix according to the normalized strength between each piece of regional semantic information in the regional semantic information set; The target connection matrix is ​​generated according to the connection matrix and the unit matrix.

12. The device according to claim 10 or 11, characterized in that The acquisition module is specifically used to calculate the enhanced semantic information set in the following manner: Among them, the represents the i-th enhanced semantic information corresponding to the k-th layer of the graph convolutional network, represents the jth enhanced semantic information corresponding to the (k-1)th layer of the graph convolutional network, represents the first network parameter of the graph convolutional network of the kth layer, represents the second network parameter of the graph convolutional network of the kth layer, the j∈neighboring(i) represents that the jth node belongs to the neighboring node of the i-th node, the E ij Represents an element in the target connection matrix.

13. The device according to claim 9, characterized in that The acquisition module is specifically used to acquire the text to be located; Acquire a text vector sequence according to the text to be located, wherein the text vector sequence includes T word vectors, each word vector corresponds to a word; Encoding each word vector in the text vector sequence to obtain text features; When the text features corresponding to the T word vectors are obtained, the text feature set is generated according to the T text features.

14. The device according to claim 13, characterized in that The acquisition module is specifically used to acquire the text features in the following manner: h t =LSTM(w t ,h t-1 ); Among them, the h t represents the tth text feature in the text feature set, the LSTM() represents the encoding using the long short-term memory LSTM network, the w t represents the tth word vector in the text vector sequence, the h t-1 represents the (t-1)th text feature in the text feature set.

15. The device according to claim 9, characterized in that Also includes: A model training device, the model training device comprising: an acquisition module, used for acquiring a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained; A determination module, configured to determine a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained acquired by the acquisition module; A training module is used to train the image region positioning network model to be trained using the target loss function determined by the determination module to obtain an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned based on a text feature set and enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

16. The device according to claim 15, characterized in that The model training device comprises: an acquisition module, used for acquiring a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained; A determination module, configured to determine a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained acquired by the acquisition module 401; A training module is used to train the image region positioning network model to be trained using the target loss function determined by the determination module 402 to obtain an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned based on the text feature set and enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

17. The device according to claim 16, characterized in that The determination module is specifically used to determine the target loss function in the following manner: Wherein, L represents the target loss function, represents the first candidate region of the image to be trained, represents the first text to be trained, the represents the second candidate region of the image to be trained, represents the second text to be trained, d() represents the data pair to be trained, max() represents the maximum value, λ1 represents the first parameter control weight, λ2 represents the second parameter control weight, u1 represents the first preset threshold, and u2 represents the second preset threshold.

18. A terminal device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, comprising the following steps: Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1; Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes region information, and the region information includes position information of the image candidate region in the image to be located, and size information of the image candidate region; When the regional semantic information corresponding to the N image candidate regions is obtained, the regional semantic information set is generated according to the N regional semantic information, wherein the regional semantic information set includes N regional semantic information, each regional semantic information corresponds to an image candidate region, and the regional semantic information is a semantic representation of the image candidate region; Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information; Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1; Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned; Determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

19. A server, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, comprising the following steps: Acquire an image candidate region set in the image to be located, wherein the image candidate region set includes N image candidate regions, and N is an integer greater than or equal to 1; Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes region information, and the region information includes position information of the image candidate region in the image to be located, and size information of the image candidate region; When the regional semantic information corresponding to the N image candidate regions is obtained, the regional semantic information set is generated according to the N regional semantic information, wherein the regional semantic information set includes N regional semantic information, each regional semantic information corresponds to an image candidate region, and the regional semantic information is a semantic representation of the image candidate region; Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information; Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1; Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned; Determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

20. The server according to claim 19, characterized in that The processor is used to execute the program in the memory, and further comprises the following steps: Acquire a set of texts to be trained and a set of candidate image regions to be trained, wherein the set of texts to be trained includes a first text to be trained and a second text to be trained, the set of candidate image regions to be trained includes a first candidate image region to be trained and a second candidate image region to be trained, the first text to be trained has a matching relationship with the first candidate image region to be trained, and the first text to be trained has no matching relationship with the second candidate image region to be trained, the second text to be trained has a matching relationship with the second candidate image region to be trained, and the second text to be trained has no matching relationship with the first candidate image region to be trained; Determining a target loss function according to the first text to be trained, the second text to be trained, the first image candidate region to be trained, and the second image candidate region to be trained; The target loss function is used to train the image region positioning network model to be trained to obtain the image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned according to the text feature set and the enhanced semantic information, the enhanced semantic information has a corresponding relationship with the image candidate region, and the text feature set has a corresponding relationship with the text to be positioned.

21. A method for locating an image region, characterized in that: include: receiving an image positioning instruction; In response to the image positioning instruction, obtaining an image candidate region set in the image to be positioned according to the image positioning instruction, wherein the image candidate region set includes N image candidate regions, where N is an integer greater than or equal to 1; Acquire regional semantic information corresponding to the image candidate region through a convolutional neural network, wherein the image candidate region includes region information, and the region information includes position information of the image candidate region in the image to be located, and size information of the image candidate region; When the regional semantic information corresponding to the N image candidate regions is obtained, the regional semantic information set is generated according to the N regional semantic information, wherein the regional semantic information set includes N regional semantic information, each regional semantic information corresponds to an image candidate region, and the regional semantic information is a semantic representation of the image candidate region; Obtaining an enhanced semantic information set corresponding to the regional semantic information set through a graph convolutional network, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponds to a regional semantic information, and the graph convolutional network is used to construct an association relationship between each regional semantic information; Obtain a text feature set corresponding to the text to be located, wherein the text to be located includes T words, the text feature set includes T word features, each word corresponds to a word feature, and T is an integer greater than or equal to 1; Obtaining the matching degree between the text feature set and each enhanced semantic information through an image region positioning network model, wherein the image region positioning network model is used to determine the matching relationship between the image candidate region and the text to be positioned; Determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information; An image generation instruction is sent to the client, so that the client displays the target image candidate area according to the image generation instruction.

Images